Closed fjperini closed 5 years ago
@luispuerto A line within def parser():
must be changed.
I'm working to make them compatible with Python 3. I hope to find the line that needs to be changed. Ha!
def parser():
"""Interface to g.parser, intended to be run from the top-level, e.g.:
::
if __name__ == "__main__":
options, flags = grass.parser()
main()
Thereafter, the global variables "options" and "flags" will be
dictionaries containing option/flag values, keyed by lower-case
option/flag names. The values in "options" are strings, those in
"flags" are Python booleans.
Overview table of parser standard options:
https://grass.osgeo.org/grass74/manuals/parser_standard_options.html
"""
if not os.getenv("GISBASE"):
print("You must be in GRASS GIS to run this program.", file=sys.stderr)
sys.exit(1)
cmdline = [basename(encode(sys.argv[0]))]
cmdline += [b'"' + encode(arg) + b'"' for arg in sys.argv[1:]]
environ[b'CMDLINE'] = b' '.join(cmdline)
Maybe the problem is in
environ[b'CMDLINE'] = b' '.join(cmdline)
Definitely related os.environ
, Python 3 requires str
.
Python 2 and Python 3 differences:
In Python 2:
Bytes == Strings
Unicodes != Strings
In Python 3:
Bytes != Strings
Unicodes == Strings
Apparently, only v.dissolve
and v.to.lines
the only ones that do not work, all the others work well.
https://trac.osgeo.org/grass/wiki/Python3Support https://gis.stackexchange.com/questions/304245/grass-processing-not-working-on-qgis https://stackoverflow.com/questions/39949587/can-python-os-environ-get-ever-return-a-non-string http://osgeo-org.1560.x6.nabble.com/Python-3-porting-and-unicode-td5344215.html
Here is a solution mentioned:
https://stackoverflow.com/questions/52269281/fix-import-error-on-using-environb-in-python
# 1. at the top, add an import
import pipes
# 2. remove the `from os import environb as environ` line altogether
# 3. in def parse(), use
cmdline = [basename(sys.argv[0])]
cmdline += (pipes.quote(a) for a in sys.argv[1:])
os.environ['CMDLINE'] = ' '.join(cmdline)
We are very close to solving it. This must be reported to the GRASS developers.
@luispuerto @nickrobison I've solved v.dissolve
and v.to.lines
🎉
With all the changes applied, GRASS should be completely compatible with Python 3.
While we wait for version 7.8/8.0, which apparently (https://gis.stackexchange.com/a/304598) will be compatible with Python 3.
Files to which I had to apply changes:
/etc/python/grass/script/core.py
--- a/etc/python/grass/script/core.py
+++ b/etc/python/grass/script/core.py
@@ -25,9 +25,11 @@
import subprocess
import shutil
import codecs
+import string
+import random
import types as python_types
-from .utils import KeyValue, parse_key_val, basename, encode
+from .utils import KeyValue, parse_key_val, basename, encode, decode
from grass.exceptions import ScriptError, CalledModuleError
# i18N
@@ -38,13 +40,13 @@
# python2
import __builtin__
from os import environ
+ __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.ugettext
except ImportError:
# python3
import builtins as __builtin__
from os import environb as environ
unicode = str
-__builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.lgettext
-
+ __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.gettext
# subprocess wrapper that uses shell on Windows
@@ -104,6 +106,21 @@
except TypeError:
pass
return bytes(val)
+
+
+def _make_unicode(val, enc):
+ """Convert value to unicode with given encoding
+
+ :param val: value to be converted
+ :param enc: encoding to be used
+ """
+ if val is None or enc is None:
+ return val
+ else:
+ if enc == 'default':
+ return decode(val)
+ else:
+ return decode(val, encoding=enc)
def get_commands():
@@ -301,7 +318,8 @@
continue
# convert string to bytes
opt = encode(opt)
- if val != None:
+ prog = encode(prog)
+ if val is not None:
if opt.startswith(b'_'):
opt = opt[1:]
warning(_("To run the module <%s> add underscore at the end"
@@ -328,7 +346,8 @@
else:
# TODO: construction of the whole command is far from perfect
args = make_command(*args, **kwargs)
- raise CalledModuleError(module=None, code=repr(args),
+ code = ''.join([decode(each) for each in args])
+ raise CalledModuleError(module=None, code=code,
returncode=returncode)
def start_command(prog, flags=b"", overwrite=False, quiet=False,
@@ -360,6 +379,9 @@
:return: Popen object
"""
+ if 'encoding' in kwargs.keys():
+ encoding = kwargs.pop('encoding')
+
options = {}
popts = {}
for opt, val in kwargs.items():
@@ -379,7 +401,6 @@
sys.stderr.flush()
return Popen(args, **popts)
-
def run_command(*args, **kwargs):
"""Execute a module synchronously
@@ -408,11 +429,18 @@
:raises: ``CalledModuleError`` when module returns non-zero return code
"""
+ encoding = 'default'
+ if 'encoding' in kwargs:
+ encoding = kwargs['encoding']
+
if _capture_stderr and 'stderr' not in kwargs.keys():
kwargs['stderr'] = PIPE
ps = start_command(*args, **kwargs)
if _capture_stderr:
stdout, stderr = ps.communicate()
+ if encoding is not None:
+ stdout = _make_unicode(stdout, encoding)
+ stderr = _make_unicode(stderr, encoding)
returncode = ps.poll()
if returncode:
sys.stderr.write(stderr)
@@ -466,10 +494,17 @@
:return: stdout
"""
+ encoding = 'default'
+ if 'encoding' in kwargs:
+ encoding = kwargs['encoding']
+
if _capture_stderr and 'stderr' not in kwargs.keys():
kwargs['stderr'] = PIPE
process = pipe_command(*args, **kwargs)
stdout, stderr = process.communicate()
+ if encoding is not None:
+ stdout = _make_unicode(stdout, encoding)
+ stderr = _make_unicode(stderr, encoding)
returncode = process.poll()
if _capture_stderr and returncode:
sys.stderr.write(stderr)
@@ -539,12 +574,22 @@
:raises: ``CalledModuleError`` when module returns non-zero return code
"""
+ encoding = 'default'
+ if 'encoding' in kwargs:
+ encoding = kwargs['encoding']
# TODO: should we delete it from kwargs?
stdin = kwargs['stdin']
+ if encoding is None or encoding == 'default':
+ stdin = encode(stdin)
+ else:
+ stdin = encode(stdin, encoding=encoding)
if _capture_stderr and 'stderr' not in kwargs.keys():
kwargs['stderr'] = PIPE
process = feed_command(*args, **kwargs)
unused, stderr = process.communicate(stdin)
+ if encoding is not None:
+ unused = _make_unicode(unused, encoding)
+ stderr = _make_unicode(stderr, encoding)
returncode = process.poll()
if _capture_stderr and returncode:
sys.stderr.write(stderr)
@@ -738,14 +783,15 @@
break
try:
[var, val] = line.split(b'=', 1)
+ [var, val] = [decode(var), decode(val)]
except:
raise SyntaxError("invalid output from g.parser: %s" % line)
- if var.startswith(b'flag_'):
+ if var.startswith('flag_'):
flags[var[5:]] = bool(int(val))
- elif var.startswith(b'opt_'):
+ elif var.startswith('opt_'):
options[var[4:]] = val
- elif var in [b'GRASS_OVERWRITE', b'GRASS_VERBOSE']:
+ elif var in ['GRASS_OVERWRITE', 'GRASS_VERBOSE']:
os.environ[var] = val
else:
raise SyntaxError("invalid output from g.parser: %s" % line)
@@ -768,7 +814,7 @@
"flags" are Python booleans.
Overview table of parser standard options:
- https://grass.osgeo.org/grass74/manuals/parser_standard_options.html
+ https://grass.osgeo.org/grass77/manuals/parser_standard_options.html
"""
if not os.getenv("GISBASE"):
print("You must be in GRASS GIS to run this program.", file=sys.stderr)
@@ -820,6 +866,30 @@
os.mkdir(tmp)
return tmp
+
+
+def tempname(length, lowercase=False):
+ """Generate a GRASS and SQL compliant random name starting with tmp_
+ followed by a random part of length "length"
+
+ :param int length: length of the random part of the name to generate
+ :param bool lowercase: use only lowercase characters to generate name
+ :returns: String with a random name of length "length" starting with a letter
+ :rtype: str
+
+ :Example:
+
+ >>> tempname(12)
+ 'tmp_MxMa1kAS13s9'
+ """
+
+ chars = string.ascii_lowercase + string.digits
+ if not lowercase:
+ chars += string.ascii_uppercase
+ random_part = ''.join(random.choice(chars) for _ in range(length))
+ randomname = 'tmp_' + random_part
+
+ return randomname
def _compare_projection(dic):
@@ -1270,7 +1340,7 @@
:return: directory of mapsets/elements
"""
- if isinstance(type, python_types.StringTypes) or len(type) == 1:
+ if isinstance(type, str) or len(type) == 1:
types = [type]
store_types = False
else:
and to the remaining files just change file
by open
/grass-7.4.3/scripts/db.py
/grass-7.4.3/scripts/v.db.addtable
/grass-7.4.3/scripts/v.dissolve
/grass-7.4.3/scripts/v.lines
note: file()
is not supported in Python 3. Using open()
I will apply the changes now!
p/d: If a similar error arises, you already know where to start. Now I want that beer :smile: Ha!
The globe
support is almost finished, the option to use the old versions (osgearth 2.7) of osgearth
openscenegraph
apparently builds well.
But I found many things to change for the new versions (3.6.3). For that I will upload new formulas that includes the support for Qt5 necessary to build some libraries, for example:
/usr/local/opt/osgearth-qt5/lib/libosgEarthQt5.dylib
needed by
/PlugIns/qgis/libglobeplugin.dylib
Apparently I just need to correct an error (after many) so that the build is completed.
../src/plugins/globe/featuresource/qgsglobefeaturesource.h:33:134: error: non-virtual member function marked 'override' hides virtual member function
osgEarth::Features::FeatureCursor *createFeatureCursor( const osgEarth::Symbology::Query &query = osgEarth::Symbology::Query() ) override;
^
/usr/local/opt/osgearth-qt5/include/osgEarthFeatures/FeatureSource:149:32: note: hidden overloaded virtual function 'osgEarth::Features::FeatureSource::createFeatureCursor' declared here: different number of parameters (2 vs 1)
virtual FeatureCursor* createFeatureCursor(const Symbology::Query& query, ProgressCallback* progress) =0;
^
I hope to solve it. Ha!
@nickrobison As there are several changes for Python 3, it might be best to create a grass7-dev
formula for QGIS 3 and use it momentarily until GRASS 7.8 / 8.0 arrives.
@luispuerto I'll give you the formula with the changes and the tests in QGIS 2 and QGIS3.
If something comes up in QGIS 2, we'll have to use grass7-dev
.
Tell me which branch in your repo or which pull request I should use.
@luispuerto You can already try it!
I just tested it v.dissolve
in QGIS3&2 and it works perfectly in both. 👍
Again, great work!
Once @nickrobison merge the changes we close this. Ha! Thanks @luispuerto!
Problem reported by @luisspuerto