QGIS3/GRASS: error while executing a script

[fjperini]

Problem reported by @luisspuerto

Your PYTHONPATH points to a site-packages dir for Python 3.x but you are running Python 2.x!

PYTHONPATH is currently: "/usr/local/opt/gdal2-python/lib/python3.7/site-packages:/usr/local/opt/qgis3/lib/python3.7/site-packages:/usr/local/opt/qgis3/libexec/python/lib/python3.7/site-packages:/usr/local/lib/python3.7/site-packages"

You should `unset PYTHONPATH` to fix this.

[fjperini]

@luispuerto A line within def parser(): must be changed. I'm working to make them compatible with Python 3. I hope to find the line that needs to be changed. Ha!

def parser():
    """Interface to g.parser, intended to be run from the top-level, e.g.:

    ::

        if __name__ == "__main__":
            options, flags = grass.parser()
            main()

    Thereafter, the global variables "options" and "flags" will be
    dictionaries containing option/flag values, keyed by lower-case
    option/flag names. The values in "options" are strings, those in
    "flags" are Python booleans.

    Overview table of parser standard options:
    https://grass.osgeo.org/grass74/manuals/parser_standard_options.html
    """
    if not os.getenv("GISBASE"):
        print("You must be in GRASS GIS to run this program.", file=sys.stderr)
        sys.exit(1)

    cmdline = [basename(encode(sys.argv[0]))]
    cmdline += [b'"' + encode(arg) + b'"' for arg in sys.argv[1:]]
    environ[b'CMDLINE'] = b' '.join(cmdline)

Maybe the problem is in

environ[b'CMDLINE'] = b' '.join(cmdline)

Definitely related os.environ, Python 3 requires str.

Python 2 and Python 3 differences:

In Python 2:
  Bytes == Strings
  Unicodes != Strings 
In Python 3:
  Bytes != Strings
  Unicodes == Strings 

Apparently, only v.dissolve and v.to.lines the only ones that do not work, all the others work well.

https://trac.osgeo.org/grass/wiki/Python3Support https://gis.stackexchange.com/questions/304245/grass-processing-not-working-on-qgis https://stackoverflow.com/questions/39949587/can-python-os-environ-get-ever-return-a-non-string http://osgeo-org.1560.x6.nabble.com/Python-3-porting-and-unicode-td5344215.html

[fjperini]

Here is a solution mentioned:

https://stackoverflow.com/questions/52269281/fix-import-error-on-using-environb-in-python

# 1. at the top, add an import
import pipes
# 2. remove the `from os import environb as environ` line altogether
# 3. in def parse(), use
cmdline = [basename(sys.argv[0])]
cmdline += (pipes.quote(a) for a in sys.argv[1:])
os.environ['CMDLINE'] = ' '.join(cmdline)

We are very close to solving it. This must be reported to the GRASS developers.

[fjperini]

@luispuerto @nickrobison I've solved v.dissolve and v.to.lines 🎉

With all the changes applied, GRASS should be completely compatible with Python 3.

While we wait for version 7.8/8.0, which apparently (https://gis.stackexchange.com/a/304598) will be compatible with Python 3.

Files to which I had to apply changes:

/etc/python/grass/script/core.py

--- a/etc/python/grass/script/core.py
+++ b/etc/python/grass/script/core.py
@@ -25,9 +25,11 @@
 import subprocess
 import shutil
 import codecs
+import string
+import random
 import types as python_types

-from .utils import KeyValue, parse_key_val, basename, encode
+from .utils import KeyValue, parse_key_val, basename, encode, decode
 from grass.exceptions import ScriptError, CalledModuleError

 # i18N
@@ -38,13 +40,13 @@
     # python2
     import __builtin__
     from os import environ
+    __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.ugettext
 except ImportError:
     # python3
     import builtins as __builtin__
     from os import environb as environ
     unicode = str
-__builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.lgettext
-
+    __builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.gettext

 # subprocess wrapper that uses shell on Windows

@@ -104,6 +106,21 @@
     except TypeError:
         pass
     return bytes(val)
+
+
+def _make_unicode(val, enc):
+    """Convert value to unicode with given encoding
+
+    :param val: value to be converted
+    :param enc: encoding to be used
+    """
+    if val is None or enc is None:
+        return val
+    else:
+        if enc == 'default':
+            return decode(val)
+        else:
+            return decode(val, encoding=enc)

 def get_commands():
@@ -301,7 +318,8 @@
             continue
         # convert string to bytes
         opt = encode(opt)
-        if val != None:
+        prog = encode(prog)
+        if val is not None:
             if opt.startswith(b'_'):
                 opt = opt[1:]
                 warning(_("To run the module <%s> add underscore at the end"
@@ -328,7 +346,8 @@
     else:
         # TODO: construction of the whole command is far from perfect
         args = make_command(*args, **kwargs)
-        raise CalledModuleError(module=None, code=repr(args),
+        code = ''.join([decode(each) for each in args])
+        raise CalledModuleError(module=None, code=code,
                                 returncode=returncode)

 def start_command(prog, flags=b"", overwrite=False, quiet=False,
@@ -360,6 +379,9 @@

     :return: Popen object
     """
+    if 'encoding' in kwargs.keys():
+        encoding = kwargs.pop('encoding')
+
     options = {}
     popts = {}
     for opt, val in kwargs.items():
@@ -379,7 +401,6 @@
         sys.stderr.flush()
     return Popen(args, **popts)

-
 def run_command(*args, **kwargs):
     """Execute a module synchronously

@@ -408,11 +429,18 @@

     :raises: ``CalledModuleError`` when module returns non-zero return code
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
+
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     ps = start_command(*args, **kwargs)
     if _capture_stderr:
         stdout, stderr = ps.communicate()
+        if encoding is not None:
+            stdout = _make_unicode(stdout, encoding)
+            stderr = _make_unicode(stderr, encoding)
         returncode = ps.poll()
         if returncode:
             sys.stderr.write(stderr)
@@ -466,10 +494,17 @@

     :return: stdout
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
+
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     process = pipe_command(*args, **kwargs)
     stdout, stderr = process.communicate()
+    if encoding is not None:
+        stdout = _make_unicode(stdout, encoding)
+        stderr = _make_unicode(stderr, encoding)
     returncode = process.poll()
     if _capture_stderr and returncode:
         sys.stderr.write(stderr)
@@ -539,12 +574,22 @@

     :raises: ``CalledModuleError`` when module returns non-zero return code
     """
+    encoding = 'default'
+    if 'encoding' in kwargs:
+        encoding = kwargs['encoding']
     # TODO: should we delete it from kwargs?
     stdin = kwargs['stdin']
+    if encoding is None or encoding == 'default':
+        stdin = encode(stdin)
+    else:
+        stdin = encode(stdin, encoding=encoding)
     if _capture_stderr and 'stderr' not in kwargs.keys():
         kwargs['stderr'] = PIPE
     process = feed_command(*args, **kwargs)
     unused, stderr = process.communicate(stdin)
+    if encoding is not None:
+        unused = _make_unicode(unused, encoding)
+        stderr = _make_unicode(stderr, encoding)
     returncode = process.poll()
     if _capture_stderr and returncode:
         sys.stderr.write(stderr)
@@ -738,14 +783,15 @@
             break
         try:
             [var, val] = line.split(b'=', 1)
+            [var, val] = [decode(var), decode(val)]
         except:
             raise SyntaxError("invalid output from g.parser: %s" % line)

-        if var.startswith(b'flag_'):
+        if var.startswith('flag_'):
             flags[var[5:]] = bool(int(val))
-        elif var.startswith(b'opt_'):
+        elif var.startswith('opt_'):
             options[var[4:]] = val
-        elif var in [b'GRASS_OVERWRITE', b'GRASS_VERBOSE']:
+        elif var in ['GRASS_OVERWRITE', 'GRASS_VERBOSE']:
             os.environ[var] = val
         else:
             raise SyntaxError("invalid output from g.parser: %s" % line)
@@ -768,7 +814,7 @@
     "flags" are Python booleans.

     Overview table of parser standard options:
-    https://grass.osgeo.org/grass74/manuals/parser_standard_options.html
+    https://grass.osgeo.org/grass77/manuals/parser_standard_options.html
     """
     if not os.getenv("GISBASE"):
         print("You must be in GRASS GIS to run this program.", file=sys.stderr)
@@ -820,6 +866,30 @@
     os.mkdir(tmp)

     return tmp
+
+
+def tempname(length, lowercase=False):
+    """Generate a GRASS and SQL compliant random name starting with tmp_
+    followed by a random part of length "length"
+
+    :param int length: length of the random part of the name to generate
+    :param bool lowercase: use only lowercase characters to generate name
+    :returns: String with a random name of length "length" starting with a letter
+    :rtype: str
+
+    :Example:
+
+    >>> tempname(12)
+    'tmp_MxMa1kAS13s9'
+    """
+
+    chars = string.ascii_lowercase + string.digits
+    if not lowercase:
+        chars += string.ascii_uppercase
+    random_part = ''.join(random.choice(chars) for _ in range(length))
+    randomname = 'tmp_' + random_part
+
+    return randomname

 def _compare_projection(dic):
@@ -1270,7 +1340,7 @@

     :return: directory of mapsets/elements
     """
-    if isinstance(type, python_types.StringTypes) or len(type) == 1:
+    if isinstance(type, str) or len(type) == 1:
         types = [type]
         store_types = False
     else:

and to the remaining files just change file by open

/grass-7.4.3/scripts/db.py
/grass-7.4.3/scripts/v.db.addtable
/grass-7.4.3/scripts/v.dissolve
/grass-7.4.3/scripts/v.lines

note: file() is not supported in Python 3. Using open()

I will apply the changes now!

p/d: If a similar error arises, you already know where to start. Now I want that beer :smile: Ha!

[fjperini]

The globe support is almost finished, the option to use the old versions (osgearth 2.7) of osgearth openscenegraph apparently builds well.

But I found many things to change for the new versions (3.6.3). For that I will upload new formulas that includes the support for Qt5 necessary to build some libraries, for example:

/usr/local/opt/osgearth-qt5/lib/libosgEarthQt5.dylib

needed by

/PlugIns/qgis/libglobeplugin.dylib

Apparently I just need to correct an error (after many) so that the build is completed.

../src/plugins/globe/featuresource/qgsglobefeaturesource.h:33:134: error: non-virtual member function marked 'override' hides virtual member function
    osgEarth::Features::FeatureCursor *createFeatureCursor( const osgEarth::Symbology::Query &query = osgEarth::Symbology::Query() ) override;
                                                                                                                                     ^
/usr/local/opt/osgearth-qt5/include/osgEarthFeatures/FeatureSource:149:32: note: hidden overloaded virtual function 'osgEarth::Features::FeatureSource::createFeatureCursor' declared here: different number of parameters (2 vs 1)
        virtual FeatureCursor* createFeatureCursor(const Symbology::Query& query, ProgressCallback* progress) =0;
                               ^

I hope to solve it. Ha!

[fjperini]

@nickrobison As there are several changes for Python 3, it might be best to create a grass7-dev formula for QGIS 3 and use it momentarily until GRASS 7.8 / 8.0 arrives.

@luispuerto I'll give you the formula with the changes and the tests in QGIS 2 and QGIS3.

If something comes up in QGIS 2, we'll have to use grass7-dev.

[luispuerto]

Tell me which branch in your repo or which pull request I should use.

[fjperini]

@luispuerto You can already try it!

[luispuerto]

I just tested it v.dissolve in QGIS3&2 and it works perfectly in both. 👍

Again, great work!

[fjperini]

Once @nickrobison merge the changes we close this. Ha! Thanks @luispuerto!