search

OSGeo / homebrew-osgeo4mac

Mac homebrew tap for maintaining a stable work environment for the OSGeo.org geospatial toolset
https://git.io/fhh3X
BSD 3-Clause "New" or "Revised" License
363 stars 111 forks source link

QGIS3/GRASS: error while executing a script #452

Closed fjperini closed 5 years ago

fjperini commented 6 years ago

Problem reported by @luisspuerto 

Your PYTHONPATH points to a site-packages dir for Python 3.x but you are running Python 2.x!

PYTHONPATH is currently: "/usr/local/opt/gdal2-python/lib/python3.7/site-packages:/usr/local/opt/qgis3/lib/python3.7/site-packages:/usr/local/opt/qgis3/libexec/python/lib/python3.7/site-packages:/usr/local/lib/python3.7/site-packages"

You should `unset PYTHONPATH` to fix this.
luispuerto commented 6 years ago

screen shot 2018-08-15 at 08 58 17

I've built Grass7 form source using this formula and rebuilt qgis3 and the problem is still there.

luispuerto commented 6 years ago

Where should I change that you mention on https://github.com/OSGeo/homebrew-osgeo4mac/issues/436#issuecomment-412962948, sorry?

fjperini commented 6 years ago

@luisspuerto I think that default_gui = "text" it seems to be a GRASS variable and #!/usr/bin/env python3 in each GRASS script. We can define that variable in the formula.

GRASS startup program GRASS variables and environment variables

luispuerto commented 6 years ago

This isn't fixed by the last merge #446, is it?

fjperini commented 6 years ago

@luisspuerto I have not reviewed this problem. Can you test the above? I think another option could be to add PYTHONPATH in the script: g.gisenv set="VARIABLE=VALUE" Example: GRASS_GUI=text, GRASS_PYTHON=3 Will work this PYTHONPATH=/usr/local/lib/python2.7/site-packages?

luispuerto commented 6 years ago

After the problems I report in #456, I've tried to run a grass script and I get the following: 

sys.stdout.write("%s\n" % grass_version)

^

TabError: inconsistent use of tabs and spaces in indentation

File "/usr/local/Cellar/grass7/7.4.1/libexec/bin/grass74", line 1693

sys.stdout.write("%s\n" % grass_version)

^

TabError: inconsistent use of tabs and spaces in indentation

Execution completed in 0.69 seconds

Here the full output.

screen shot 2018-08-16 at 21 52 52

luispuerto commented 6 years ago

By the way after the rebuilding of qgis3 —I don't know how many times— I've noticed the following lines... 

NOTE: Your current PATH and PYTHONPATH environment variables are honored
      when launching via the wrapper script, while launching QGIS.app
      bundle they are not.

For standalone Python3 development, set the following environment variable:
  export PYTHONPATH=/usr/local/opt/qgis3/lib/python3.7/site-packages:/usr/local/opt/gdal2-python/lib/python3.7/site-packages:/usr/local/opt/qgis3/libexec/vendor/lib/python3.7/site-packages:$PYTHONPATH
nickrobison commented 6 years ago

@luisspuerto The tab/space issue is an upstream problem in grass, I think it was first reported in #228. I'll try and create a patch and upgrade the grass formula.

luispuerto commented 6 years ago

In qgis2 everything works! thankgod!

luispuerto commented 6 years ago

I just updated to QGIS3 and QGIS2. In QGIS2 everything seems to work perfectly, but in QGIS3 the error is still there... 😞

fjperini commented 6 years ago

@luisspuerto, the problem is in GRASS, @nickrobison will create a patch to solve it.

fjperini commented 5 years ago

@luispuerto, is this still happening? If it is solved we close this issue

luispuerto commented 5 years ago

@fjperini No... sadly isn't working... but I think it's different error now: 

Processing algorithm…
Algorithm 'v.buffer' starting…
Input parameters:
{ '-c' : False, '-s' : False, '-t' : False, 'GRASS_MIN_AREA_PARAMETER' : 0.0001, 'GRASS_OUTPUT_TYPE_PARAMETER' : 0, 'GRASS_REGION_PARAMETER' : None, 'GRASS_SNAP_TOLERANCE_PARAMETER' : -1, 'GRASS_VECTOR_DSCO' : '', 'GRASS_VECTOR_LCO' : '', 'angle' : 0, 'cats' : '', 'column' : None, 'distance' : 1000, 'input' : '/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data_modified/Geomorphology/grens_waterschap.shp', 'minordistance' : 10, 'output' : '/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp', 'scale' : 1, 'tolerance' : 0.01, 'type' : [0,1,4], 'where' : '' }

g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
v.in.ogr min_area=0.0001 snap=-1.0 input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data_modified/Geomorphology/grens_waterschap.shp" output="vector_5bdd689130a3b10" --overwrite -o
g.region n=503941.924 s=461701.483 e=269799.23 w=222306.806
v.buffer input=vector_5bdd689130a3b10 type="point,line,area" distance=1000 minordistance=10 angle=0 scale=1 tolerance=0.01 output=output327607dda92d44cf87efbd523b81e66e --overwrite
v.out.ogr -c type="auto" input="output327607dda92d44cf87efbd523b81e66e" output="/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp" format="ESRI_Shapefile" --overwrite
File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693
sys.stdout.write("%s\n" % grass_version)
^
TabError: inconsistent use of tabs and spaces in indentation
File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693
sys.stdout.write("%s\n" % grass_version)
^
TabError: inconsistent use of tabs and spaces in indentation
Execution completed in 0.66 seconds
Results:
{'output': <QgsProcessingOutputLayerDefinition {'sink':/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
The following layers were not correctly generated.<ul><li>/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_421a71786e17438da2dd7f2f345db0ea/1170294241af4750a49e0ae27b196c73/output.shp</li></ul>You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

grass log window: 

2018-11-03T11:24:03     INFO    processInputs end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.in.ogr min_area=0.0001 snap=-1.0 input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geomorphology/grens_waterschap.shp" output="vector_5bdd69339c08d2" --overwrite -o', 'g.region n=503941.924 s=461701.483 e=269799.23 w=222306.806']
2018-11-03T11:24:03     INFO    processCommands end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.in.ogr min_area=0.0001 snap=-1.0 input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geomorphology/grens_waterschap.shp" output="vector_5bdd69339c08d2" --overwrite -o', 'g.region n=503941.924 s=461701.483 e=269799.23 w=222306.806', 'v.buffer input=vector_5bdd69339c08d2 type="point,line,area" distance=1000 minordistance=10 angle=0 scale=1 tolerance=0.01 output=output32e22da1751f43b3a65a38768a715e77 --overwrite']

In QGIS2 works.

I also tested other script in grass I the same/similar error.

luispuerto commented 5 years ago

PS/ by the way all the others works perfectly, including saga, que didn't work before.

orfeo6 disappeared again, but I guess you still working in implementation, don't you?

luispuerto commented 5 years ago

I just install QGIS 3.4.1 and the error still happening... I guess it's a QGIS issue.

luispuerto commented 5 years ago

Additional info in the geoprocessing log: 

2018-11-03T17:29:07     INFO    GRASS GIS 7 execution commands
             g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
             v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingpleistoceen.shp" output="vector_5bddbec33f9fb2" --overwrite -o
             g.region n=500000.0 s=461702.4058968832 e=269723.15625 w=230968.3125
             v.buffer input=vector_5bddbec33f9fb2 type="point,line,area" distance=100 angle=0 scale=1 tolerance=0.01 output=outpute32856ce1f2042d58ece8ab94ba33b2b --overwrite
             v.out.ogr -c type="auto" input="outpute32856ce1f2042d58ece8ab94ba33b2b" output="/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_0cb9d254b42a4b6b8ad15f38c86c2294/2416a9a64bf74f0a94a8fb03288f1edb/output.shp" format="ESRI_Shapefile" --overwrite
2018-11-03T17:29:07     INFO    GRASS GIS 7 execution console output
              File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693

              sys.stdout.write("%s\n" % grass_version)

              ^

             TabError: inconsistent use of tabs and spaces in indentation

             File "/usr/local/Cellar/grass7/7.4.2/libexec/bin/grass74", line 1693
             sys.stdout.write("%s\n" % grass_version)
             ^
             TabError: inconsistent use of tabs and spaces in indentation

TabError: inconsistent use of tabs and spaces in indentation

Well... I don't know. I don't know why this happens in QGIS3 but not in 2.

fjperini commented 5 years ago

@luispuerto This commented @nickrobison in #228

@bolliger32 That's a really strange error, I can see the incorrect indenting in my grass74 file, but no error in thrown.

Which version of python are you using?

If you remove the tabs at the beginning on the line, and ensure that the indentation is lined up correctly, does the error still occur?

Regarding the problem with PYTHON, I noticed that in src/plugins/grass/qgsgrassshell.cpp:

env << QStringLiteral( "GRASS_PYTHON=python" );

It is very likely that you are calling python3 by default when grass is configured to use python2

Another thing you can do to make sure:

export GRASS_PYTHON=$(brew --prefix)/opt/python2/bin/python2

GRASS addons may fail due to python3 incompatibility. If that is tha case you can change the shebang a the beginning of the script to enforce python2 usage.

#!/usr/bin/env python

Should be changed into

#!/usr/bin/env python2

Try this and then tell me how it turned out!

p/d: Yes, I am working on the implementation of orfeo6, I applied several changes with respect to #461 #462.

luispuerto commented 5 years ago

@fjperini I can try to change python 3 for 2 for grass, but I really don't know how to do it inside the QGIS3 app. I really understand about the shebang, but I'm not scripting, I'm trying to use just the v.buffer script form the grass toolbox in QIGS3.

fjperini commented 5 years ago

@luispuerto I get it! I will sit with this.

fjperini commented 5 years ago

@luispuerto @nickrobison It seems that grass7 works in qgis3! :tada:

I had to apply several changes. Other errors, I solved it with:

For the error "ValueError: unknown locale: UTF-8" in /qgis3/libexec/vendor/lib/python3.7/locale.py

I think this error occurs when QGIS starts with another LANG other than en_US.UTF-8.

Check what kind of local variables:

$ ls /usr/share/locale/

I added it to /qgis3/3.4.2/bin/qgis3 (only if it is executed from console), it is better to place it in ~/.bash_profile

export LANG=en_US.UTF-8
export LC_CTYPE=en_US.UTF-8
export LC_ALL=en_US.UTF-8
$ source  ~/.bash_profile
$ python -c 'import locale; print(locale.getdefaultlocale());'
('en_US', 'UTF-8')

UPDATE:

It is not enough to put it in ~/.bash_profile and /qgis3/3.4.2/bin/qgis3, it is better to put it in /grass7/7.4.3/libexec/bin/grass74 for a definitive solution.

For the error TypeError("str expected, not %s" % type(value).name) / TypeError: str expected, not bytes in /grass7/7.4.3/libexec/bin/grass74

I had to change the next line

os.environ['MANPATH'] = sys_man_path

to 

  os.environ['MANPATH'] = str(sys_man_path)

p/d: I hope it works, at least I was not presented with the errors that were previously.

luispuerto commented 5 years ago

😱 this is grasstantic I can wait to test it!

fjperini commented 5 years ago

It is! First we will have to merge qt5-webkit, then I apply the changes for grass7 and qgis3.

Grasstantic :smile: Well said!

fjperini commented 5 years ago

As mentioned @msandifo (see), a solution:

a temp fix is to replace tabs with spaces by editing line 1693 in /usr/local/Cellar/grass7/7.4.1/libexec/bin/grass74

which can be applied by @nickrobison #549 :

# Patch grass.py to remove bad tab at line 1693 (and insert 12 spaces)
# Needs to be pushed upstream.
inreplace "lib/init/grass.py", "\t\t\t", "

or, with the patch that was to apply:

--- a/lib/init/grass.py
+++ b/lib/init/grass.py
@@ -1668,7 +1668,7 @@ def print_params():
             filerev.close()
             sys.stdout.write("%s\n" % val[0].split(':')[1].rstrip('$"\n').strip())
         elif arg == 'version':
-           sys.stdout.write("%s\n" % grass_version)
+            sys.stdout.write("%s\n" % grass_version)
         else:
             message(_("Parameter <%s> not supported") % arg)

Once merged #549, I will apply the other changes.

@luispuerto When the bottle for qt5-webkit is ready, I to apply changes in qgis3 and improve integration with GRASS, R and ORFEO.

fjperini commented 5 years ago

@luispuerto I'm about to upload the other changes!

fjperini commented 5 years ago

@luispuerto You can try the changes, in the course of the day I apply to qgis3.

qgis3

They are all integrated and functioning.

luispuerto commented 5 years ago

Great! @fjperini are the changes related to the last 3 pull request? I can download them and merge locally in a branch and test them

luispuerto commented 5 years ago

I need to switch back to qt5.12 rebuild qt-WebKit and then rebuild QGIS3 with r and Orfeo. Is that correct?

fjperini commented 5 years ago

When you apply the changes in qgis3, you will need qt5-webkit 5.12. You can try grass7.

luispuerto commented 5 years ago

But are you going to upload more pull requests? Or is everything there?

luispuerto commented 5 years ago

merging locally: #551 #550 #547 and #450 Is that correct?

luispuerto commented 5 years ago

after I rebuilt built qt5-webkit and grass7:

grass7 buffer script log: 

Processing algorithm…
Algorithm 'v.buffer' starting…
Input parameters:
{ '-c' : False, '-s' : False, '-t' : False, 'GRASS_MIN_AREA_PARAMETER' : 0.0001, 'GRASS_OUTPUT_TYPE_PARAMETER' : 0, 'GRASS_REGION_PARAMETER' : None, 'GRASS_SNAP_TOLERANCE_PARAMETER' : -1, 'GRASS_VECTOR_DSCO' : '', 'GRASS_VECTOR_LCO' : '', 'angle' : 0, 'cats' : '', 'column' : None, 'distance' : 100, 'input' : '/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp', 'minordistance' : None, 'output' : '/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/64e4252cf30d450c91ce2eb65d0d5313/output.gpkg', 'scale' : 1, 'tolerance' : 0.01, 'type' : [0,1,4], 'where' : '' }

Attempting to use v.external for direct layer read
g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp" output="vector_5c0cef99281d93" --overwrite -o
g.region n=500000.0 s=463205.4616311212 e=268530.3125 w=231382.5
v.buffer input=vector_5c0cef99281d93 type="point,line,area" distance=100 angle=0 scale=1 tolerance=0.01 output=output3e654eb3186843f59a04a481ceba2f35 --overwrite
v.out.ogr -c type="auto" input="output3e654eb3186843f59a04a481ceba2f35" output="/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/64e4252cf30d450c91ce2eb65d0d5313/output.gpkg" format="GPKG" --overwrite
Cleaning up temporary files...
access: No such file or directory
ERROR: LOCATION </Users/lpuerto/Documents/App Data/grass/nc_spm_08_grass7> not available
Starting GRASS GIS...
Executing </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/grassdata/grass_batch_job.sh> ...
Default region was updated to the new projection, but if you have multiple mapsets `g.region -d` should be run in each to update the region from the default
Projection information updated
Over-riding projection check
Building topology for vector map <vector_5c0cef99281d93@PERMANENT>...
Using external data format 'ESRI Shapefile' (feature type 'polygon')
Registering primitives...
186 primitives registered
18143 vertices registered
Number of nodes: 100
Number of primitives: 186
Number of points: 0
Number of lines: 0
Number of boundaries: 104
Number of centroids: 82
Number of areas: 104
Number of isles: 104
v.external complete. Link to vector map <vector_5c0cef99281d93> created.
WARNING: Option 'tolerance' is not available with GEOS buffering
Buffering areas...
/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/grassdata/grass_batch_job.sh: line 5: 20945 Segmentation fault: 11 v.buffer input=vector_5c0cef99281d93 type="point,line,area" distance=100 angle=0 scale=1 tolerance=0.01 output=output3e654eb3186843f59a04a481ceba2f35 --overwrite

GRASS command crashed :( Try a different set of input parameters and consult the GRASS algorithm manual for more information.

Suggest disabling the experimental "use v.external" option from the Processing GRASS Provider options.

WARNING: Unable to open vector map <output3e654eb3186843f59a04a481ceba2f35@PERMANENT> on level 2. Try to rebuild vector topology with v.build.
ERROR: Unable to open vector map <output3e654eb3186843f59a04a481ceba2f35>
Execution of </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/grassdata/grass_batch_job.sh> finished.
Cleaning up temporary files...
Cleaning up temporary files...
access: No such file or directory
ERROR: LOCATION </Users/lpuerto/Documents/App Data/grass/nc_spm_08_grass7> not available
Starting GRASS GIS...
Executing </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/grassdata/grass_batch_job.sh> ...
WARNING: Unable to open vector map <output3e654eb3186843f59a04a481ceba2f35@PERMANENT> on level 2. Try to rebuild vector topology with v.build.
ERROR: Unable to open vector map <output3e654eb3186843f59a04a481ceba2f35>
Execution of </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/grassdata/grass_batch_job.sh> finished.
Cleaning up temporary files...
Execution completed in 1.82 seconds
Results:
{'output': <QgsProcessingOutputLayerDefinition {'sink':/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/64e4252cf30d450c91ce2eb65d0d5313/output.gpkg, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
The following layers were not correctly generated.<ul><li>/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/64e4252cf30d450c91ce2eb65d0d5313/output.gpkg</li></ul>You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

geoprocessing log: 

2018-12-09T12:33:15     INFO    GRASS GIS 7 execution commands
             g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
             v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp" output="vector_5c0cef6b92a632" --overwrite -o
             g.region n=500000.0 s=463205.4616311212 e=268530.3125 w=231382.5
             v.buffer input=vector_5c0cef6b92a632 type="point,line,area" angle=0 scale=1 tolerance=0.01 output=outputd6740eabfe49453896b43d760b1755e8 --overwrite
             v.out.ogr -c type="auto" input="outputd6740eabfe49453896b43d760b1755e8" output="/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/eba3c974748647278ed5f966415ca7fd/output.gpkg" format="GPKG" --overwrite
2018-12-09T12:33:17     INFO    GRASS GIS 7 execution console output
             Cleaning up temporary files...

             access: No such file or directory

             ERROR: LOCATION not available

             Starting GRASS GIS...

             Executing ...

             Default region was updated to the new projection, but if you have multiple mapsets `g.region -d` should be run in each to update the region from the default

             Projection information updated

             Over-riding projection check

             Building topology for vector map ...

             Using external data format 'ESRI Shapefile' (feature type 'polygon')

             Registering primitives...

             186 primitives registered

             18143 vertices registered

             Number of nodes: 100

             Number of primitives: 186

             Number of points: 0

             Number of lines: 0

             Number of boundaries: 104

             Number of centroids: 82

             Number of areas: 104

             Number of isles: 104

             v.external complete. Link to vector map created.

             ERROR: Select a buffer distance/minordistance/angle or column, but not both.

             ERROR: Vector map not found

             Execution of finished.

             Cleaning up temporary files...

             Cleaning up temporary files...
             access: No such file or directory
             ERROR: LOCATION not available
             Starting GRASS GIS...
             Executing ...
             ERROR: Vector map not found
             Execution of finished.
             Cleaning up temporary files...
2018-12-09T12:34:01     INFO    GRASS GIS 7 execution commands
             g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
             v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp" output="vector_5c0cef99281d93" --overwrite -o
             g.region n=500000.0 s=463205.4616311212 e=268530.3125 w=231382.5
             v.buffer input=vector_5c0cef99281d93 type="point,line,area" distance=100 angle=0 scale=1 tolerance=0.01 output=output3e654eb3186843f59a04a481ceba2f35 --overwrite
             v.out.ogr -c type="auto" input="output3e654eb3186843f59a04a481ceba2f35" output="/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/64e4252cf30d450c91ce2eb65d0d5313/output.gpkg" format="GPKG" --overwrite
2018-12-09T12:34:02     INFO    GRASS GIS 7 execution console output
             Cleaning up temporary files...

             access: No such file or directory

             ERROR: LOCATION not available

             Starting GRASS GIS...

             Executing ...

             Default region was updated to the new projection, but if you have multiple mapsets `g.region -d` should be run in each to update the region from the default

             Projection information updated

             Over-riding projection check

             Building topology for vector map ...

             Using external data format 'ESRI Shapefile' (feature type 'polygon')

             Registering primitives...

             186 primitives registered

             18143 vertices registered

             Number of nodes: 100

             Number of primitives: 186

             Number of points: 0

             Number of lines: 0

             Number of boundaries: 104

             Number of centroids: 82

             Number of areas: 104

             Number of isles: 104

             v.external complete. Link to vector map created.

             WARNING: Option 'tolerance' is not available with GEOS buffering

             Buffering areas...

             /private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_e79b4508904e45f592f249d0e5ac4fe0/grassdata/grass_batch_job.sh: line 5: 20945 Segmentation fault: 11 v.buffer input=vector_5c0cef99281d93 type="point,line,area" distance=100 angle=0 scale=1 tolerance=0.01 output=output3e654eb3186843f59a04a481ceba2f35 --overwrite

             WARNING: Unable to open vector map on level 2. Try to rebuild vector topology with v.build.

             ERROR: Unable to open vector map 

             Execution of finished.

             Cleaning up temporary files...

             Cleaning up temporary files...
             access: No such file or directory
             ERROR: LOCATION not available
             Starting GRASS GIS...
             Executing ...
             WARNING: Unable to open vector map on level 2. Try to rebuild vector topology with v.build.
             ERROR: Unable to open vector map 
             Execution of finished.
             Cleaning up temporary files...

grass7 log window

2018-12-09T12:33:15     INFO    processInputs end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp" output="vector_5c0cef6b92a632" --overwrite -o', 'g.region n=500000.0 s=463205.4616311212 e=268530.3125 w=231382.5']
2018-12-09T12:33:15     INFO    processCommands end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp" output="vector_5c0cef6b92a632" --overwrite -o', 'g.region n=500000.0 s=463205.4616311212 e=268530.3125 w=231382.5', 'v.buffer input=vector_5c0cef6b92a632 type="point,line,area" angle=0 scale=1 tolerance=0.01 output=outputd6740eabfe49453896b43d760b1755e8 --overwrite']
2018-12-09T12:34:01     INFO    processInputs end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp" output="vector_5c0cef99281d93" --overwrite -o', 'g.region n=500000.0 s=463205.4616311212 e=268530.3125 w=231382.5']
2018-12-09T12:34:01     INFO    processCommands end. Commands: ['g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"', 'v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/bedekkingholoceen.shp" output="vector_5c0cef99281d93" --overwrite -o', 'g.region n=500000.0 s=463205.4616311212 e=268530.3125 w=231382.5', 'v.buffer input=vector_5c0cef99281d93 type="point,line,area" distance=100 angle=0 scale=1 tolerance=0.01 output=output3e654eb3186843f59a04a481ceba2f35 --overwrite']
luispuerto commented 5 years ago

by the way... do I need to do anything additional to get R? I build --with-r but it isn't showing there. screen shot 2018-12-09 at 12 43 03

fjperini commented 5 years ago

@luispuerto The R and ORFEO, are integrated with the new changes in qgis3. I apply them later.

Check this error you got in GRASS.

access: No such file or directory
ERROR: LOCATION </Users/lpuerto/Documents/App Data/grass/nc_spm_08_grass7> not available

The good thing is that you can run it, I'll see what that error is.

p/d: the pulls you used are fine, and you added: Add libagg support to orfeo6 :+1:

luispuerto commented 5 years ago

Hey!!

Now it work 🎉 but two things.

that error was a config error, probably due to previous tweaking trying to make it work.

I delete the config: 

$ rm -r ~/.grass*

Then I also have to disable the v.external option from qgis3 processing options → grass screen shot 2018-12-09 at 19 49 46

I can't see that option in QGIS2... but in qgis2 the same script is faster than in qgis3. I don't know if everything is exactly the same, since I haven't thoroughly checked.

BUT....

🎉 IT WORKS! 🎉

(I owe you a beer @fjperini !)

fjperini commented 5 years ago

@luispuerto We finally beat GRASS. At some point you'll have to give me that beer. Ha!

Yes, it is necessary to delete the previous profile to avoid inconveniences.

Keep testing and then tell me.

luispuerto commented 5 years ago

@fjperini where do you live? if you live in Spain perhaps we have a chance this Christmas

fjperini commented 5 years ago

@luispuerto We are far, I to a sea. Ha! But soon I will be walking through the old continent.

The integration of R and ORFEO is completed. It will only be necessary to enable them from Manage and Install Plugins and and ready.

I was workining to configure the default folders and activate the plugin when it is enabled:

Manage and Install Plugins -> Installed -> Plugin name (click its checkbox)

I correct the --with-globe integration and apply the changes. With that would be all the corrections to supports that were by default.

nickrobison commented 5 years ago

@fjperini @luispuerto I would be more than happy to buy you each a beer this Christmas, if you let me know where to send the money! One for @luispuerto for finding all the errors, and one for @fjperini for fixing them!

luispuerto commented 5 years ago

@nickrobison as far I see you are based in Seattle, perhaps I will go around the great Pacific North West after the summer for a couple of months. 🍻 ?

PS/ You have a lot of trees around there 🌲 🌲 🌲 🌲 🌲 🌲 🌲 🌲 I'm a forester.

fjperini commented 5 years ago

@luispuerto Did you verify that GRASS works 100%? To confirm that it was resolved and close this issue.

@nickrobison With the R, ORFEO and GLOBE resolved, it will have to be a full beer box. What do we do, do we demand it? :smile:

We will continue the talk here #431 for the integration of ORFEO. I finish testing the --with-globe and apply the changes.

p/p: It would be great to be able to meet!

luispuerto commented 5 years ago

@fjperini I run a couple of scripts more... but they aren't working.

this is the log while running v.dissolve in a layer. In QGIS2 work the same script... 

Processing algorithm…
Algorithm 'v.dissolve' starting…
Input parameters:
{ 'GRASS_MIN_AREA_PARAMETER' : 0.0001, 'GRASS_OUTPUT_TYPE_PARAMETER' : 0, 'GRASS_REGION_PARAMETER' : None, 'GRASS_SNAP_TOLERANCE_PARAMETER' : -1, 'GRASS_VECTOR_DSCO' : '', 'GRASS_VECTOR_LCO' : '', 'column' : 'FORMATIE', 'input' : '/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/gestuwd.shp', 'output' : '/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg' }

Attempting to use v.external for direct layer read
g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
v.external input="/Users/lpuerto/Documents/Carto/iGEON/Introduction_NRM/06_Case_Study/Data/Geology/gestuwd.shp" output="vector_5c0e85082c4be11" --overwrite -o
g.region n=500000.0 s=471782.28125 e=266037.78125 w=231537.2496543302
v.dissolve input=vector_5c0e85082c4be11 column=FORMATIE output=output2ebe2052cd0c40dcb7ad19ca23723de2 --overwrite
v.out.ogr -c type="auto" input="output2ebe2052cd0c40dcb7ad19ca23723de2" output="/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg" format="GPKG" --overwrite
Starting GRASS GIS...
Executing </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> ...
Default region was updated to the new projection, but if you have multiple mapsets `g.region -d` should be run in each to update the region from the default
Projection information updated
Over-riding projection check
Building topology for vector map <vector_5c0e85082c4be11@PERMANENT>...
Using external data format 'ESRI Shapefile' (feature type 'polygon')
Registering primitives...
262 primitives registered
18997 vertices registered
Number of nodes: 120
Number of primitives: 262
Number of points: 0
Number of lines: 0
Number of boundaries: 141
Number of centroids: 121
Number of areas: 141
Number of isles: 141
v.external complete. Link to vector map <vector_5c0e85082c4be11> created.
Traceback (most recent call last):
File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/scripts/v.dissolve", line 110, in <module>
options, flags = grass.parser()
File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py", line 798, in parser
return _parse_opts(lines[1:])
File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py", line 749, in _parse_opts
os.environ[var] = val
File "/usr/local/opt/qgis3/libexec/vendor/bin/../lib/python3.7/os.py", line 682, in __setitem__
key = self.encodekey(key)
File "/usr/local/opt/qgis3/libexec/vendor/bin/../lib/python3.7/os.py", line 753, in encode
raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not bytes
ERROR: Vector map <output2ebe2052cd0c40dcb7ad19ca23723de2> not found
Execution of </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> finished.
Cleaning up temporary files...
Starting GRASS GIS...
Executing </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> ...
ERROR: Vector map <output2ebe2052cd0c40dcb7ad19ca23723de2> not found
Execution of </private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/grassdata/grass_batch_job.sh> finished.
Cleaning up temporary files...
Execution completed in 1.85 seconds
Results:
{'output': <QgsProcessingOutputLayerDefinition {'sink':/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
The following layers were not correctly generated.<ul><li>/private/var/folders/wf/41gjf2mx7m7fmvfd8dr22_5h0000gn/T/processing_297f7f14064f4dd88b41ac9adb863ef7/25849cea13f848f3804011c7ba051148/output.gpkg</li></ul>You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

This time seems something related with python (again).

luispuerto commented 5 years ago

by the way... I just tried the vector buffer from gdal in QGIS2 and it's not working, but in QGIS3 it's...

I just testing.... but I really convinced that we are going to need to have two qgis for quite some time.

fjperini commented 5 years ago

@luispuerto You have that error:

TypeError: str expected, not bytes
ERROR: Vector map <output2ebe2052cd0c40dcb7ad19ca23723de2> not found`

I'll check.

TypeError: str expected, not bytes I solved that with: 

os.environ['MANPATH'] = str(sys_man_path)

There may be another line with os.environ to change.

Verify that the change is applied in /grass7/7.4.3/libexec/bin/grass74

luispuerto commented 5 years ago

my set.paths function: 

def set_paths(grass_config_dir):
    # addons (path)
    addon_path = os.getenv('GRASS_ADDON_PATH')
    if addon_path:
        for path in addon_path.split(os.pathsep):
            path_prepend(addon_path, 'PATH')

    # addons (base)
    addon_base = os.getenv('GRASS_ADDON_BASE')
    if not addon_base:
        addon_base = os.path.join(grass_config_dir, 'addons')
        os.environ['GRASS_ADDON_BASE'] = addon_base
    if not windows:
        path_prepend(os.path.join(addon_base, 'scripts'), 'PATH')
    path_prepend(os.path.join(addon_base, 'bin'), 'PATH')

    # standard installation
    if not windows:
        path_prepend(gpath('scripts'), 'PATH')
    path_prepend(gpath('bin'), 'PATH')

    # Set PYTHONPATH to find GRASS Python modules
    if os.path.exists(gpath('etc', 'python')):
        path_prepend(gpath('etc', 'python'), 'PYTHONPATH')

    # set path for the GRASS man pages
    grass_man_path = gpath('docs', 'man')
    addons_man_path = os.path.join(addon_base, 'docs', 'man')
    man_path = os.getenv('MANPATH')
    sys_man_path = None
    if man_path:
        path_prepend(addons_man_path, 'MANPATH')
        path_prepend(grass_man_path, 'MANPATH')
    else:
        try:
            nul = open(os.devnull, 'w')
            p = Popen(['manpath'], stdout=subprocess.PIPE, stderr=nul)
            nul.close()
            s = p.stdout.read()
            p.wait()
            sys_man_path = s.strip()
        except:
            pass

        if sys_man_path:
            # to_text_string disabled, see https://trac.osgeo.org/grass/ticket/3508
            # os.environ['MANPATH'] = to_text_string(sys_man_path)
            os.environ['MANPATH'] = str(sys_man_path)
            path_prepend(addons_man_path, 'MANPATH')
            path_prepend(grass_man_path, 'MANPATH')
        else:
            # to_text_string disabled, see https://trac.osgeo.org/grass/ticket/3508
            # os.environ['MANPATH'] = to_text_string(addons_man_path)
            os.environ['MANPATH'] = addons_man_path
            path_prepend(grass_man_path, 'MANPATH')

    # Set LD_LIBRARY_PATH (etc) to find GRASS shared libraries
    # this works for subprocesses but won't affect the current process
    path_prepend(gpath("lib"), ld_library_path_var)
fjperini commented 5 years ago

@luispuerto I checked that with v.buffer everything works fine, doing v.dissolve the error TypeError exits again.

It seems that now we have to correct the file: /usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py. Since it is the last (of GRASS) to run.

fjperini commented 5 years ago

@luispuerto I'm testing others v.* and they are working well.

luispuerto commented 5 years ago

@fjperini you are right... what an eye have! I swear that v.dissolve want the second script I tried after v.buffer. I've been trying to use some other scripts —from r.* or i.* toolboxes— but I don't have anything at hand right now to test them —that I can say the script it's working properly.

fjperini commented 5 years ago

The other one that I found is: v.to.lines

I will try to solve the problem in core.py if that is the correct file. Ha!

It would be necessary to report them to the developers.

They are upstream problems anyway.

Everything else seems to work well.

luispuerto commented 5 years ago

Great 🎈

But if it’s an upstream problem why do they work on QGIS2?

Sent with GitHawk

fjperini commented 5 years ago

@luispuerto It may be by the python version:

File "/usr/local/Cellar/grass7/7.4.3_2/grass-7.4.3/etc/python/grass/script/core.py"
...
File "/usr/local/opt/qgis3/libexec/vendor/bin/../lib/python3.7/os.py"

QGIS2 uses Python 2.

If you see in the comment

# to_text_string disabled, see https://trac.osgeo.org/grass/ticket/3508
# os.environ['MANPATH'] = to_text_string(sys_man_path)
os.environ['MANPATH'] = str(sys_man_path)

for example for the first case: v.buffer.

There are things that change and may need to be adjusted from version to version. We just have to locate that os.environ we have to apply str to solve v.dissolve and v.to.lines and done.