search

qgis / QGIS-Mac-Packager

Scripts for Official QGIS MacOS Packages
https://www.qgis.org
GNU General Public License v2.0
54 stars 21 forks source link

The Raster: Merge tool fails with ModuleNotFoundError: No module named '_gdal' #102

Closed thayeray closed 3 years ago

thayeray commented 3 years ago

The Raster: Merge tool, which uses gdal_merge.py is not working. It gives the following error: gdal_merge_processing_error.txt

This seems to be a problem only with the gdal python scripts. I have tested several of the C binaries from the command line and they seem to be working.

thayeray commented 3 years ago

I neglected to mention that this is for 3.16.0.

PeterPetrik commented 3 years ago

can you please retest with qgis_pr_final-3_16_1_20201130_063613.dmg ? also have you moved your QGIS installation? any other messages in the QGIS processing log?

thayeray commented 3 years ago

Hi Peter, happy to do that but I can not find the .dmg, I only see a .log and .png. Also those have a different number: 043516 not 063613. Am I looking in the wrong place? https://qgis.org/downloads/macos/pr/

On Monday, November 30, 2020, 07:11:00 AM EST, Peter Petrik <notifications@github.com> wrote:

can you please retest with qgis_pr_final-3_16_1_20201130_063613.dmg ? also have you moved your QGIS installation? any other messages in the QGIS processing log?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

thayeray commented 3 years ago

Hi Peter, Sorry for the extra message, but it looks like the compilation of PR (3.16.1) has been failing since November 13th. The last version that compiled successfully was October 27th, that was 3.16.0, which appears to be the version that I am using. I reinstalled the 3.16.0 and did not rename it or move it, but I am still getting the same error. I also switched my User Profile with the same effect. Attached below is the full processing log. -T

2020-11-30T08:46:12     INFO    Loading OTB '6.6.0'.

2020-11-30T08:46:13     WARNING    Duplicate parameter coordinates registered for alg v.net.visibility

2020-11-30T08:47:21     INFO    gdal_merge.py -ot Int32 -of GTiff -o /private/var/folders/n9/d8sh13b51r14rb00ncq5lhd00000gn/T/processing_cekQEZ/e5ed1369ac0a4c6aa465cde0a12c00b6/OUTPUT.tif --optfile /private/var/folders/n9/d8sh13b51r14rb00ncq5lhd00000gn/T/processing_cekQEZ/ab2b456dd0fb494aa8560199e1e5480e/mergeInputFiles.txt

2020-11-30T08:47:22     INFO    GDAL execution console output
             Traceback (most recent call last):
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/__init__.py", line 18, in swig_import_helper
             
              return importlib.import_module(mname)
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/importlib/__init__.py", line 127, in import_module
             
              return _bootstrap._gcd_import(name[level:], package, level)
             
              File "", line 1006, in _gcd_import
             
              File "", line 983, in _find_and_load
             
              File "", line 967, in _find_and_load_unlocked
             
              File "", line 670, in _load_unlocked
             
              File "", line 583, in module_from_spec
             
              File "", line 1043, in create_module
             
              File "", line 219, in _call_with_frames_removed
             
             ImportError: dlopen(/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/_gdal.cpython-37m-darwin.so, 2): Symbol not found: _curl_mime_addpart
             
              Referenced from: /Applications/QGIS.app/Contents/MacOS/bin/../lib/libgdal.27.dylib
             
              Expected in: /usr/lib/libcurl.4.dylib
             
              in /Applications/QGIS.app/Contents/MacOS/bin/../lib/libgdal.27.dylib
             
             
             
             During handling of the above exception, another exception occurred:
             
             
             
             Traceback (most recent call last):
             
              File "/Applications/QGIS.app/Contents/MacOS/bin/gdal_merge.py", line 4, in 
             
              __import__('pkg_resources').run_script('GDAL==3.1.2', 'gdal_merge.py')
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 667, in run_script
             
              self.require(requires)[0].run_script(script_name, ns)
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1464, in run_script
             
              exec(code, namespace, namespace)
             
              File "/Applications/QGIS.app/Contents/Resources/python/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/EGG-INFO/scripts/gdal_merge.py", line 38, in 
             
              from osgeo import gdal
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/__init__.py", line 32, in 
             
              _gdal = swig_import_helper()
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/__init__.py", line 31, in swig_import_helper
             
              return importlib.import_module('_gdal')
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/importlib/__init__.py", line 127, in import_module
             
              return _bootstrap._gcd_import(name[level:], package, level)
             
             ModuleNotFoundError: No module named '_gdal'
             

2020-11-30T08:54:03     INFO    gdal_merge.py -ot Int16 -of GTiff -o /Users/thayer/Downloads/gdal_merge_test.tif --optfile /private/var/folders/n9/d8sh13b51r14rb00ncq5lhd00000gn/T/processing_cekQEZ/ed309b52b71748699a59af6f06151254/mergeInputFiles.txt

2020-11-30T08:54:03     INFO    GDAL execution console output
             Traceback (most recent call last):
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/__init__.py", line 18, in swig_import_helper
             
              return importlib.import_module(mname)
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/importlib/__init__.py", line 127, in import_module
             
              return _bootstrap._gcd_import(name[level:], package, level)
             
              File "", line 1006, in _gcd_import
             
              File "", line 983, in _find_and_load
             
              File "", line 967, in _find_and_load_unlocked
             
              File "", line 670, in _load_unlocked
             
              File "", line 583, in module_from_spec
             
              File "", line 1043, in create_module
             
              File "", line 219, in _call_with_frames_removed
             
             ImportError: dlopen(/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/_gdal.cpython-37m-darwin.so, 2): Symbol not found: _curl_mime_addpart
             
              Referenced from: /Applications/QGIS.app/Contents/MacOS/bin/../lib/libgdal.27.dylib
             
              Expected in: /usr/lib/libcurl.4.dylib
             
              in /Applications/QGIS.app/Contents/MacOS/bin/../lib/libgdal.27.dylib
             
             
             
             During handling of the above exception, another exception occurred:
             
             
             
             Traceback (most recent call last):
             
              File "/Applications/QGIS.app/Contents/MacOS/bin/gdal_merge.py", line 4, in 
             
              __import__('pkg_resources').run_script('GDAL==3.1.2', 'gdal_merge.py')
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 667, in run_script
             
              self.require(requires)[0].run_script(script_name, ns)
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1464, in run_script
             
              exec(code, namespace, namespace)
             
              File "/Applications/QGIS.app/Contents/Resources/python/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/EGG-INFO/scripts/gdal_merge.py", line 38, in 
             
              from osgeo import gdal
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/__init__.py", line 32, in 
             
              _gdal = swig_import_helper()
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/__init__.py", line 31, in swig_import_helper
             
              return importlib.import_module('_gdal')
             
              File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/importlib/__init__.py", line 127, in import_module
             
              return _bootstrap._gcd_import(name[level:], package, level)
             
             ModuleNotFoundError: No module named '_gdal'
             
PeterPetrik commented 3 years ago

sorry, just uploaded qgis_pr_final-3_16_1_20201130_063613.dmg

thayeray commented 3 years ago

Thanks for uploading 3.16.1, and your continued attention to this, Peter.  I tried the Raster: Merge tool in QGIS after installing. I got the same result, pasted below. By the way, I did not stumble on this myself, I was made aware of the problem by someone who posted to the QGIS User list serve. http://osgeo-org.1560.x6.nabble.com/asking-for-help-td5450765.html

-Thayer I get the same in the tool: The following layers were not correctly generated. • /private/var/folders/n9/d8sh13b51r14rb00ncq5lhd00000gn/T/processing_nYIfab/407436fbc0514c2784d9856f5fa8ca3a/OUTPUT.tif You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

And the same in the Processing tab: 2020-11-30T12:10:50     INFO    Loading OTB '6.6.0'.

2020-11-30T12:10:51     WARNING    Duplicate parameter coordinates registered for alg v.net.visibility

2020-11-30T12:11:34     INFO    gdal_merge.py -ot Int16 -of GTiff -o /private/var/folders/n9/d8sh13b51r14rb00ncq5lhd00000gn/T/processing_nYIfab/407436fbc0514c2784d9856f5fa8ca3a/OUTPUT.tif --optfile /private/var/folders/n9/d8sh13b51r14rb00ncq5lhd00000gn/T/processing_nYIfab/5729705e857c4fc8af8c68b29f15085c/mergeInputFiles.txt

2020-11-30T12:11:34     INFO    GDAL execution console output              Traceback (most recent call last):                             File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/init.py", line 18, in swig_import_helper                             return importlib.import_module(mname)                             File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/importlib/init.py", line 127, in import_module                             return _bootstrap._gcd_import(name[level:], package, level)                             File "", line 1006, in _gcd_import                             File "", line 983, in _find_and_load                             File "", line 967, in _find_and_load_unlocked                             File "", line 670, in _load_unlocked                             File "", line 583, in module_from_spec                             File "", line 1043, in create_module                             File "", line 219, in _call_with_frames_removed                            ImportError: dlopen(/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/_gdal.cpython-37m-darwin.so, 2): Symbol not found: _curl_mime_addpart                             Referenced from: /Applications/QGIS.app/Contents/MacOS/bin/../lib/libgdal.27.dylib                             Expected in: /usr/lib/libcurl.4.dylib                             in /Applications/QGIS.app/Contents/MacOS/bin/../lib/libgdal.27.dylib                                                        During handling of the above exception, another exception occurred:                                                        Traceback (most recent call last):                             File "/Applications/QGIS.app/Contents/MacOS/bin/gdal_merge.py", line 4, in                             import('pkg_resources').run_script('GDAL==3.1.2', 'gdal_merge.py')                             File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/pkg_resources/init.py", line 667, in run_script                             self.require(requires)[0].run_script(script_name, ns)                             File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/pkg_resources/init.py", line 1464, in run_script                             exec(code, namespace, namespace)                             File "/Applications/QGIS.app/Contents/Resources/python/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/EGG-INFO/scripts/gdal_merge.py", line 38, in                             from osgeo import gdal                             File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/init.py", line 32, in                             _gdal = swig_import_helper()                             File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/site-packages/GDAL-3.1.2-py3.7-macosx-10.13.0-x86_64.egg/osgeo/init.py", line 31, in swig_import_helper                             return importlib.import_module('_gdal')                             File "/Applications/QGIS.app/Contents/MacOS/lib/python3.7/importlib/init.py", line 127, in import_module                             return _bootstrap._gcd_import(name[level:], package, level)                            ModuleNotFoundError: No module named '_gdal'              

On Monday, November 30, 2020, 09:54:20 AM EST, Peter Petrik <notifications@github.com> wrote:

sorry, just uploaded qgis_pr_final-3_16_1_20201130_063613.dmg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

news24lor commented 3 years ago

This bug is only present in QGIS 3.16.x: GDAL does not load. QGIS 3.14.x is OK

PeterPetrik commented 3 years ago

Can you retest with https://qgis.org/downloads/macos/nightly/qgis_nightly_master_20210205_032843.dmg ? It is working for me here, so I am wondering if it works also for non-developer machines? thanks

news24lor commented 3 years ago

Now it works! Thank you

thayeray commented 3 years ago

Hi Peter, it took me a while to get back to this. When I test it, with the 2/5/2021 nightly master build 32843, I get the same "ModuleNotFoundError: No module named '_gdal'". The same is true when I try to run gdal_merge.py from the command line using the included python3.7.

/Applications/QGIS-Dev.app/Contents/MacOS/bin/python3 \ /Applications/QGIS-Dev.app/Contents/MacOS/bin/gdal_merge.py

PeterPetrik commented 3 years ago

I fixed it yesterday, can you try today's nighlty?

thayeray commented 3 years ago

Thanks, yes it does work now! I get a merged raster. The tool's log does show this error though, not sure if it is important:

ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db

I guess it goes without saying but, gdal_merge.py also works from the command line.