Closed janzandr closed 5 months ago
I tested it on Ubuntu 20.04 in a mambaforge environment with QGIS 3.28.4 (see appended *.yml.txt), EnMAP-Box 3.11.1 and Google Earth Engine plugin 0.0.6, but could not reproduce it.
Install both plugins by placing it into the default plugin location of qgis: /opt/conda/envs/gui/share/qgis/python/plugins/
This location is for QGIS core plugins only and should not be modified manually. Plugins loaded from the QGIS plugion repository are installed into the active user-profile environment located in ~/.local/share/QGIS/QGIS3/profiles
I matched the main conda packages as per the yaml file and also shifted the plugin location to ~/.local/share/QGIS/QGIS3/profiles/enmapbox/python/plugins
. But I still get the same crash while activating the plugin (whereas google earth gets activated). @jakimowb, could you share your conda install
command to see the pinned versions?
@nikhilstcorp you may try it with one of these environments:
mamba env create -n enmapbox_full_3.28 -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_full_3.28.yml
mamba env create -n enmapbox_light_3.28 -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_light_3.28.yml
The light
environment installs QGIS only. This should be enough to install the EnMAP-Box using the plugin manager.
The full
environment installs many other packages required by sub-applications, e.g. scikit-learn. package
@nikhilstcorp does the crash produce any output to the shell from which you have started QGIS?
Meanwhile I also tested it on Ubuntu 22.04.2 LTS with this environment:
mamba env create -n enmapbox_full_latest -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_full_latest.yml
using the EnMAP-Box 3.11 from the QGIS plugin repository, as well the upcomming 3.12. In both cases I can install and start the EnMAP-Box.
@jakimowb Thank you for the env create scripts. QGIS gets installed but the issue didn't get resolved.
From terminal I get the following error while activating the plugin.
QGIS died on signal 11
.
Is it possible to activate the enmapbox plugin from terminal by default before opening QGIS?
I looked into this solution :https://github.com/qgis/QGIS/issues/31350 for QGIS died on signal 11
But that doesn't help with qgis with plugins.
@nikhilstcorp in case the error still exists, could you please send me a prinout of your environmental variables of your CEM-PAL environment, after you activated the mamba
environment?
I alos updated the conda/mamba environment files in https://github.com/EnMAP-Box/enmap-box/tree/main/.conda .
@jakimowb Thanks for the support. The error was resolved with the later version as follows. No extra tweaks were needed.
/opt/conda/bin/mamba env create -n gui -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_full_latest.yml
with plugin versions : enmapboxplugin-3.12.1
and ee_plugin-0.0.6
The issue can be closed as well.
Thank you for your help @nikhilstcorp
Please reopen.
I get this crash
QGIS died on signal 11Aborted (core dumped)
at installing enmapbox plugin 3.13.0 from QGIS 3.32.3-Lima on ubuntu 22.04
and then, once I restart Qgis, it crashes as soon as the plugin is activated.
Note that
$ python3 -m pip install -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/requirements.txt
works fine
QGIS version 3.32.3-Lima QGIS code revision 67d46100b5 Qt version 5.15.3 Python version 3.10.12 GDAL/OGR version 3.4.1 PROJ version 8.2.1 EPSG Registry database version v10.041 (2021-12-03) GEOS version 3.10.2-CAPI-1.16.0 SQLite version 3.37.2 PDAL version 2.3.0 PostgreSQL client version 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1) SpatiaLite version 5.0.1 QWT version 6.1.4 QScintilla2 version 2.11.6 OS version Pop!_OS 22.04 LTS
Please note this is as critical as [#671], as it renders enmap-box unusable.
Can confirm this, just tested in a docker environment with Ubuntu 22.04.3 and QGIS 3.32.3 (installed via apt, not conda).
The crash also occurs with QGIS 3.28.12 and enmapbox 3.12.1 on Ubuntu 22.04 It is surprising this issue is not marked as critical.
keeps crashing with QGIS 3.34 and enmapbox 3.13.1
I can reproduce it.
@aloboa It is caused by netCDF4:
import netCDF4
@janzandr as far as I remember we had netCDF / HDF issues already in the past. How did you solve them?
Looks like netCDF needs to be installed with apt instead pip3
@aloboa does this workaround solve your problem?
pip: pip3 uninstall netCDF4
sudo apt install python3-netcdf4
Then restart QGIS and try import netCDF4
@jakimowb now that we have OS-specific requirement.txt files, we can easily do this.
For Windows everything should be fine:
For Linux, you need to exclude the netCDF4 from the PIP requirements and explicitely describe the APT install intruction.
@janzandr as far as I remember we had netCDF / HDF issues already in the past. How did you solve them?
No. I think you confuse netCDF4 with h5py :-)
Looks like netCDF needs to be installed with apt instead pip3
@aloboa does this workaround solve your problem?
1. Remove netCDF from `pip: pip3 uninstall netCDF4` 2. Install netCDF via apt: `sudo apt install python3-netcdf4`
Then restart QGIS and try
import netCDF4
I run both commands, start QGIS (without enmapbox installed) and open the python console.
import netCDF4
crashes QGIS.
So this does not solve the issue but at least points to netCDF4 as the culprit. Whether netCDF4 is installed using pip3 or apt does not seem to make any difference.
Some more experiments...
import scipy
and works import netCDF4
and QGIS crashes.import scipy
and import netCDF4
. I just get "ModuleNotFoundError: No module named 'scipy'/'netCDF4'", but no crash.import netCDF4
: simply No module named 'netCDF4'QGIS crashing despite not having netCDF4 installed implies that netCDF4 is not really the culprit...
Looks like netCDF needs to be installed with apt instead pip3
@aloboa does this workaround solve your problem?
1. Remove netCDF from `pip: pip3 uninstall netCDF4` 2. Install netCDF via apt: `sudo apt install python3-netcdf4`
Then restart QGIS and try
import netCDF4
Seems to do the trick (in Ubuntu 22.04 Docker environment). But, the error mentioned in #466 shows up. After installing pyqt5-dev-tools
via apt the EnMAP-Box starts.
In Ubuntu 23.10 the pip version of netCDF does not seem to cause any problems. But here also pyqt5-dev-tools
needs to be installed. I'll try to incorporate this in new installation instructions.
No luck here. I can open the python console in QGIS and import scipy/sys.executable/pip, butimport netCDF
crashes QGIS (with pip-removed netCDF and both with and without netCDF installed via apt).
Installing
sudo apt install pyqt5-dev-tools
does not make any difference
Having netCDF installed via apt (sudo apt install python3-netcdf4), I open a python console outside QGIS and:
$ python
Python 3.9.5 (default, Jun 4 2021, 12:28:51)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'netCDF'
More experiments:
In all cases I have done
sudo apt install pyqt5-dev-tools
1.1 These packages do not import despite having been installed with pip3 uninstall from console (outside QGIS):
PyOpenGL
1.2. These packages are imported despite having been uninstalled with pip3 uninstall from console (outside QGIS):
sympy catboost lightgbm xgboost astropy typeguard
netCDF4
$ python
Python 3.9.5 (default, Jun 4 2021, 12:28:51)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
>>> import sympy
>>> import catboost
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'catboost'
>>> import ligthtgbm
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ligthtgbm'
>>> import lightgbm
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'lightgbm'
>>> import xgboost
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'xgboost'
>>> import astropy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'astropy'
>>> import typeguard
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'typeguard'
>>> import PyOpenGL
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'PyOpenGL'
>>> import scikit_learn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'scikit_learn'
>>> import enpt_enmapbox
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'enpt_enmapbox'
I think there might be a parallel issue with python environments that complicates the real issue. Is there a way to use pip from within the QGIS python console?
I think there might be a parallel issue with python environments that complicates the real issue. Is there a way to use pip from within the QGIS python console?
Yes, if you are running a newer QGIS (I think >=3.32) you can use !
at the beginning of a command in the QGIS Python console, and everything afterwards will run as a normal shell command, e.g.
!python3 -m pip list -v
I am currently revising the installation instructions, and I would recommend using a virtual environment, in the latest Ubuntu version you will have to, as the system python is protected ("externally managed", see PEP 668). This way, system packages can not be messed with by pip and everything the EnMAP-Box needs will be inside a separate environment. Similar to conda.
Here the steps:
sudo apt install python3-pip python3-venv pyqt5-dev-tools python3-h5py python3-pyqt5.qtopengl
~/.virtualenvs/enmapbox
):python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox
source ~/.virtualenvs/enmapbox/bin/activate
Install missing python dependencies inside the virtual environment: MAKE SURE NOT to install netcdf here, or uninstall, I need to generate a separate/new requirements file ...
python3 -m pip install -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/osgeo4w/requirements_osgeo4w.txt
Start QGIS (from the activated environment, see 3.):
qgis
In order to easily start QGIS with the required python environment add a shortcut to your applications menu:
Create the file ~/.local/share/applications/enmapbox.desktop
with the following content (if you used another installation path in the instructions above change accordingly):
[Desktop Entry]
Name=QGIS (EnMAP-Box)
Exec=/bin/bash -c "source ~/.virtualenvs/enmapbox/bin/activate && qgis %F"
Terminal=false
Icon=qgis
Type=Application
Done down to step 5. There I have a doubt:
How do I run QGIS within the enmapbox.desktop environment? Just by running the existing/usr/bin/qgis
from within that directory? Or should I perform a dedicated QGIS installation?
If I open QGIS using the icon created by the file ~/.local/share/applications/enmapbox.desktop
,
and list packages with:
>>> !python3 -m pip list -v
I do get PyOpenGL listed, but:
>>> import PyOpenGL
Traceback (most recent call last):
File "/usr/lib/python3.10/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'PyOpenGL'
Not sure this could be a problem.
For the rest of packages:
import typeguard
import astropy
import xgboost
import lightgbm
import catboost
import sympy
import numba
import scikit-learn
Traceback (most recent call last):
File "/usr/lib/python3.10/code.py", line 63, in runsource
code = self.compile(source, filename, symbol)
File "/usr/lib/python3.10/codeop.py", line 153, in __call__
return _maybe_compile(self.compiler, source, filename, symbol)
File "/usr/lib/python3.10/codeop.py", line 73, in _maybe_compile
return compiler(source, filename, symbol)
File "/usr/lib/python3.10/codeop.py", line 118, in __call__
codeob = compile(source, filename, symbol, self.flags, True)
File "<input>", line 1
import scikit-learn
^
SyntaxError: invalid syntax
import enpt-enmapbox
Traceback (most recent call last):
File "/usr/lib/python3.10/code.py", line 63, in runsource
code = self.compile(source, filename, symbol)
File "/usr/lib/python3.10/codeop.py", line 153, in __call__
return _maybe_compile(self.compiler, source, filename, symbol)
File "/usr/lib/python3.10/codeop.py", line 73, in _maybe_compile
return compiler(source, filename, symbol)
File "/usr/lib/python3.10/codeop.py", line 118, in __call__
codeob = compile(source, filename, symbol, self.flags, True)
File "<input>", line 1
import enpt-enmapbox
^
SyntaxError: invalid syntax
>>> import netCDF4
crashes QGIS
@aloboa some import differ from the PIP package names:
PyOpenGL -> import OpenGL
scikit-learn -> import sklearn
EnPT app is imported by import enpt_enmapboxapp
(note the underscore _
)
@aloboa some import differ from the PIP package names: PyOpenGL -> import OpenGL scikit-learn -> import sklearn
EnPT app is imported by import enptenmapboxapp (note the underscore )
Good. But this does not solve the real issue: import netCDF4 still crashes QGIS even if
' >>> !python3 -m pip list -v'
confirms that is not installed.
Does QGIS have to be reinstalled from/on the enmapbox.desktop environment?
Does QGIS have to be reinstalled from/on the enmapbox.desktop environment?
No, QGIS needs to be installed the default way via apt as described on the official QGIS web page. The virtual environment uses everything it needs from the QGIS/Base python (hence the --system-site-packages
), but does not allow it to be altered with pip. Pip is only allowed to install/alter packages in the virtual environment.
I suspect there is still a netcdf installation maybe in the site-packages of the base python?
What does (!python3 -m site
) called from within the QGIS Python console return?
There should be several paths, maybe check if there is one or multiple site-packages folders which contain netcdf:
!python3 -m site | grep -o "/[^ ']*" | xargs -I% find % -type d -name "*netCDF*"
What does (!python3 -m site) called from within the QGIS Python console return?
!python3 -m site sys.path = [ '/home/alobo', '/home/alobo/miniconda3/lib/python39.zip', '/home/alobo/miniconda3/lib/python3.9', '/home/alobo/miniconda3/lib/python3.9/lib-dynload', '/home/alobo/.virtualenvs/enmapbox/lib/python3.9/site-packages', '/home/alobo/.local/lib/python3.9/site-packages', '/home/alobo/miniconda3/lib/python3.9/site-packages', ] USER_BASE: '/home/alobo/.local' (exists) USER_SITE: '/home/alobo/.local/lib/python3.9/site-packages' (exists) ENABLE_USER_SITE: True
check if there is one or multiple site-packages folders which contain netcdf
!python3 -m site | grep -o "/[^ ']*" | xargs -I% find % -type d -name "*netCDF*"
/home/alobo/my_repositories/enmap-box/tests/non-blocking-dependencies/netCDF4
/home/alobo/Dropbox/WikiMind/notes/Notebooks/ZimNotes/Home/Work/Software/SoftRSGISImProc/netCDF_(nc)
/home/alobo/Downloads/enmapboxplugin/enmapboxtesting/non-blocking-dependencies/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs
/home/alobo/.local/lib/python3.10/site-packages/netCDF4
/home/alobo/.local/lib/python3.10/site-packages/netCDF4-1.6.4.dist-info
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs
/home/alobo/miniconda3/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4-1.6.2.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4
/home/alobo/miniconda3/pkgs/libnetcdf-4.8.1-nompi_hb3fd0d9_101/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4
/home/alobo/miniconda3/envs/qgis_stable/lib/cmake/netCDF
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4
find: ‘/home/alobo/miniconda3/lib/python39.zip’: No such file or directory
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs
/home/alobo/.local/lib/python3.10/site-packages/netCDF4
/home/alobo/.local/lib/python3.10/site-packages/netCDF4-1.6.4.dist-info
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs
So yes, it looks like there is a lot of netCDF4 cleaning to do... Any suggestions on how to proceed?
I am confused why !python3 -m site
returns the miniconda python, maybe the shell call is the wrong thing here as conda is the system default. What does this call from the QGIS Python console return?
import sys
sys.path
Does it show miniconda
paths there also, or rather /usr/lib/python3.10
?
I am confused why
!python3 -m site
returns the miniconda python, maybe the shell call is the wrong thing here as conda is the system default. What does this call from the QGIS Python console return?import sys sys.path
i``` mport sys sys.path ['/usr/share/qgis/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/alobo/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python']
>
> Does it show `miniconda` paths there also, or rather `/usr/lib/python3.10`?
No miniconda in the path, AFAICS.
Then I assume if you run this from the normal shell (not QGIS) you will see a netcdf installation?
/usr/bin/python3.10 -m pip list -v
?
If it shows up and is listed with a path pointing to your home directory, try to uninstall it using
/usr/bin/python3.10 -m pip uninstall netCDF4
It turns out that my by default python version is 3.9.5
$ which python
/home/alobo/miniconda3/bin/python
(base) alobo@pop-os:~$ python
Python 3.9.5 (default, Jun 4 2021, 12:28:51)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
Is this not odd? I thought python
would always run the latest version.
If I do
$ /usr/lib/python3.10 -m pip list -v
bash: /usr/lib/python3.10: Is a directory
I can use
python3.10 -m pip list -v
and then I see
netCDF4 1.6.4 /home/alobo/.local/lib/python3.10/site-packages pip
Then
$ python3.10 -m pip uninstall netCDF4
Found existing installation: netCDF4 1.6.4
Uninstalling netCDF4-1.6.4:
Would remove:
/home/alobo/.local/lib/python3.10/site-packages/netCDF4-1.6.4.dist-info/*
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libaec-e300f322.so.0.0.12
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libblosc-a9e8a7c7.so.1.21.1
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libcurl-47074032.so.4.8.0
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libhdf5-c6d05200.so.200.2.0
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libhdf5_hl-f45ebdf9.so.200.1.0
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libnetcdf-1281958a.so.19
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libsz-57467d8a.so.2.0.1
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libzstd-c4cc7033.so.1.5.2
/home/alobo/.local/lib/python3.10/site-packages/netCDF4/*
Proceed (Y/n)? Y
Successfully uninstalled netCDF4-1.6.4
Unfortunately, even now, running import netCDF4 from within the QGIS python console crashes QGIS
I have found that
$ which python3
/home/alobo/miniconda3/bin/python3
thus run:
$ python3 -m pip list -v
and there is no netCDF4 listed there
$ python3 -m pip uninstall netCDF4
WARNING: Skipping netCDF4 as it is not installed.
Yet, a miniconda3 netCDF4 is listed from within QGIS:
!python3 -m site | grep -o "/[^ ']*" | xargs -I% find % -type d -name "*netCDF*"
/home/alobo/my_repositories/enmap-box/tests/non-blocking-dependencies/netCDF4
/home/alobo/Dropbox/WikiMind/notes/Notebooks/ZimNotes/Home/Work/Software/SoftRSGISImProc/netCDF_(nc)
/home/alobo/Downloads/enmapboxplugin/enmapboxtesting/non-blocking-dependencies/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs
/home/alobo/miniconda3/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4-1.6.2.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4
/home/alobo/miniconda3/pkgs/libnetcdf-4.8.1-nompi_hb3fd0d9_101/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4
/home/alobo/miniconda3/envs/qgis_stable/lib/cmake/netCDF
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4
find: ‘/home/alobo/miniconda3/lib/python39.zip’: No such file or directory
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs
There was an error in my reply where I confused lib
and bin
, I silently edited this several minutes after posting. But the way you proceeded yielded the same result I assume.
Regarding QGIS and the conda python I would assume it should not be a problem, as the sys.path
call inside the QGIS Python console does not show any references to conda. And here the !
shell commands do not seem reliable as it seems to forward it to the system shell with all its defaults, and there python3
is linked to the miniconda installation.
What does sys.executable
in the QGIS Python console return?
There was an error in my reply where I confused
lib
andbin
, I silently edited this several minutes after posting. But the way you proceeded yielded the same result I assume. Regarding QGIS and the conda python I would assume it should not be a problem, as thesys.path
call inside the QGIS Python console does not show any references to conda. And here the!
shell commands do not seem reliable as it seems to forward it to the system shell with all its defaults, and therepython3
is linked to the miniconda installation. What doessys.executable
in the QGIS Python console return?
import sys.executable
Traceback (most recent call last):
File "/usr/lib/python3.10/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'sys.executable'; 'sys' is not a package
import sys works
Yes, sorry, I was a bit unprecise, I meant both
import sys; sys.executable
Yes, sorry, I was a bit unprecise, I meant both
import sys; sys.executable
import sys; sys.executable
'/home/alobo/miniconda3/bin/python3'
and
$ /home/alobo/miniconda3/bin/python3 -m pip uninstall netCDF4
WARNING: Skipping netCDF4 as it is not installed.
import sys; sys.executable '/home/alobo/miniconda3/bin/python3'
Now I'm confused. Did you run that from within QGIS python console? You are using a standard QGIS installation (installed via apt as described on qgis.org), not conda QGIS, right?
It still puzzles me why the conda python is returned there. What result does this show (again, QGIS Python console)?
import numpy
numpy.__path__
I can think of two things to try:
conda activate base
conda remove netCDF4
or
I realize I was using/usr/bin/qgis
instead of the one from the environment you indicated yesterday, so I have deleted some of my previous answers and tried again. Some of the answers regarding paths may be different, but QGIS keeps crashing after import netCDF4
.
From the linux console:
$ which python3
/home/alobo/miniconda3/bin/python3
Running the python console through the QGIS icon you indicated yesterday:
import sys; sys.executable
'/home/alobo/.virtualenvs/enmapbox/bin/python3'
import sys
sys.path
['/usr/share/qgis/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/alobo/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python']
import numpy
numpy.__path__
['/home/alobo/.local/lib/python3.10/site-packages/numpy']
I realize I was using
/usr/bin/qgis
instead of the one from the environment you indicated yesterday
That was not the problem, if QGIS crashes even without the virtual environment, it means that there is still some netCDF installation which causes a problem and that has to be adressed before installing the additional python packages. So ignore the virtual environment thing for now.
I assume the miniconda environment is somehow your default system python or some python related environment variables are linked to this conda installation. And, you also have a netCDF from a python3.8 installation in your home directory. Did you do a system upgrade at some point from 20.04 to 22.04? But this should not be the problem actually...
Maybe try this:
Open the Terminal, type conda deactivate
, then check which python3
, and then start qgis
from this terminal session. Try to import netCDF4
in the QGIS Python console.
No crash!
$ conda deactivate
alobo@pop-os:~$ which python3
/usr/bin/python3
alobo@pop-os:~$ qgis
# Python Console
# Use iface to access QGIS API interface or type '?' for more info
# Security warning: typing commands from an untrusted source can harm your computer
import sys
sys.path
['/usr/share/qgis/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/alobo/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python']
import numpy
import sys; sys.executable
'/usr/bin/python3'
>> import netCFD4
Traceback (most recent call last):
File "/usr/lib/python3.10/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'netCFD4'
Ok, that's good, now try to repeat the installation steps starting with the virtual environment (step 5). You may just delete the existing ~/.virtualenvs/enmapbox
. Before you start, make sure the first thing you type in the terminal session is conda deactivate
. Also start QGIS from this session and see if the EnMAP-Box starts and "everything" works.
(base) alobo@pop-os:~$ cd .virtualenvs
(base) alobo@pop-os:~/.virtualenvs$ ls
enmapbox rgee
(base) alobo@pop-os:~/.virtualenvs$ rm -r enmapbox
(base) alobo@pop-os:~/.virtualenvs$ conda deactivate
alobo@pop-os:~/.virtualenvs$ which python3
/usr/bin/python3
alobo@pop-os:~/.virtualenvs$ python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox
alobo@pop-os:~/.virtualenvs$ cd enmapbox
alobo@pop-os:~/.virtualenvs/enmapbox$ source ~/.virtualenvs/enmapbox/bin/activate
(enmapbox) alobo@pop-os:~/.virtualenvs/enmapbox$ python3 -m pip install -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt
Requirement already satisfied: typeguard==2.13.3 in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 2)) (2.13.3)
Requirement already satisfied: astropy in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (5.3.4)
Requirement already satisfied: PyOpenGL in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 4)) (3.1.7)
Requirement already satisfied: xgboost in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 5)) (2.0.0)
Requirement already satisfied: lightgbm in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 6)) (4.1.0)
Requirement already satisfied: catboost in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.2.2)
Requirement already satisfied: sympy in /usr/lib/python3/dist-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 10)) (1.9)
Requirement already satisfied: numba>=0.57 in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 11)) (0.58.0)
Requirement already satisfied: enpt-enmapboxapp in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 12)) (0.7.6)
Requirement already satisfied: scikit-learn>=1.0 in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 13)) (1.3.1)
Requirement already satisfied: packaging>=19.0 in /home/alobo/.local/lib/python3.10/site-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (23.2)
Requirement already satisfied: PyYAML>=3.13 in /usr/lib/python3/dist-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (5.4.1)
Requirement already satisfied: pyerfa>=2.0 in /home/alobo/.local/lib/python3.10/site-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (2.0.1)
Requirement already satisfied: numpy<2,>=1.21 in /home/alobo/.local/lib/python3.10/site-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (1.25.2)
Requirement already satisfied: scipy in /home/alobo/.local/lib/python3.10/site-packages (from xgboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 5)) (1.11.3)
Requirement already satisfied: matplotlib in /home/alobo/.local/lib/python3.10/site-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (3.8.0)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.16.0)
Requirement already satisfied: pandas>=0.24 in /home/alobo/.local/lib/python3.10/site-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2.1.1)
Requirement already satisfied: graphviz in /home/alobo/.local/lib/python3.10/site-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (0.20.1)
Requirement already satisfied: plotly in /usr/lib/python3/dist-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (5.4.0)
Requirement already satisfied: llvmlite<0.42,>=0.41.0dev0 in /home/alobo/.local/lib/python3.10/site-packages (from numba>=0.57->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 11)) (0.41.0)
Requirement already satisfied: psutil in /home/alobo/.local/lib/python3.10/site-packages (from enpt-enmapboxapp->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 12)) (5.9.6)
Requirement already satisfied: importlib-metadata in /usr/lib/python3/dist-packages (from enpt-enmapboxapp->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 12)) (4.6.4)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/alobo/.local/lib/python3.10/site-packages (from scikit-learn>=1.0->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 13)) (3.2.0)
Requirement already satisfied: joblib>=1.1.1 in /home/alobo/.local/lib/python3.10/site-packages (from scikit-learn>=1.0->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 13)) (1.3.2)
Requirement already satisfied: tzdata>=2022.1 in /home/alobo/.local/lib/python3.10/site-packages (from pandas>=0.24->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2023.3)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/alobo/.local/lib/python3.10/site-packages (from pandas>=0.24->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3/dist-packages (from pandas>=0.24->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2022.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.4.5)
Requirement already satisfied: cycler>=0.10 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (0.12.1)
Requirement already satisfied: pillow>=6.2.0 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (10.1.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.1.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (4.43.1)
Requirement already satisfied: pyparsing>=2.3.1 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (3.1.1)
alobo@pop-os:~/.virtualenvs/enmapbox$ qgis
and QGIS crahes again with import netCDF4 If I uninstall netCDF4 before running QGIS
import netCDF4 does not crash. Therefore, I understand that your list of requirements is not correct and netCDF4 gets installed.
After uninstalling netCDF4,
alobo@pop-os:~$ sudo python3 -m pip uninstall netCDF4
I can install enmapbox from within QGIS normally, but get:
EnMAPBoxApplication error(s):
ensomap:
ImportError:cannot import name 'docstring' from 'matplotlib' (/home/alobo/.local/lib/python3.10/site-packages/matplotlib/__init__.py)
Traceback:
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/gui/applications.py", line 368, in addApplicationFolder
apps = factory(self.mEnMAPBox)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/__init__.py", line 35, in enmapboxApplicationFactory
from ensomap.enmapboxintegration import EnSoMAP
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/enmapboxintegration.py", line 32, in
import hys
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/hys/__init__.py", line 27, in
from .gui import *
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/hys/gui.py", line 12, in
import matplotlib.pyplot as plt
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/lib/python3.10/site-packages/matplotlib/pyplot.py", line 66, in
from matplotlib.figure import Figure, FigureBase, figaspect
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/lib/python3.10/site-packages/matplotlib/figure.py", line 43, in
from matplotlib import _blocking_input, backend_bases, _docstring, projections
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py", line 58, in
from mpl_toolkits.mplot3d import Axes3D
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in
from .axes3d import Axes3D
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in
from matplotlib import _api, cbook, docstring, _preprocess_data
All the packages from the requirements.txt should NOT be in /home/alobo/.local/lib/python3.10/site-packages
but somewhere in ~/.virtualenvs/enmapbox
. So I assume at some point during the installation or rather at previous attempts you ran the pip install
without the virtual environment being activated (indicated by the (enmapbox)
at the beginning of the line). So the error shown is maybe a result of version mismatch between packages, matplotlib in this case. Does the EnMAP-Box start at least? If you do not use EnSoMAP you can ignore this error. I will check again if some package in the requirements file needs/installs netcdf, but it was not installed on my test system if I remember correctly.
netCFD4 got installed for sure in the sequence of commands:
$ python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox
$ cd enmapbox
$ source ~/.virtualenvs/enmapbox/bin/activate
I can tell for sure because, as I wrote, QGIS was not crashing in the previous run ,and the problem was corrected
by
$ sudo python3 -m pip uninstall netCDF4
Perhaps the sequence should have been:
$ source ~/.virtualenvs/enmapbox/bin/activate
$ python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox
$ cd enmapbox
Does the EnMAP-Box start at least?
AFICS, yes. At least I can display images and spectra.
netCFD4 got installed for sure in the sequence of commands:
Can not confirm this on my Ubuntu 23.10 VM. I doubt that it is any different on 22.04.
As the automatic activation of the conda environment in bash is rather inconvenient in your case (I assume?), it might make sense to deactivate this behavior, via
conda config --set auto_activate_base false
You can still run conda activate
anytime you need the conda environment.
Also, I am not sure if that has had any effect on the QGIS installation (as python environment variables are altered). E.g., it surprises me that numpy is installed in /home/alobo/.local/lib/python3.10/site-packages
, in my case its part of the dist-packages at /usr/lib/python3/dist-packages/numpy
.
Does the EnMAP-Box start at least?
AFICS, yes. At least I can display images and spectra.
Good if that works, maybe check if the Classification workflow runs with the Example dataset, then most functionality should be there 👍
Reported by Kevin Alonso via email.
Summary:
The Enmapbox plugin is crashing QGIS when activated in the CEM-PAL dataviewer environment (observed in native Ubuntu 20.04 machine as well). The plugin is installed using conda(mamba) in CEM.PAL and tried with apt in native ubuntu machine.
Steps to Reproduce:
Expected Result:
The Enmapbox plugin should activate without crashing QGIS.
Error log in Journalctl:
Module linux-vdso.so.1 with build-id 6e10761710ea7ed473c819bcfd49208133158806 Stack trace of thread 147: #0 0x00007f8dc3dae2ab n/a (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so + 0x142ab) ELF object binary architecture: AMD x86-64
Environment:
Ubuntu 20.04 LTS
QGIS 3.28.3
Enmapbox plugin 3.11.1
Additional Notes: