QGIS crashes when activating the EnMAP-Box plugin

[janzandr]

Reported by Kevin Alonso via email.

Summary:

The Enmapbox plugin is crashing QGIS when activated in the CEM-PAL dataviewer environment (observed in native Ubuntu 20.04 machine as well). The plugin is installed using conda(mamba) in CEM.PAL and tried with apt in native ubuntu machine.

Steps to Reproduce:

1. Install QGIS using conda with conda-forge channel
2. Download zip file for enmapbox and google earth engine (requirement for enmapbox) from the following links, https://plugins.qgis.org/plugins/enmapboxplugin/version/3.11.1/download/ https://plugins.qgis.org/plugins/ee_plugin/version/0.0.6/download/
3. Install both plugins by placing it into the default plugin location of qgis: /opt/conda/envs/gui/share/qgis/python/plugins/
4. Activate Enmapbox plugin in QGIS plugin window
5. Observe QGIS crash.

Expected Result:

The Enmapbox plugin should activate without crashing QGIS.

Error log in Journalctl:

Module linux-vdso.so.1 with build-id 6e10761710ea7ed473c819bcfd49208133158806 Stack trace of thread 147: #0 0x00007f8dc3dae2ab n/a (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so + 0x142ab) ELF object binary architecture: AMD x86-64

Environment:

Ubuntu 20.04 LTS

QGIS 3.28.3

Enmapbox plugin 3.11.1

Additional Notes:

1. Few pip packages are installed with pip within mamba.(Packages installed with mamba and pip are attached )
2. The Enmapbox plugin does not crash on Windows and wslg
3. Other plugins in QGIS activate without an issue (google earth engine) with the same process.
4. Issue is replicated with older versions of qgis and older versions of enmapbox plugin
5. The issue persists with resolutions provided on the internet including reinstallation of specific versions of numpy, h5py, numba and sympy.

[jakimowb]

I tested it on Ubuntu 20.04 in a mambaforge environment with QGIS 3.28.4 (see appended *.yml.txt), EnMAP-Box 3.11.1 and Google Earth Engine plugin 0.0.6, but could not reproduce it.

Install both plugins by placing it into the default plugin location of qgis: /opt/conda/envs/gui/share/qgis/python/plugins/

This location is for QGIS core plugins only and should not be modified manually. Plugins loaded from the QGIS plugion repository are installed into the active user-profile environment located in ~/.local/share/QGIS/QGIS3/profiles

enmapbox_3_28.yml.txt

[nikhilstcorp]

I matched the main conda packages as per the yaml file and also shifted the plugin location to ~/.local/share/QGIS/QGIS3/profiles/enmapbox/python/plugins. But I still get the same crash while activating the plugin (whereas google earth gets activated). @jakimowb, could you share your conda install command to see the pinned versions?

[jakimowb]

@nikhilstcorp you may try it with one of these environments:

mamba env create -n enmapbox_full_3.28 -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_full_3.28.yml
mamba env create -n enmapbox_light_3.28 -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_light_3.28.yml

The lightenvironment installs QGIS only. This should be enough to install the EnMAP-Box using the plugin manager. The full environment installs many other packages required by sub-applications, e.g. scikit-learn. package

[jakimowb]

@nikhilstcorp does the crash produce any output to the shell from which you have started QGIS?

Meanwhile I also tested it on Ubuntu 22.04.2 LTS with this environment: mamba env create -n enmapbox_full_latest -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_full_latest.yml

using the EnMAP-Box 3.11 from the QGIS plugin repository, as well the upcomming 3.12. In both cases I can install and start the EnMAP-Box.

[nikhilstcorp]

@jakimowb Thank you for the env create scripts. QGIS gets installed but the issue didn't get resolved. From terminal I get the following error while activating the plugin. QGIS died on signal 11. Is it possible to activate the enmapbox plugin from terminal by default before opening QGIS?

I looked into this solution :https://github.com/qgis/QGIS/issues/31350 for QGIS died on signal 11 But that doesn't help with qgis with plugins.

[jakimowb]

@nikhilstcorp in case the error still exists, could you please send me a prinout of your environmental variables of your CEM-PAL environment, after you activated the mamba environment?

I alos updated the conda/mamba environment files in https://github.com/EnMAP-Box/enmap-box/tree/main/.conda .

[nikhilstcorp]

@jakimowb Thanks for the support. The error was resolved with the later version as follows. No extra tweaks were needed.

/opt/conda/bin/mamba env create -n gui -f https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.conda/enmapbox_full_latest.yml

with plugin versions : enmapboxplugin-3.12.1 and ee_plugin-0.0.6 The issue can be closed as well.

[jakimowb]

Thank you for your help @nikhilstcorp

[aloboa]

Please reopen. I get this crash QGIS died on signal 11Aborted (core dumped) at installing enmapbox plugin 3.13.0 from QGIS 3.32.3-Lima on ubuntu 22.04 and then, once I restart Qgis, it crashes as soon as the plugin is activated. Note that

$ python3 -m pip install -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/requirements.txt 

works fine

QGIS version 3.32.3-Lima QGIS code revision 67d46100b5 Qt version 5.15.3 Python version 3.10.12 GDAL/OGR version 3.4.1 PROJ version 8.2.1 EPSG Registry database version v10.041 (2021-12-03) GEOS version 3.10.2-CAPI-1.16.0 SQLite version 3.37.2 PDAL version 2.3.0 PostgreSQL client version 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1) SpatiaLite version 5.0.1 QWT version 6.1.4 QScintilla2 version 2.11.6 OS version Pop!_OS 22.04 LTS

[aloboa]

Please note this is as critical as [#671], as it renders enmap-box unusable.

[thielfab]

Can confirm this, just tested in a docker environment with Ubuntu 22.04.3 and QGIS 3.32.3 (installed via apt, not conda).

[aloboa]

The crash also occurs with QGIS 3.28.12 and enmapbox 3.12.1 on Ubuntu 22.04 It is surprising this issue is not marked as critical.

[aloboa]

keeps crashing with QGIS 3.34 and enmapbox 3.13.1

[jakimowb]

I can reproduce it.

[jakimowb]

@aloboa It is caused by netCDF4:

1. Start QGIS and open the python terminal (CTRL+ALT+P)
2. Run import netCDF4

@janzandr as far as I remember we had netCDF / HDF issues already in the past. How did you solve them?

[jakimowb]

Looks like netCDF needs to be installed with apt instead pip3

@aloboa does this workaround solve your problem?

1. Remove netCDF from pip: pip3 uninstall netCDF4
2. Install netCDF via apt: sudo apt install python3-netcdf4

Then restart QGIS and try import netCDF4

[janzandr]

@jakimowb now that we have OS-specific requirement.txt files, we can easily do this. For Windows everything should be fine:

For Linux, you need to exclude the netCDF4 from the PIP requirements and explicitely describe the APT install intruction.

[janzandr]

@janzandr as far as I remember we had netCDF / HDF issues already in the past. How did you solve them?

No. I think you confuse netCDF4 with h5py :-)

[aloboa]

Looks like netCDF needs to be installed with apt instead pip3

@aloboa does this workaround solve your problem?

1. Remove netCDF from `pip: pip3 uninstall netCDF4`

2. Install netCDF via apt: `sudo apt install python3-netcdf4`

Then restart QGIS and try import netCDF4

I run both commands, start QGIS (without enmapbox installed) and open the python console. import netCDF4 crashes QGIS.

So this does not solve the issue but at least points to netCDF4 as the culprit. Whether netCDF4 is installed using pip3 or apt does not seem to make any difference.

[aloboa]

Some more experiments...

1. I uninstall the apt netCDF4package as well (I make sure with Synaptic: it is not installed).
2. I Run QGIS (3.34) (from Debian installation). No enmapbox plugin installed.
3. I open the python console,
4. run import scipy and works
5. run import netCDF4 and QGIS crashes.
6. I Run QGIS (3.34) (from Debian installation) again and install enmapbox plugin (3.13.1): QGIS crashes.
7. I run the flatpack QGIS installation (3.32.3)
8. I open the python console and run import scipy and import netCDF4 . I just get "ModuleNotFoundError: No module named 'scipy'/'netCDF4'", but no crash.
9. I install enmpabox on the flatpack-installed QGIS. Successful installation, no crash but the enmapbox package installer does not work and actually no python modules can be imported (see https://github.com/flathub/org.qgis.qgis/issues/370).
10. I open a python3 console outside QGIS and run import netCDF4: simply No module named 'netCDF4'

QGIS crashing despite not having netCDF4 installed implies that netCDF4 is not really the culprit...

[thielfab]

Looks like netCDF needs to be installed with apt instead pip3

@aloboa does this workaround solve your problem?

1. Remove netCDF from `pip: pip3 uninstall netCDF4`

2. Install netCDF via apt: `sudo apt install python3-netcdf4`

Then restart QGIS and try import netCDF4

Seems to do the trick (in Ubuntu 22.04 Docker environment). But, the error mentioned in #466 shows up. After installing pyqt5-dev-tools via apt the EnMAP-Box starts. In Ubuntu 23.10 the pip version of netCDF does not seem to cause any problems. But here also pyqt5-dev-tools needs to be installed. I'll try to incorporate this in new installation instructions.

[aloboa]

No luck here. I can open the python console in QGIS and import scipy/sys.executable/pip, butimport netCDF crashes QGIS (with pip-removed netCDF and both with and without netCDF installed via apt). Installing
sudo apt install pyqt5-dev-tools does not make any difference

Having netCDF installed via apt (sudo apt install python3-netcdf4), I open a python console outside QGIS and:

$ python
Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'netCDF'

[aloboa]

More experiments:

In all cases I have done sudo apt install pyqt5-dev-tools

1. Import from QGIS python console

1.1 These packages do not import despite having been installed with pip3 uninstall from console (outside QGIS):

PyOpenGL

1.2. These packages are imported despite having been uninstalled with pip3 uninstall from console (outside QGIS):

sympy catboost lightgbm xgboost astropy typeguard

3. Crash QGIS despite having been uninstalled with pip3 uninstall (install or not with app does not matter):

netCDF4

2. Import from a python3 console (outside QGIS):

$ python
Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
>>> import sympy
>>> import catboost
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'catboost'
>>> import ligthtgbm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ligthtgbm'
>>> import lightgbm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'lightgbm'
>>> import xgboost
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'xgboost'
>>> import astropy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'astropy'
>>> import typeguard
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'typeguard'
>>> import PyOpenGL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'PyOpenGL'
>>> import scikit_learn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'scikit_learn'
>>> import enpt_enmapbox
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'enpt_enmapbox'

[aloboa]

I think there might be a parallel issue with python environments that complicates the real issue. Is there a way to use pip from within the QGIS python console?

[thielfab]

I think there might be a parallel issue with python environments that complicates the real issue. Is there a way to use pip from within the QGIS python console?

Yes, if you are running a newer QGIS (I think >=3.32) you can use ! at the beginning of a command in the QGIS Python console, and everything afterwards will run as a normal shell command, e.g. !python3 -m pip list -v

[thielfab]

I am currently revising the installation instructions, and I would recommend using a virtual environment, in the latest Ubuntu version you will have to, as the system python is protected ("externally managed", see PEP 668). This way, system packages can not be messed with by pip and everything the EnMAP-Box needs will be inside a separate environment. Similar to conda.

Here the steps:

1. Make sure the following packages are installed using the system package manager:

sudo apt install python3-pip python3-venv pyqt5-dev-tools python3-h5py python3-pyqt5.qtopengl

2. Create a virtual python environment (https://docs.python.org/3/library/venv.html) in a directory of your choice (e.g. ~/.virtualenvs/enmapbox):

python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox

3. Activate the environment:

source ~/.virtualenvs/enmapbox/bin/activate

4. Install missing python dependencies inside the virtual environment: MAKE SURE NOT to install netcdf here, or uninstall, I need to generate a separate/new requirements file ... python3 -m pip install -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/osgeo4w/requirements_osgeo4w.txt

5. Start QGIS (from the activated environment, see 3.): qgis

In order to easily start QGIS with the required python environment add a shortcut to your applications menu:

Create the file ~/.local/share/applications/enmapbox.desktop with the following content (if you used another installation path in the instructions above change accordingly):

[Desktop Entry]
Name=QGIS (EnMAP-Box)
Exec=/bin/bash -c "source ~/.virtualenvs/enmapbox/bin/activate && qgis %F"
Terminal=false
Icon=qgis
Type=Application

[aloboa]

Done down to step 5. There I have a doubt: How do I run QGIS within the enmapbox.desktop environment? Just by running the existing/usr/bin/qgis from within that directory? Or should I perform a dedicated QGIS installation?

[aloboa]

If I open QGIS using the icon created by the file ~/.local/share/applications/enmapbox.desktop, and list packages with:

>>> !python3 -m pip list -v

I do get PyOpenGL listed, but:

>>> import PyOpenGL
Traceback (most recent call last):
  File "/usr/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'PyOpenGL'

Not sure this could be a problem.

For the rest of packages:

import typeguard
import astropy
import xgboost
import lightgbm
import catboost
import sympy
import numba
import scikit-learn
Traceback (most recent call last):
  File "/usr/lib/python3.10/code.py", line 63, in runsource
    code = self.compile(source, filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 153, in __call__
    return _maybe_compile(self.compiler, source, filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 73, in _maybe_compile
    return compiler(source, filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 118, in __call__
    codeob = compile(source, filename, symbol, self.flags, True)
  File "<input>", line 1
    import scikit-learn
                 ^
SyntaxError: invalid syntax
import enpt-enmapbox
Traceback (most recent call last):
  File "/usr/lib/python3.10/code.py", line 63, in runsource
    code = self.compile(source, filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 153, in __call__
    return _maybe_compile(self.compiler, source, filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 73, in _maybe_compile
    return compiler(source, filename, symbol)
  File "/usr/lib/python3.10/codeop.py", line 118, in __call__
    codeob = compile(source, filename, symbol, self.flags, True)
  File "<input>", line 1
    import enpt-enmapbox
               ^
SyntaxError: invalid syntax

>>> import netCDF4 crashes QGIS

[jakimowb]

@aloboa some import differ from the PIP package names: PyOpenGL -> import OpenGL scikit-learn -> import sklearn

EnPT app is imported by import enpt_enmapboxapp (note the underscore _)

[aloboa]

@aloboa some import differ from the PIP package names: PyOpenGL -> import OpenGL scikit-learn -> import sklearn

EnPT app is imported by import enptenmapboxapp (note the underscore )

Good. But this does not solve the real issue: import netCDF4 still crashes QGIS even if ' >>> !python3 -m pip list -v'
confirms that is not installed. Does QGIS have to be reinstalled from/on the enmapbox.desktop environment?

[thielfab]

Does QGIS have to be reinstalled from/on the enmapbox.desktop environment?

No, QGIS needs to be installed the default way via apt as described on the official QGIS web page. The virtual environment uses everything it needs from the QGIS/Base python (hence the --system-site-packages), but does not allow it to be altered with pip. Pip is only allowed to install/alter packages in the virtual environment. I suspect there is still a netcdf installation maybe in the site-packages of the base python? What does (!python3 -m site) called from within the QGIS Python console return? There should be several paths, maybe check if there is one or multiple site-packages folders which contain netcdf: !python3 -m site | grep -o "/[^ ']*" | xargs -I% find % -type d -name "*netCDF*"

[aloboa]

What does (!python3 -m site) called from within the QGIS Python console return?

!python3 -m site
sys.path = [
'/home/alobo',
'/home/alobo/miniconda3/lib/python39.zip',
'/home/alobo/miniconda3/lib/python3.9',
'/home/alobo/miniconda3/lib/python3.9/lib-dynload',
'/home/alobo/.virtualenvs/enmapbox/lib/python3.9/site-packages',
'/home/alobo/.local/lib/python3.9/site-packages',
'/home/alobo/miniconda3/lib/python3.9/site-packages',
]
USER_BASE: '/home/alobo/.local' (exists)
USER_SITE: '/home/alobo/.local/lib/python3.9/site-packages' (exists)
ENABLE_USER_SITE: True

check if there is one or multiple site-packages folders which contain netcdf

!python3 -m site | grep -o "/[^ ']*" | xargs -I% find % -type d -name "*netCDF*"
/home/alobo/my_repositories/enmap-box/tests/non-blocking-dependencies/netCDF4
/home/alobo/Dropbox/WikiMind/notes/Notebooks/ZimNotes/Home/Work/Software/SoftRSGISImProc/netCDF_(nc)
/home/alobo/Downloads/enmapboxplugin/enmapboxtesting/non-blocking-dependencies/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs
/home/alobo/.local/lib/python3.10/site-packages/netCDF4
/home/alobo/.local/lib/python3.10/site-packages/netCDF4-1.6.4.dist-info
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs
/home/alobo/miniconda3/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4-1.6.2.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4
/home/alobo/miniconda3/pkgs/libnetcdf-4.8.1-nompi_hb3fd0d9_101/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4
/home/alobo/miniconda3/envs/qgis_stable/lib/cmake/netCDF
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4
find: ‘/home/alobo/miniconda3/lib/python39.zip’: No such file or directory
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs
/home/alobo/.local/lib/python3.10/site-packages/netCDF4
/home/alobo/.local/lib/python3.10/site-packages/netCDF4-1.6.4.dist-info
/home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs

So yes, it looks like there is a lot of netCDF4 cleaning to do... Any suggestions on how to proceed?

[thielfab]

I am confused why !python3 -m site returns the miniconda python, maybe the shell call is the wrong thing here as conda is the system default. What does this call from the QGIS Python console return?

import sys
sys.path

Does it show miniconda paths there also, or rather /usr/lib/python3.10?

[aloboa]

I am confused why !python3 -m site returns the miniconda python, maybe the shell call is the wrong thing here as conda is the system default. What does this call from the QGIS Python console return?

import sys
sys.path

i``` mport sys sys.path ['/usr/share/qgis/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/alobo/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python']

> 
> Does it show `miniconda` paths there also, or rather `/usr/lib/python3.10`?

No miniconda in the path, AFAICS.

[thielfab]

Then I assume if you run this from the normal shell (not QGIS) you will see a netcdf installation? /usr/bin/python3.10 -m pip list -v ? If it shows up and is listed with a path pointing to your home directory, try to uninstall it using /usr/bin/python3.10 -m pip uninstall netCDF4

[aloboa]

It turns out that my by default python version is 3.9.5

$ which python
/home/alobo/miniconda3/bin/python
(base) alobo@pop-os:~$ python
Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

Is this not odd? I thought python would always run the latest version.

If I do

$ /usr/lib/python3.10 -m pip list -v 
bash: /usr/lib/python3.10: Is a directory

I can use python3.10 -m pip list -v and then I see netCDF4 1.6.4 /home/alobo/.local/lib/python3.10/site-packages pip

Then

$ python3.10 -m pip uninstall netCDF4
Found existing installation: netCDF4 1.6.4
Uninstalling netCDF4-1.6.4:
  Would remove:
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4-1.6.4.dist-info/*
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libaec-e300f322.so.0.0.12
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libblosc-a9e8a7c7.so.1.21.1
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libcurl-47074032.so.4.8.0
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libhdf5-c6d05200.so.200.2.0
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libhdf5_hl-f45ebdf9.so.200.1.0
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libnetcdf-1281958a.so.19
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libsz-57467d8a.so.2.0.1
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4.libs/libzstd-c4cc7033.so.1.5.2
    /home/alobo/.local/lib/python3.10/site-packages/netCDF4/*
Proceed (Y/n)? Y
  Successfully uninstalled netCDF4-1.6.4

Unfortunately, even now, running import netCDF4 from within the QGIS python console crashes QGIS

I have found that

$ which python3
/home/alobo/miniconda3/bin/python3

thus run: $ python3 -m pip list -v

and there is no netCDF4 listed there

$ python3 -m pip uninstall netCDF4
WARNING: Skipping netCDF4 as it is not installed.

Yet, a miniconda3 netCDF4 is listed from within QGIS:

!python3 -m site | grep -o "/[^ ']*" | xargs -I% find % -type d -name "*netCDF*"
/home/alobo/my_repositories/enmap-box/tests/non-blocking-dependencies/netCDF4
/home/alobo/Dropbox/WikiMind/notes/Notebooks/ZimNotes/Home/Work/Software/SoftRSGISImProc/netCDF_(nc)
/home/alobo/Downloads/enmapboxplugin/enmapboxtesting/non-blocking-dependencies/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs
/home/alobo/miniconda3/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4-1.6.2.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.6.2-py39h89d13dc_0/lib/python3.9/site-packages/netCDF4
/home/alobo/miniconda3/pkgs/libnetcdf-4.8.1-nompi_hb3fd0d9_101/lib/cmake/netCDF
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/pkgs/netcdf4-1.5.8-nompi_py37hf784469_101/lib/python3.7/site-packages/netCDF4
/home/alobo/miniconda3/envs/qgis_stable/lib/cmake/netCDF
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4-1.5.8.dist-info
/home/alobo/miniconda3/envs/qgis_stable/lib/python3.7/site-packages/netCDF4
find: ‘/home/alobo/miniconda3/lib/python39.zip’: No such file or directory
/home/alobo/.local/lib/python3.8/site-packages/netCDF4-1.5.6.dist-info
/home/alobo/.local/lib/python3.8/site-packages/netCDF4
/home/alobo/.local/lib/python3.8/site-packages/netCDF4.libs

[thielfab]

There was an error in my reply where I confused lib and bin, I silently edited this several minutes after posting. But the way you proceeded yielded the same result I assume. Regarding QGIS and the conda python I would assume it should not be a problem, as the sys.path call inside the QGIS Python console does not show any references to conda. And here the ! shell commands do not seem reliable as it seems to forward it to the system shell with all its defaults, and there python3 is linked to the miniconda installation. What does sys.executable in the QGIS Python console return?

[aloboa]

There was an error in my reply where I confused lib and bin, I silently edited this several minutes after posting. But the way you proceeded yielded the same result I assume. Regarding QGIS and the conda python I would assume it should not be a problem, as the sys.path call inside the QGIS Python console does not show any references to conda. And here the ! shell commands do not seem reliable as it seems to forward it to the system shell with all its defaults, and there python3 is linked to the miniconda installation. What does sys.executable in the QGIS Python console return?

import sys.executable
Traceback (most recent call last):
  File "/usr/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'sys.executable'; 'sys' is not a package

import sys works

[thielfab]

Yes, sorry, I was a bit unprecise, I meant both import sys; sys.executable

[aloboa]

Yes, sorry, I was a bit unprecise, I meant both import sys; sys.executable

import sys; sys.executable
'/home/alobo/miniconda3/bin/python3'

and

$ /home/alobo/miniconda3/bin/python3 -m pip uninstall netCDF4
WARNING: Skipping netCDF4 as it is not installed.

[thielfab]

import sys; sys.executable '/home/alobo/miniconda3/bin/python3'

Now I'm confused. Did you run that from within QGIS python console? You are using a standard QGIS installation (installed via apt as described on qgis.org), not conda QGIS, right?

[thielfab]

It still puzzles me why the conda python is returned there. What result does this show (again, QGIS Python console)?

import numpy
numpy.__path__

I can think of two things to try:

1. Uninstall netCDF4 in conda (although I would be suprised if this works)

   conda activate base
   conda remove netCDF4

   or

2. Go conda all the way, I updated the installation instructions, see Option B: https://enmap-box.readthedocs.io/en/48-restructure-user-installation-section-by-operating-system/usr_section/usr_installation_new.html#qgis-installation-on-linux-ubuntu

[aloboa]

I realize I was using/usr/bin/qgis instead of the one from the environment you indicated yesterday, so I have deleted some of my previous answers and tried again. Some of the answers regarding paths may be different, but QGIS keeps crashing after import netCDF4.

From the linux console:

$ which python3
/home/alobo/miniconda3/bin/python3

Running the python console through the QGIS icon you indicated yesterday:

import sys; sys.executable
'/home/alobo/.virtualenvs/enmapbox/bin/python3'

import sys
sys.path
['/usr/share/qgis/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/alobo/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python']

import numpy
numpy.__path__
['/home/alobo/.local/lib/python3.10/site-packages/numpy']

[thielfab]

I realize I was using/usr/bin/qgis instead of the one from the environment you indicated yesterday

That was not the problem, if QGIS crashes even without the virtual environment, it means that there is still some netCDF installation which causes a problem and that has to be adressed before installing the additional python packages. So ignore the virtual environment thing for now. I assume the miniconda environment is somehow your default system python or some python related environment variables are linked to this conda installation. And, you also have a netCDF from a python3.8 installation in your home directory. Did you do a system upgrade at some point from 20.04 to 22.04? But this should not be the problem actually... Maybe try this: Open the Terminal, type conda deactivate, then check which python3, and then start qgis from this terminal session. Try to import netCDF4 in the QGIS Python console.

[aloboa]

No crash!

$ conda deactivate
alobo@pop-os:~$ which python3
/usr/bin/python3
alobo@pop-os:~$ qgis

# Python Console
# Use iface to access QGIS API interface or type '?' for more info
# Security warning: typing commands from an untrusted source can harm your computer
import sys
sys.path
['/usr/share/qgis/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/alobo/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python']
import numpy
import sys; sys.executable
'/usr/bin/python3'
>> import netCFD4
Traceback (most recent call last):
  File "/usr/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'netCFD4'

[thielfab]

Ok, that's good, now try to repeat the installation steps starting with the virtual environment (step 5). You may just delete the existing ~/.virtualenvs/enmapbox. Before you start, make sure the first thing you type in the terminal session is conda deactivate. Also start QGIS from this session and see if the EnMAP-Box starts and "everything" works.

[aloboa]

(base) alobo@pop-os:~$ cd .virtualenvs
(base) alobo@pop-os:~/.virtualenvs$ ls
enmapbox  rgee
(base) alobo@pop-os:~/.virtualenvs$ rm -r enmapbox
(base) alobo@pop-os:~/.virtualenvs$ conda deactivate
alobo@pop-os:~/.virtualenvs$ which python3
/usr/bin/python3
alobo@pop-os:~/.virtualenvs$ python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox
alobo@pop-os:~/.virtualenvs$ cd enmapbox
alobo@pop-os:~/.virtualenvs/enmapbox$ source ~/.virtualenvs/enmapbox/bin/activate
(enmapbox) alobo@pop-os:~/.virtualenvs/enmapbox$ python3 -m pip install -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt
Requirement already satisfied: typeguard==2.13.3 in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 2)) (2.13.3)
Requirement already satisfied: astropy in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (5.3.4)
Requirement already satisfied: PyOpenGL in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 4)) (3.1.7)
Requirement already satisfied: xgboost in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 5)) (2.0.0)
Requirement already satisfied: lightgbm in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 6)) (4.1.0)
Requirement already satisfied: catboost in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.2.2)
Requirement already satisfied: sympy in /usr/lib/python3/dist-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 10)) (1.9)
Requirement already satisfied: numba>=0.57 in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 11)) (0.58.0)
Requirement already satisfied: enpt-enmapboxapp in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 12)) (0.7.6)
Requirement already satisfied: scikit-learn>=1.0 in /home/alobo/.local/lib/python3.10/site-packages (from -r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 13)) (1.3.1)
Requirement already satisfied: packaging>=19.0 in /home/alobo/.local/lib/python3.10/site-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (23.2)
Requirement already satisfied: PyYAML>=3.13 in /usr/lib/python3/dist-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (5.4.1)
Requirement already satisfied: pyerfa>=2.0 in /home/alobo/.local/lib/python3.10/site-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (2.0.1)
Requirement already satisfied: numpy<2,>=1.21 in /home/alobo/.local/lib/python3.10/site-packages (from astropy->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 3)) (1.25.2)
Requirement already satisfied: scipy in /home/alobo/.local/lib/python3.10/site-packages (from xgboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 5)) (1.11.3)
Requirement already satisfied: matplotlib in /home/alobo/.local/lib/python3.10/site-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (3.8.0)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.16.0)
Requirement already satisfied: pandas>=0.24 in /home/alobo/.local/lib/python3.10/site-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2.1.1)
Requirement already satisfied: graphviz in /home/alobo/.local/lib/python3.10/site-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (0.20.1)
Requirement already satisfied: plotly in /usr/lib/python3/dist-packages (from catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (5.4.0)
Requirement already satisfied: llvmlite<0.42,>=0.41.0dev0 in /home/alobo/.local/lib/python3.10/site-packages (from numba>=0.57->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 11)) (0.41.0)
Requirement already satisfied: psutil in /home/alobo/.local/lib/python3.10/site-packages (from enpt-enmapboxapp->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 12)) (5.9.6)
Requirement already satisfied: importlib-metadata in /usr/lib/python3/dist-packages (from enpt-enmapboxapp->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 12)) (4.6.4)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/alobo/.local/lib/python3.10/site-packages (from scikit-learn>=1.0->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 13)) (3.2.0)
Requirement already satisfied: joblib>=1.1.1 in /home/alobo/.local/lib/python3.10/site-packages (from scikit-learn>=1.0->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 13)) (1.3.2)
Requirement already satisfied: tzdata>=2022.1 in /home/alobo/.local/lib/python3.10/site-packages (from pandas>=0.24->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2023.3)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/alobo/.local/lib/python3.10/site-packages (from pandas>=0.24->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3/dist-packages (from pandas>=0.24->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (2022.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.4.5)
Requirement already satisfied: cycler>=0.10 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (0.12.1)
Requirement already satisfied: pillow>=6.2.0 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (10.1.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (1.1.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (4.43.1)
Requirement already satisfied: pyparsing>=2.3.1 in /home/alobo/.local/lib/python3.10/site-packages (from matplotlib->catboost->-r https://raw.githubusercontent.com/EnMAP-Box/enmap-box/main/.env/linux/requirements.txt (line 7)) (3.1.1)
alobo@pop-os:~/.virtualenvs/enmapbox$ qgis

and QGIS crahes again with import netCDF4 If I uninstall netCDF4 before running QGIS

import netCDF4 does not crash. Therefore, I understand that your list of requirements is not correct and netCDF4 gets installed.

After uninstalling netCDF4, alobo@pop-os:~$ sudo python3 -m pip uninstall netCDF4 I can install enmapbox from within QGIS normally, but get:

EnMAPBoxApplication error(s): 
ensomap: 
ImportError:cannot import name 'docstring' from 'matplotlib' (/home/alobo/.local/lib/python3.10/site-packages/matplotlib/__init__.py)
Traceback:
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/gui/applications.py", line 368, in addApplicationFolder
apps = factory(self.mEnMAPBox)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/__init__.py", line 35, in enmapboxApplicationFactory
from ensomap.enmapboxintegration import EnSoMAP
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/enmapboxintegration.py", line 32, in 
import hys
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/hys/__init__.py", line 27, in 
from .gui import *
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps/ensomap/hys/gui.py", line 12, in 
import matplotlib.pyplot as plt
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/lib/python3.10/site-packages/matplotlib/pyplot.py", line 66, in 
from matplotlib.figure import Figure, FigureBase, figaspect
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/lib/python3.10/site-packages/matplotlib/figure.py", line 43, in 
from matplotlib import _blocking_input, backend_bases, _docstring, projections
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/alobo/.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py", line 58, in 
from mpl_toolkits.mplot3d import Axes3D
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in 
from .axes3d import Axes3D
File "/usr/lib/python3/dist-packages/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in 
from matplotlib import _api, cbook, docstring, _preprocess_data

[thielfab]

All the packages from the requirements.txt should NOT be in /home/alobo/.local/lib/python3.10/site-packages but somewhere in ~/.virtualenvs/enmapbox. So I assume at some point during the installation or rather at previous attempts you ran the pip install without the virtual environment being activated (indicated by the (enmapbox) at the beginning of the line). So the error shown is maybe a result of version mismatch between packages, matplotlib in this case. Does the EnMAP-Box start at least? If you do not use EnSoMAP you can ignore this error. I will check again if some package in the requirements file needs/installs netcdf, but it was not installed on my test system if I remember correctly.

[aloboa]

netCFD4 got installed for sure in the sequence of commands:

$ python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox
$ cd enmapbox
$ source ~/.virtualenvs/enmapbox/bin/activate

I can tell for sure because, as I wrote, QGIS was not crashing in the previous run ,and the problem was corrected by $ sudo python3 -m pip uninstall netCDF4

Perhaps the sequence should have been:

$ source ~/.virtualenvs/enmapbox/bin/activate
$ python3 -m venv --system-site-packages ~/.virtualenvs/enmapbox
$ cd enmapbox

Does the EnMAP-Box start at least?

AFICS, yes. At least I can display images and spectra.

[thielfab]

netCFD4 got installed for sure in the sequence of commands:

Can not confirm this on my Ubuntu 23.10 VM. I doubt that it is any different on 22.04.

As the automatic activation of the conda environment in bash is rather inconvenient in your case (I assume?), it might make sense to deactivate this behavior, via conda config --set auto_activate_base false You can still run conda activate anytime you need the conda environment. Also, I am not sure if that has had any effect on the QGIS installation (as python environment variables are altered). E.g., it surprises me that numpy is installed in /home/alobo/.local/lib/python3.10/site-packages, in my case its part of the dist-packages at /usr/lib/python3/dist-packages/numpy.

Does the EnMAP-Box start at least?

AFICS, yes. At least I can display images and spectra.

Good if that works, maybe check if the Classification workflow runs with the Example dataset, then most functionality should be there 👍