Cannot install supy using new OSGeo version (QGIS 3.20)

[biglimp]

I cannot use either pip install supy outside as well as inside of a QGIS session. Error that comes up when try (I forced the installation to stop as it goes on trying to install other versions of tables:

C:\Users\xlinfr\Documents\PythonScripts\UMEP>pip install supy
Collecting supy
  Using cached supy-2021.5.26-py3-none-any.whl (400 kB)
Requirement already satisfied: scipy in c:\osgeo4w\apps\python39\lib\site-packages (from supy) (1.6.2)
Collecting numdifftools
  Using cached numdifftools-0.9.40-py2.py3-none-any.whl (99 kB)
Collecting supy-driver==2021a2
  Using cached supy_driver-2021a2-cp39-cp39-win_amd64.whl (2.0 MB)
Collecting f90nml
  Using cached f90nml-1.3.1.tar.gz (62 kB)
Collecting lmfit
  Downloading lmfit-1.0.2.tar.gz (306 kB)
     |████████████████████████████████| 306 kB 6.4 MB/s
Collecting atmosp
  Using cached atmosp-0.2.9-py3-none-any.whl (104 kB)
Collecting platypus-opt==1.0.4
  Using cached Platypus_Opt-1.0.4-py3-none-any.whl (70 kB)
Collecting dask
  Downloading dask-2021.6.2-py3-none-any.whl (973 kB)
     |████████████████████████████████| 973 kB 6.4 MB/s
Requirement already satisfied: matplotlib in c:\osgeo4w\apps\python39\lib\site-packages (from supy) (3.3.2)
Collecting multiprocess
  Downloading multiprocess-0.70.12.2-py39-none-any.whl (128 kB)
     |████████████████████████████████| 128 kB 6.4 MB/s
Requirement already satisfied: click in c:\osgeo4w\apps\python39\lib\site-packages (from supy) (8.0.1)
Collecting xarray
  Using cached xarray-0.18.2-py3-none-any.whl (807 kB)
Collecting cdsapi
  Using cached cdsapi-0.5.1.tar.gz (12 kB)
Collecting seaborn
  Using cached seaborn-0.11.1-py3-none-any.whl (285 kB)
Collecting tables
  Using cached tables-3.6.1.tar.gz (4.6 MB)
    ERROR: Command errored out with exit status 1:
     command: 'C:\OSGeo4W\bin\python3.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xlinfr\\AppData\\Local\\Temp\\pip-install-o67ykg5j\\tables_504b0e1b54a74161a0d00c838cfa0d50\\setup.py'"'"'; __file__='"'"'C:\\Users\\xlinfr\\AppData\\Local\\Temp\\pip-install-o67ykg5j\\tables_504b0e1b54a74161a0d00c838cfa0d50\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\xlinfr\AppData\Local\Temp\pip-pip-egg-info-1b6kds_o'        
         cwd: C:\Users\xlinfr\AppData\Local\Temp\pip-install-o67ykg5j\tables_504b0e1b54a74161a0d00c838cfa0d50\
    Complete output (10 lines):
    H5closenggyiar6.c
    C:\Users\xlinfr\AppData\Local\Temp\H5closenggyiar6.c(2): warning C4013: 'H5close' undefined; assuming extern returning int
    LINK : fatal error LNK1181: cannot open input file 'hdf5.lib'
    * Using Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)]
    * USE_PKGCONFIG: False
    * Found HDF5 using system PATH ('C:\OSGeo4W\bin')
    .. ERROR:: Could not find a local HDF5 installation.
       You may need to explicitly state where your local HDF5 headers and
       library can be found by setting the ``HDF5_DIR`` environment
       variable or by using the ``--hdf5`` command-line option.
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/2b/32/847ee3f521aae6a0be380d923a736162d698586f444df1ac24b98c65025c/tables-3.6.1.tar.gz#sha256=49a972b8a7c27a8a173aeb05f67acb45fe608b64cd8e9fa667c0962a60b71b49 (from https://pypi.org/simple/tables/) (requires-python:>=3.5). Command errored out with exit status 1: python setup.py egg_info Check 
the logs for full command output.
  Using cached tables-3.5.2.tar.gz (7.8 MB)
    ERROR: Command errored out with exit status 1:
     command: 'C:\OSGeo4W\bin\python3.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\xlinfr\\AppData\\Local\\Temp\\pip-install-o67ykg5j\\tables_d95c7a61748e4498a736540f7dba9061\\setup.py'"'"'; __file__='"'"'C:\\Users\\xlinfr\\AppData\\Local\\Temp\\pip-install-o67ykg5j\\tables_d95c7a61748e4498a736540f7dba9061\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\xlinfr\AppData\Local\Temp\pip-pip-egg-info-xqxixj5b'        
         cwd: C:\Users\xlinfr\AppData\Local\Temp\pip-install-o67ykg5j\tables_d95c7a61748e4498a736540f7dba9061\
    Complete output (10 lines):
    H5closec8bafxw0.c
    C:\Users\xlinfr\AppData\Local\Temp\H5closec8bafxw0.c(2): warning C4013: 'H5close' undefined; assuming extern returning int
    LINK : fatal error LNK1181: cannot open input file 'hdf5.lib'
    * Using Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)]
    * USE_PKGCONFIG: False
    * Found HDF5 using system PATH ('C:\OSGeo4W\bin')
    .. ERROR:: Could not find a local HDF5 installation.
       You may need to explicitly state where your local HDF5 headers and
       library can be found by setting the ``HDF5_DIR`` environment
       variable or by using the ``--hdf5`` command-line option.
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/73/51/6dabb2b94826e5db3aa2542b80f1382780b96a0cd13e0cfb637b36ede5c5/tables-3.5.2.tar.gz#sha256=b220e32262bab320aa41d33125a7851ff898be97c0de30b456247508e2cc33c2 (from https://pypi.org/simple/tables/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached tables-3.5.1.tar.gz (8.3 MB)
ERROR: Operation cancelled by user
WARNING: You are using pip version 21.1.1; however, version 21.1.3 is available.
You should consider upgrading via the 'C:\OSGeo4W\bin\python3.exe -m pip install --upgrade pip' command.

C:\Users\xlinfr\Documents\PythonScripts\UMEP> 

[sunt05]

@biglimp please try 2021.7.1 one hour later when it will be online.

[biglimp]

Still not possible to install within a QGIS session but now possible to install via OSGeoShell outside of a session.

Maybe this helps:

Couldn't load plugin 'UMEP' due to an error when calling its classFactory() method 

subprocess.CalledProcessError: Command '['C:\\OSGeo4W\\apps\\Python39\\pythonw.exe', '-m', 'pip', 'install', 'netCDF4', '-U', '--user']' returned non-zero exit status 1. 
Traceback (most recent call last):
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\suewsmodel\__init__.py", line 9, in 
    import supy as sp
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 799, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'supy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\suewsmodel\supy_installer.py", line 102, in setup_supy
    import supy as sp
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 799, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'supy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 335, in _startPlugin
    plugins[packageName] = package.classFactory(iface)
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\__init__.py", line 34, in classFactory
    from .UMEP import UMEP
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 799, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\UMEP.py", line 50, in 
    from .SuewsSimple.suews_simple import SuewsSimple
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 799, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\SuewsSimple\suews_simple.py", line 35, in 
    from ..suewsmodel import suews_wrapper
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 799, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\suewsmodel\__init__.py", line 17, in 
    setup_supy(ver=None)
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\suewsmodel\supy_installer.py", line 108, in setup_supy
    install_supy(ver)
  File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\suewsmodel\supy_installer.py", line 58, in install_supy
    str_info0 = subprocess.check_output(
  File "C:\OSGeo4W\apps\Python39\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\OSGeo4W\apps\Python39\lib\subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\OSGeo4W\\apps\\Python39\\pythonw.exe', '-m', 'pip', 'install', 'netCDF4', '-U', '--user']' returned non-zero exit status 1.

Python version: 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] 
QGIS version: 3.20.0-Odense Odense, decaadbb31 

Python Path:
C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\SUEWSPrepare/Modules
C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\SUEWSPrepare/Modules
C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\processing_fusion
C:/OSGeo4W/apps/qgis/./python
C:/Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:/Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins
C:/OSGeo4W/apps/qgis/./python/plugins
C:\OSGeo4W\bin\python39.zip
C:\OSGeo4W\apps\Python39\DLLs
C:\OSGeo4W\apps\Python39\lib
C:\OSGeo4W\bin
C:\Users\xlinfr\AppData\Roaming\Python\Python39\site-packages
C:\OSGeo4W\apps\Python39
C:\OSGeo4W\apps\Python39\lib\site-packages
C:\OSGeo4W\apps\Python39\lib\site-packages\win32
C:\OSGeo4W\apps\Python39\lib\site-packages\win32\lib
C:\OSGeo4W\apps\Python39\lib\site-packages\Pythonwin
C:/Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python

[sunt05]

just had a test with QGIS3.20.1 on Mac and got no issue: UMEP works smoothly with required python dependencies installed as expected during the initialisation.

just reading through the error message again and found this line suspicious:

subprocess.CalledProcessError: Command '['C:\\OSGeo4W\\apps\\Python39\\pythonw.exe', '-m', 'pip', 'install', 'netCDF4', '-U', '--user']' returned non-zero exit status 1.

@biglimp can you try to run this line in your windows OSGeo terminal and let us know the results?

C:\\OSGeo4W\\apps\\Python39\\pythonw.exe -m pip install netCDF4 -U --user

[biglimp]

I tried and pythonw.exe didnt work. I changed to python.exe and got the following error:

However, if I use pip install netCDF4, the package is installed without errors.

[biglimp]

@sunt05, your pull request only worked halfway.

Actually, I found that it is not QGIS version that decides where the python is located, it is the actual python version. Hence, we should check for python version and if we have 3.9 should do as we do now, when we identify QGIS 3.20. If lower version of python we should use the old location of python.

I noticed this as I installed the long term version of QGIS (3.16) alongside 3.20. 3.16 is using the same python version (3.9.5)

[biglimp]

Fixed. Now Python version is used instead of QGIS version