geodms.pyd fails to load due to gdal.dll version conflict (osgeo.gdal vs geodms gdal) if they are different versions

[eoudejans]

The following imports geodms successfully, but fails to import osgeo.gdal (version 3.9.0):

Scenario 1

import sys

sys.path.append('C:/Program Files/ObjectVision/GeoDms15.1.0/')
import geodms

from osgeo import gdal

The following imports the osgeo.gdal successfully, but fails to import the geodms gdal (version 3.8.1)

Scenario 2

import sys
from osgeo import gdal

sys.path.append('C:/Program Files/ObjectVision/GeoDms15.1.0/')
import geodms

Crash occurs as result of Stg.dll load, which in terms attempts to load gdal:

[eoudejans]

These dlls get loaded when first loading geodms (gdal 3.8.1):

Results generated using WinDbg (x64):

• Stop in breakpoint before including geodms
• Attach to python process
• step through in WinDbg until focus goes back to visual studio code
• take screenshot

[eoudejans]

When including osgeo.gdal (gdal 3.9.0) first, the output is:

ModLoad: 00007ffd73e80000 00007ffd73f5c000 c:\Users\Cicada\miniconda3\envs\py3_9_18\lib\site-packages\osgeo_gdal.cp39-win_amd64.pyd ModLoad: 00007ffc9bac0000 00007ffc9cee8000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\gdal.dll ModLoad: 00007ffd76930000 00007ffd769f0000 C:\Windows\SYSTEM32\ODBC32.dll ModLoad: 00007ffd77630000 00007ffd776c5000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libcurl.dll ModLoad: 00007ffd6d0d0000 00007ffd6d224000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\archive.dll ModLoad: 00007ffdaccb0000 00007ffdacd0e000 C:\Windows\System32\WLDAP32.dll ModLoad: 00007ffd6cdf0000 00007ffd6cf4b000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libxml2.dll ModLoad: 00007ffd7ce10000 00007ffd7ce40000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\deflate.dll ModLoad: 00007ffd7c770000 00007ffd7c791000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\liblz4.dll ModLoad: 00007ffd7cf40000 00007ffd7cf56000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\blosc.dll ModLoad: 00007ffd4ff20000 00007ffd50253000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\proj_9_3.dll ModLoad: 00007ffd6eff0000 00007ffd6f0b8000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\jpeg8.dll ModLoad: 00007ffd4c9a0000 00007ffd4cc4d000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\xerces-c_3_2.dll ModLoad: 00007ffdab740000 00007ffdabeab000 C:\Windows\System32\SHELL32.dll ModLoad: 00007ffd7a9d0000 00007ffd7a9fc000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\geotiff.dll ModLoad: 00007ffd7a1b0000 00007ffd7a1e5000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libpng16.dll ModLoad: 00007ffd6d9c0000 00007ffd6da44000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\Lerc.dll ModLoad: 00007ffd6d390000 00007ffd6d434000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\zstd.dll ModLoad: 00007ffd7f630000 00007ffd7f63d000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\aec.dll ModLoad: 00007ffd14b90000 00007ffd1505d000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\poppler.dll ModLoad: 00007ffd46f20000 00007ffd471da000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\cfitsio.dll ModLoad: 00007ffd50a50000 00007ffd50c0c000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\netcdf.dll ModLoad: 00007ffd77710000 00007ffd77772000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libwebp.dll ModLoad: 00007ffd59110000 00007ffd59281000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\sqlite3.dll ModLoad: 00007ffd700f0000 00007ffd70145000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\LIBPQ.dll ModLoad: 00007ffd6fd20000 00007ffd6fd7f000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libkea.dll ModLoad: 00007ffd70e10000 00007ffd70e5e000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\hdf5_cpp.dll ModLoad: 00007ffd6f590000 00007ffd6f5eb000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\openjp2.dll ModLoad: 00007ffc91800000 00007ffc927be000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\tiledb.dll ModLoad: 00007ffd6cfa0000 00007ffd6d02f000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\pcre2-8.dll ModLoad: 00007ffdac540000 00007ffdac595000 C:\Windows\System32\SHLWAPI.dll ModLoad: 00007ffc90b00000 00007ffc917f5000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\spatialite.dll ModLoad: 00007ffd75c70000 00007ffd75c95000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\freexl.dll ModLoad: 00007ffd6d950000 00007ffd6d9b9000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libexpat.dll ModLoad: 00007ffd6ca20000 00007ffd6ca87000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\geos_c.dll ModLoad: 00007ffd6a9d0000 00007ffd6aab9000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\iconv.dll ModLoad: 00007ffd6c520000 00007ffd6c59b000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\tiff.dll ModLoad: 00007ffd754e0000 00007ffd75507000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\mfhdf.dll ModLoad: 00007ffd6a7e0000 00007ffd6a888000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\hdf.dll ModLoad: 00007ffd36cf0000 00007ffd37062000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\hdf5.dll ModLoad: 00007ffd73e40000 00007ffd73e7f000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libssh2.dll ModLoad: 00007ffd78df0000 00007ffd78df9000 C:\Windows\SYSTEM32\WSOCK32.dll ModLoad: 00007ffd7cec0000 00007ffd7cec8000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\charset.dll ModLoad: 00007ffd7bef0000 00007ffd7bf08000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\snappy.dll ModLoad: 00007ffd59780000 00007ffd59827000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\freetype.dll ModLoad: 00007ffd5ae10000 00007ffd5ae9f000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\lcms2.dll ModLoad: 00007ffd736a0000 00007ffd736c2000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\hdf5_hl.dll ModLoad: 00007ffd7a9b0000 00007ffd7a9d0000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\zip.dll ModLoad: 00007ffd7ce00000 00007ffd7ce0c000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libsharpyuv.dll ModLoad: 00007ffd91640000 00007ffd9164c000 C:\Windows\SYSTEM32\Secur32.dll ModLoad: 0000000075d90000 0000000075df2000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\gssapi64.dll ModLoad: 00007ffd7c970000 00007ffd7c97b000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\crc32c.dll ModLoad: 00007ffcb23b0000 00007ffcb28a1000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\azure-storage-blobs.dll ModLoad: 00007ffd1c9c0000 00007ffd1caef000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\azure-core.dll ModLoad: 00007ffdaa4d0000 00007ffdaa4fe000 C:\Windows\SYSTEM32\USERENV.dll ModLoad: 00007ffd9bc00000 00007ffd9c0d6000 C:\Windows\SYSTEM32\WININET.dll ModLoad: 00007ffda3820000 00007ffda392a000 C:\Windows\SYSTEM32\WINHTTP.dll ModLoad: 00007ffdaa080000 00007ffdaa0a8000 C:\Windows\SYSTEM32\ncrypt.dll ModLoad: 00007ffd149c0000 00007ffd14b83000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\abseil_dll.dll ModLoad: 00007ffd70ba0000 00007ffd70bc1000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\libminizip.dll ModLoad: 00007ffcbca10000 00007ffcbcc46000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\geos.dll ModLoad: 00007ffd7c4a0000 00007ffd7c4a9000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\xdr.dll ModLoad: 00007ffdaa3d0000 00007ffdaa3da000 C:\Windows\SYSTEM32\DPAPI.DLL ModLoad: 00007ffd7c3d0000 00007ffd7c3dd000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\szip.dll ModLoad: 0000000075c60000 0000000075d8c000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\krb5_64.dll ModLoad: 0000000077350000 0000000077363000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\k5sprt64.dll ModLoad: 0000000077340000 0000000077348000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\comerr64.dll ModLoad: 00007ffda82f0000 00007ffda84d4000 C:\Windows\SYSTEM32\dbghelp.dll ModLoad: 00007ffd1ef20000 00007ffd1eff4000 C:\Users\Cicada\miniconda3\envs\py3_9_18\Library\bin\azure-storage-common.dll ModLoad: 00007ffdaa490000 00007ffdaa4c2000 C:\Windows\SYSTEM32\SSPICLI.DLL ModLoad: 00007ffda9a50000 00007ffda9b1b000 C:\Windows\SYSTEM32\DNSAPI.dll ModLoad: 00007ffd9a880000 00007ffd9a9d3000 C:\Windows\SYSTEM32\webservices.dll ModLoad: 00007ffdacee0000 00007ffdacee8000 C:\Windows\System32\NSI.dll ModLoad: 00007ffdaa040000 00007ffdaa07b000 C:\Windows\SYSTEM32\NTASN1.dll ModLoad: 00007ffd7bec0000 00007ffd7bece000 c:\Users\Cicada\miniconda3\envs\py3_9_18\lib\site-packages\osgeo_gdalconst.cp39-win_amd64.pyd ModLoad: 00007ffd189c0000 00007ffd18a79000 c:\Users\Cicada\miniconda3\envs\py3_9_18\lib\site-packages\osgeo_ogr.cp39-win_amd64.pyd ModLoad: 00007ffd598e0000 00007ffd59944000 c:\Users\Cicada\miniconda3\envs\py3_9_18\lib\site-packages\osgeo_osr.cp39-win_amd64.pyd

[eoudejans]

after: conda install gdal=3.8.1

both geodms and osgeo have the same gdal versions and no crash occurs with both scenario's:

[MaartenHilferink]

We should diagnose further what the actual problem is with loading both ObjectVsion\GeoDms15.1.0\gdal.dll (3.8.1.) and ...\microconda3\env...\gdal.dll (3.9.0) and why that is not an issue when both are 3.8.1. as the actual problem might indicate the feasibility of the following possible solutions:

1. in globio4env, require: conda install gdal=3.8.1 (now 3.1.4, email d.d. 10-4-2024)
2. fix loading two installs of gdal (being two versions or not).
3. let GeoDms Delay Load the gdal. dll
4. retrieve what went wrong when we did try to let geodms deal installation specific gdal versions and apply that experience here
5. distribute geodms as conda module (how?) so that geodms -> gdal link is made when installing the module, or updating any side of the arrow.
6. distribute geodms as python module (how?), idem

[MaartenHilferink]

@bakkenesm: can you let us know if solution 1 works for you, at least as temporary solution, as you might be able to revert to the environment of 10-4-2024 by conda install gdal=3.1.4.

We will pause for the moment and continue to work on this next week anyway.

[eoudejans]

We should diagnose further what the actual problem is with loading both ObjectVsion\GeoDms15.1.0\gdal.dll (3.8.1.) and ...\microconda3\env...\gdal.dll (3.9.0) and why that is not an issue when both are 3.8.1. as the actual problem might indicate the feasibility of the following possible solutions:

1. in globio4env, require: conda install gdal=3.8.1 (now 3.1.4, email d.d. 10-4-2024)

This solution leads to conflicts between gdal=3.8.1 and taudem:

...

3. let GeoDms Delay Load the gdal. dll

This solution is likely implementable in the short-term, and will be elegant as geodms does not need gdal for most common geodata formats, in this case gtiff. The only thing to be changed in the Globio-dms configuration would be the data io, switching from gdal storage manager to native tiff storage manager.

[MaartenHilferink]

I'll check why gdal (3.8.1) is loaded at all when initially loading GeoDMS, as it is linked as delay loaded. As we can run the globio allocation without gdal, using the native tiff driver for reading and writing results, we could also compile a setup for python usage that omits linking and installing gdal at all. This would be a separate configuration named "DllIOnly" aside from "Debug" and "Release", that supports being used from an existing process with minimal library restrictions.

[eoudejans]

There may still be a dependency conflict using libtiff, which is used by both gdal and geodms native tiff storage manager. But this can be tested when we have the delay load mechanism working again.

[eoudejans]

Adding delay loading of gdal and proj to clc and geo modules makes delay loading to work properly:

[eoudejans]

Confirming that the delay load route gets past initialization of geodms and importing gdal python module with different gdal version (3.9.0) than that of geodms (3.8.5).

[bakkenesm]

That is good news. Can I test this? Which version of the GeoDMS do I need to download?

[eoudejans]

Today I will be working on changing GLOBIO dms code to no longer use GDAL, I'll post updates here as soon as they are available

[eoudejans]

Brief update:

• Geodms has been updated to make sure gdal and proj are delay loaded.
• Globio dms allocation module gdal.grid storage manager occurences are replaced by geodms native tif storage manager
• Globio dms allocation module gdal.vect storage manager occurences are replaced by geodms native str storage manager

The above three changes allow the geodms python module to work along side other python modules that rely on a different gdal version than geodms uses.

This will require to make use of the latest commit of this globio geodms allocation repository and the upcoming prerelease of geodms.

[MaartenHilferink]

The above mentioned changes are now available in: https://github.com/ObjectVision/GeoDMS/releases/tag/v15.2.0

Unknown issues (yet to be tested): when the module globio and then geodms have been loaded and then through the geodms interface, a GDAL or PROJ function is called (by using a gdal StorageType or inter-projection conversion), an uncaught Windows Structured Exception might be thrown from the delay loaded function stub. Catching this exception and reporting it back to the called can be located in gdalComponent, gdalThread, and/or GDAL_ErrorFrame.

[eoudejans]

Steps to reproduce using geodms:

• Install miniconda https://docs.anaconda.com/free/miniconda/
• clone this (GLOBIO_dms) repository locally
• Start anaconda prompt and navigate to the GLOBIO_dms local repo, within the prompt:
  • Create conda environment if not done so already with python version 3.9.18: conda create -n globio_python_env python=3.9
  • Activate environment: conda activate globio_python_env
  • Install gdal: conda install gdal
  • Navigate to globio_dms_repo/examples/
  • Change path/to/dms/installation to the correct directory
  • Run globio_dms_repo/examples/example_01_run_dms_globio_alloc.py

[MaartenHilferink]

[MaartenHilferink]

the for_each for per_region could not be expanded as the name-range depends on %SourceDataDir%/GLOBIO/input/Regions.csv, which is read by the StrStorageManager.

[MaartenHilferink]

Result in Release version of running example_01_run_dms_globio_alloc.py: GeoDms has loaded gdal.dll, from Conda env, as well as Proj_9_3.dll, but initialisation of proj.dll fails.

Results in Debug version: Apparently, loading of gdal_d.dll fails as to incompatibility

[MaartenHilferink]

Now loading the release built of the GeoDMS provides a similar error on loading the proj.dll: