Unable to import module due to python unable to resolve dependecies

[3c400886-56e8-4c72-a3cd-dd3cdf1f8465]

BPO	40906
Nosy	@pfmoore, @tjguk, @zware, @zooba, @SabaKauser, @schribl
Files	python3.8_1.png python3.7.png python3.8_2.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['3.8', 'invalid', 'type-bug', 'expert-installation', 'OS-windows'] title = 'Unable to import module due to python unable to resolve dependecies' updated_at = user = 'https://github.com/SabaKauser' ``` bugs.python.org fields: ```python activity = actor = 'zach.ware' assignee = 'none' closed = True closed_date = closer = 'zach.ware' components = ['Installation', 'Windows'] creation = creator = 'sabakauser' dependencies = [] files = ['49220', '49221', '49222'] hgrepos = [] issue_num = 40906 keywords = [] message_count = 6.0 messages = ['370964', '371078', '371113', '371177', '371244', '371332'] nosy_count = 6.0 nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'sabakauser', 'schribl'] pr_nums = [] priority = 'normal' resolution = 'not a bug' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue40906' versions = ['Python 3.8'] ```

[3c400886-56e8-4c72-a3cd-dd3cdf1f8465]

Hi, I am building python ibm_db C extension for Python 3.8 support. while the binary is generated successfully and installed to site-packages, I am unable to load the same. The error I get is:

Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ibm_db
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing ibm_db: The specified module could not be found.
>>> quit()

I have correctly set PATH and LIB to point to python installation. I have also updated LIB to point to correct runtime dependencies. Seeing through procmon, python is unable to resolve the runtime lib dependency path from PATH. The same steps work just fine for python 3.7 and other versions. This problem is only seen on windows. Linux and MAC works fine.

I am attaching the process monitor snippets for python 3.8(failing case) as well as python (3.7) success case.

Can you kindly look into it and share insights on what could lead to this problem and any possible resolution/s that are currently available. I tried with python 3.8.2 as same problem there as well.

Attached are the procmon logs for python 3.7(success case) and python 3.8(failing case).

python 3.7 : python3.7.log python 3.8 : python3.8_1.log , python3.8_2.log

[06b544ef-ac00-47eb-9823-9890996bdf53]

From what I can see this behavior is intended starting with Python 3.8 for Windows and DLL loading for extension modules.

You can use add_dll_directory to add directories to the DLL load path (see https://docs.python.org/3/library/os.html#os.add_dll_directory). More information can be found e.g. here https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew

[3c400886-56e8-4c72-a3cd-dd3cdf1f8465]

Thanks a ton! That worked.. As I understand, users are required to add the directory path of the dll. Is there any way I can set this while building the extension with dependency? I basically want to hard-code the dependency path so that import module works fine without additional settings like you mentioned.

Thank you!

[3c400886-56e8-4c72-a3cd-dd3cdf1f8465]

Hello Steve, You have added changes to load windows dependent DLLs for python C extension modules. After building the source, I can import the module only after I set dll path as: os.add_dll_directory('C:\\Program Files\\IBM\\IBM DATA SERVER DRIVER_01\\bin')

Is there a backward compatibility or any other way I can hardcode the dependency resolution path while building the extension itself.

I have read through the discussion in https://bugs.python.org/issue36085 but I don't seem to understand if there is a real way for users to import the module without any additional step like os.add_dll_directory() from python.

Thanks!

[zooba]

add_dlldirectory can be safely used within your library, so if you have any .py files that will be imported first (such as your package \_init__.py), you can add it then. The safest way is to use something like:

with os.add_dll_directory(THE_DIR):
    import the_module

As this won't leave your directory on the search path for anyone else's DLLs.

The most correct fix is to put your DLL alongside your .pyd file. But I suspect in your case, the DLLs can't be moved, so using the search directory is likely best. (Or you could put your .pyd with the DLLs and update sys.path for when you import it.)

Presumably in the past, you were relying on the DLLs being on the user's PATH, which makes your application vulnerable to DLL hijacking (malicious or accidental, we've seen both). Not having to worry about that is a good thing.

[3c400886-56e8-4c72-a3cd-dd3cdf1f8465]

Thanks Steve!