32-bit dll not found (or one of its dependencies)

[pfjarschel]

Hi! It's been a while since I last used your library. It has helped me a lot in the past. Recently, I wanted to use a script that I used to use a lot in the past, and always worked fine, and I kept getting this error:

Instantiating 'PSLInGaAs32' raised the following exception:

Traceback (most recent call last):
  File "PyInstaller\loader\pyimod03_ctypes.py", line 53, in __init__
  File "ctypes\__init__.py", line 376, in __init__
FileNotFoundError: Could not find module 'C:\Generic\Path\To\snakecamlinkcontrol.dll' (or one of its dependencies). Try using the full path with constructor syntax.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "start_server32.py", line 183, in main
  File "C:\Generic\Path\To\PSLInGaAs32.py", line 22, in __init__
    super(PSLInGaAs32, self).__init__("./snakecamlinkcontrol.dll", 'cdll', host, port)
  File "server32.py", line 72, in __init__
  File "load_library.py", line 162, in __init__
  File "PyInstaller\loader\pyimod03_ctypes.py", line 55, in __init__
pyimod03_ctypes.install.<locals>.PyInstallerImportError: Failed to load dynlib/dll 'C:\\Generic\\Path\\To\\snakecamlinkcontrol.dll'. Most likely this dynlib/dll was not found when the application was frozen.

Cannot start the 32-bit server.

After spending quite a few hours to debug the problem, I found out that downgrading from version 0.10.0 to version 0.9.0 solved the issue. I tried to find what changed, and if it was something I should change on my code to "update" it, but to no avail. So I ask you: what could be the problem? I suspect it is a problem that many people will have, so it would be good to understand what is causing it.

Note: Using Python 3.11.4 on Windows 11. I also tried using the full path and moving the dll to a "simpler" location, but nothing helped.

Thank you!

Edit: I understand it could possibly be due to how the included 32-bit python was frozen, but honestly, I don't see how a simple user of the library could go on to fix this. Re-freezing seems to be the way to go, but without understanding exactly what to change during the process, it can be a very tough challenge, so downgrading is just simpler and saves a lot of time at this point.

[jborbely]

Thank you for raising this issue.

Is https://github.com/pfjarschel/PhotonicScience_InGaAsCam the code that you are running that causes the error with version 0.10.0 installed? If not, would you be able to provide a minimal example that shows how the Client64 and Server32 are implemented?

[pfjarschel]

Thank you for taking the time to look into it!

Good catch! I created a repository mostly to share the code, and ended up forgetting it was there! But not really, that one is a little more complicated, as I was trying to reverse-engineer a driver for the camera I am trying to access. It is unnecessarily complicated, so on my day-to-day use, I ended up creating a simple interface for the driver instead. The snakecamlinkcontrol.dll in that repo is the correct one, though. It also depends on some other dlls available from an old package named Pleora eBus Vision Package. Unfortunately, it is a closed software, and it seems it is not freely available anymore. (That's the reason I was trying to reverse engineer it).

Here is a very simple example:

from msl.loadlib import Server32, Client64
import os, ctypes

class PSLInGaAs32(Server32):
    """Wrapper around a 32-bit C++ library 'snakecamlinkcontrol.dll'."""
    def __init__(self, host, port, **kwargs):
        full_path = os.path.join(os.path.dirname(__file__), "snakecamlinkcontrol.dll")
        super(PSLInGaAs32, self).__init__(full_path, 'cdll', host, port)

    def init_iface(self, hgain=0):
        caminit = self.lib.PSL_VHR_Init()

class PSLInGaAs64(Client64):
    """Call functions in 'snakecamlinkcontrol.dll' via the 'PSLInGaAs32' wrapper."""
    def __init__(self, hgain = 0):
        # Specify the name of the Python module to execute on the 32-bit server (i.e., 'my_server')
        super(PSLInGaAs64, self).__init__(module32='PSLInGaAs32')
        self.ifaceOK = self.request32('init_iface', hgain)

cam = PSLInGaAs64(0)

[jborbely]

I suppose your simple example was written quickly (and untested) because it will not work in any version of msl-loadlib. The main reason being that both the client and server are defined in the same file, which must have the filename of PSLInGaAs32.py based on PSLInGaAs64 using the value of module32='PSLInGaAs32' in the call to super(), and so the line

cam = PSLInGaAs64(0)

will also be executed by the 32-bit server when the module is imported by the server. There is nothing wrong with defining the client and server in the same file, but you must protect what code gets executed by the 32-bit process (server) and by the 64-bit process (client). To fix the main issue with your simple example you must use the following so that an instance of PSLInGaAs64 is not created on the 32-bit server

if __name__ == '__main__':
    cam = PSLInGaAs64(0)

From what I can infer from your example, there is nothing that would cause a difference with the library loading. I suspect that your issue is that a dependency of snakecamlinkcontrol.dll is not on PATH. Here's an example script that you can run to determine what's different when you use 0.9.0 vs 0.10.0 on your system

# issue41.py
import os
from msl.loadlib import Server32, Client64

class PSLInGaAs32(Server32):
    def __init__(self, host, port):
        full_path = os.path.join(Server32.examples_dir(), 'cpp_lib32.dll')
        super(PSLInGaAs32, self).__init__(full_path, 'cdll', host, port)

    def os_path(self):
        return os.environ['PATH'].split(os.pathsep)

class PSLInGaAs64(Client64):
    def __init__(self):
        super(PSLInGaAs64, self).__init__(module32=__file__)

    def os_path(self):
        return self.request32('os_path')

if __name__ == '__main__':
    cam = PSLInGaAs64()
    for path in cam.os_path():
        print(path)
    cam.shutdown_server32()

then run

pip install msl-loadlib==0.9.0
python issue41.py
pip install msl-loadlib==0.10.0
python issue41.py

The paths that are printed when 0.9.0 is installed and when 0.10.0 is installed should be the same (they are when I run it). Are they the same for you?

Are you able to provide an example that I can run which illustrates that some example code works using msl-loadlib 0.9.0 but not with 0.10.0? I understand that some of the dependencies of snakecamlinkcontrol.dll cannot be shared, but perhaps you can create a mock DLL library that behaves similarly and share the example via a repository that I can clone.

[pfjarschel]

Thank you for your reply! You are correct, the example was stitched together from some snippets I took from two separate files, sorry about that! Just after I posted, I realized they should not be in the same file. Well, indeed your suggested modification fixes the issue with the example. I ran your example (issue41) and yes, all the paths printed are exactly the same using the two versions. I would be really surprised if they weren't!

Regarding an example you can run, I will try to work on something here. Is it alright if I share the required dlls with you? That might be the fastest way to allow you to run tests on your end.

Just to provide you with context, this snakecamlink dll is very messy, poorly documented (actually, not documented at all), so it was a challenge to get a simple interface with it working! The camera manufacturer use this dll in their proprietary software (that requires a dongle to run), and this dll interfaces the driver of an ethernet camera interface, another proprietary software from a different company and currently impossible to get unless you buy a new version. So it's all very messed up, and msl-loadlib is currently the only way to provide full functionality for the camera. I was able to create a "new driver" for basic frame grabbing, but without all the camera configuration functions. I hope I can eventually complete this project, but your package has been a life saver so far!

[jborbely]

Yes, you can share the required DLLs with me. You can send the information privately if you wish, see here for contact details. Email filters will probably block DLLs as attachments, so you can send a link to download the files.

You can also try a few more things on your end.

1) Ask yourself: Are the dependencies of snakecamlink.dll available in the paths that were printed when running my example issue41.py script?

2) The server bundled with msl-loadlib version 0.9.0 runs on Python 3.7.10 and the server bundled with version 0.10.0 runs on Python 3.11.4. You can install 32-bit Python versions from python.org/ftp and temporarily remove msl-loadlib as being a potential source of the issue (note: 3.7.9 was the last version to provide 32-bit binaries so install this version or build 3.7.10 from source or download the 3.7.10 installer from this repo -- I am not affiliated with that repo, so use your own judgement before choosing this option).

Once you install 32-bit versions of Python 3.7.9 (or 3.7.10) and 3.11.4, cd to the directory where your code is located and perform the following in each interactive console

>>> import ctypes
>>> ctypes.CDLL('snakecamlink.dll')

That is essentially how the 32-bit server loads your library. Since you do not use the append_environ_path keyword argument in your Client64 subclass, you do not need to modify the PATH environment available in the terminal you invoke the interactive console from.

3) If step 2 works for both 3.7.x and 3.11.4, you can repeat the process by directly using the 32-bit server bundled with msl-loadlib. Run the following commands

pip install msl-loadlib==0.9.0
python -c "from msl.loadlib import Server32; Server32.interactive_console()"

then, in the interactive console that pops up enter

>>> import ctypes
>>> ctypes.CDLL('snakecamlink.dll')

then

pip install msl-loadlib==0.10.0
python -c "from msl.loadlib import Server32; Server32.interactive_console()"

then

>>> import ctypes
>>> ctypes.CDLL('snakecamlink.dll')

[pfjarschel]

I was about to reply to the topic when a notification of your reply popped up. Thank you very much for your suggestions! Indeed, the dll is loaded correctly using python version 3.7.9 (32 bit), but did not load correctly using 3.11.4 (32 bit).

It might make sense with the reply I was about to post:

While trying to determine the exact dlls that are dependencies of the problematic one, I figured something out. The camera ethernet interface manufacturer provides both 32 and 64 bit dlls. The installer inserts both locations for these dlls in the PATH. To run the simplest test possible, I removed them from the path, and copied the dlls from the 32 bit location to the test folder. Everything ran fine on 0.9. When I updated again to 0.10, voila, everything kept working fine! But, if I don't put the dlls on the same folder as the script, it doesn't work. I tried removing the 64 bit folder from the path, and also manually adding a custom folder with the correct dlls to the path. None of these tests worked on version 0.10 (everything is fine in 0.9).

Now with the knowledge that ctypes only load the dll correctly with an older version of python, it is clear that the issue is not related to your package. I tried searching for this issue but now specifying ctypes dll loading, and, yes, the issue was introduced on python 3.8, which loads dll dependencies more securely (see this stack overflow answer ).

I did what was suggested on that post:

import os
os.add_dll_directory("Path/to/dependencies/dlls")

And it worked fine.

So, to sum everything up: The issue I was having is introduced by a security update that started on python 3.8. The bottom line is: Know your dlls, and manually add the dependencies paths to the os search path! It can be done on the init function of the 32 bit server subclass, before the super initialization.

With this, I believe the issue can be closed, as there is no action from your end that could "fix" it, besides maybe adding all the folders in the PATH using the above code during the 32 bit server initialization. I don't think that's a good idea, though, and I'm sure you'd agree!

Thank you very much for the helpful comments and suggestions! With them, I was able to pinpoint the exact cause and find a solution for my case, that can possibly help future users.

[jborbely]

Thank you for your feedback and for the prompt replies.

Perhaps I should add a add_dll_directory kwarg to Client64 to perform the action on the server if someone wants to. At least that will provide a way for me to document the DLL search path for people loading a library on Windows.