cffi.FFI().dlopen gives error 0x7e on Windows

[Wainberg]

Opening the DLL for the R programming language fails with CFFI on Windows on my machine:

>>> import cffi
>>> dll = 'C:\\Users\\Wainberg\\miniforge3\\lib\\R\\bin\\x64\\R.dll'
>>> cffi.FFI().dlopen(dll)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Wainberg\miniforge3\Lib\site-packages\cffi\api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Wainberg\miniforge3\Lib\site-packages\cffi\api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Wainberg\miniforge3\Lib\site-packages\cffi\api.py", line 828, in _load_backend_lib
    return backend.load_library(path, flags)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: cannot load library 'C:\Users\Wainberg\miniforge3\lib\R\bin\x64\R.dll': error 0x7e

But opening the same DLL with ctypes first "magically" makes it open with CFFI as well (note: import cffi is not enough, you have to call ctypes.CDLL(dll) as well):

>>> import ctypes
>>> import cffi
>>> dll = 'C:\\Users\\Wainberg\\miniforge3\\lib\\R\\bin\\x64\\R.dll'
>>> ctypes.CDLL(dll)
<CDLL 'C:\Users\Wainberg\miniforge3\lib\R\bin\x64\R.dll', handle 6c700000 at 0x16358ac1460>
>>> cffi.FFI().dlopen(dll)
<cffi.api._make_ffi_library.<locals>.FFILibrary object at 0x000001635A832EA0>

I am using the base environment of Miniforge (https://github.com/conda-forge/miniforge) for this, but I get the same 0x7e error on my system Python installation. Although there, ctypes.CDLL() also gives an error:

>>> ctypes.CDLL(dll)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\ctypes\__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'C:\Users\Wainberg\miniforge3\lib\R\bin\x64\R.dll' (or one of its dependencies). Try using the full path with constructor syntax.

Using ctypes.util.find_library(dll) doesn't help:

>>> dll
'C:\\Users\\Wainberg\\miniforge3\\lib\\R\\bin\\x64\\R.dll'
>>> ctypes.util.find_library(dll)
'C:\\Users\\Wainberg\\miniforge3\\lib\\R\\bin\\x64\\R.dll'

[Wainberg]

I was able to get this working by adding the directory containing the DLL to the PATH before the call to cffi.FFI().dlopen():

os.environ['PATH'] += ';C:\\Users\\Wainberg\\miniforge3\\lib\\R\\bin\\x64'

I would still consider this a bug since it shouldn't be necessary to manually add it to the PATH first. It's not for ctypes.

[arigo]

It probably didn't change recently, but can you give us your Python version for reference? (for ctypes)

[Wainberg]

$ python
Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:42:31) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.__version__
'1.1.0'
>>> import cffi
>>> cffi.__version__
'1.16.0'

[arigo]

I think that ctypes passes some flags to LoadLibraryExW() which we don't. One of the flags is "if this DLL has got dependencies, you can find them in the same directory". So calling ctypes.CDLL() first loads the DLL and all its dependencies, and that's why a future ffi.dlopen() will succeed.

I guess we should pass by default the same flags as ctypes does, on Windows. We could also support giving explicit flags: ffi.dlopen() accepts a flags argument but ignores it completely on Windows.

[arigo]

Does this help? https://github.com/python-cffi/cffi/pull/65

[Wainberg]

That fixes it! After installing your fork of cffi, I don't need to use the ctypes hack anymore.

[arigo]

OK! Added some documentation and merging.