Import error: No module named 'ffcv._libffcv'

[thom-OrigamiLabs]

Hi,

I'm trying to use FFCV on Windows 11 but I'm failing on initial import:

import ffcv Traceback (most recent call last): File "", line 1, in File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv__init.py", line 1, in from .loader import Loader File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\loader\init.py", line 1, in from .loader import Loader, OrderOption File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\loader\loader.py", line 14, in from ffcv.fields.base import Field File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\fields\init.py", line 1, in from .base import Field File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\fields\base.py", line 6, in from ..pipeline.operation import Operation File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\pipeline\init.py", line 2, in from .pipeline_spec import PipelineSpec File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\pipeline\pipeline_spec.py", line 5, in from ..transforms.module import ModuleWrapper File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\transforms\init__.py", line 5, in from .random_resized_crop import RandomResizedCrop File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\transforms\random_resized_crop.py", line 5, in from .utils import fast_crop File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\transforms\utils\fast_crop.py", line 4, in from ...libffcv import ctypes_resize File "C:\Users\Thom\mambaforge\envs\leo\lib\site-packages\ffcv\libffcv.py", line 6, in import ffcv._libffcv ImportError: DLL load failed while importing _libffcv: The specified module could not be found.

I've followed the instructions for Windows and have the file _libffcv.cp310-win_amd64.pyd in my site-packages\ffcv if that's relevant. This is in a fresh conda env with only mosiacml also installed.

Thanks,

Thom

[thom-OrigamiLabs]

Update: it looks like an issue with either finding the .pyd file or its dependencies. Modifying libffcv.py in the following way is a workaround:

Old

import ffcv._libffcv lib = CDLL(ffcv._libffcv.__file__)

New

lib = CDLL(r"path/to/ffcv/_libffcv.cp310-win_amd64.pyd", winmode=0)

The winmode==0 came from here and appears to be the actual fix for finding the dependencies. I'll have a look at making this more robust, as I'm currently hard coding it.

Note that to then do any file writing on windows, the DatasetWriter class needs to be wrapped in if __name__ == '__main__': to get multiprocessing to work.

I've created some datasets now but haven't tried using them, so it could still be broken!

[andrewilyas]

Hi @thom-OrigamiLabs ! Thanks for looking into this - unfortunately our time is a bit too limited to have official support for Windows, but if you figure it out happy to merge a PR into the next version.

[SerezD]

Hi, I have encountered the same Import Error on the Linux server I am using, so I guess this is not only a Windows problem.

On my local machine, everything works fine. The specs are:

Operating System: Ubuntu 22.04.2 LTS              
Kernel: Linux 5.19.0-46-generic
Architecture: x86-64

On the server machine, I got the Import Error. The specs are:

Operating System: Rocky Linux 8.6 (Green Obsidian)
Kernel: Linux 4.18.0-372.19.1.el8_6.x86_64
Architecture: x86-64

On both machines, I have installed ffcv like:

conda create -n my_env python=3.9 cupy pkg-config libjpeg-turbo opencv pytorch torchvision cudatoolkit=11.3 numba -c conda-forge -c pytorch  
conda activate my_env
pip install ffcv

Then, from python console:

import ffcv

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/__init__.py", line 1, in <module>
    from .loader import Loader
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/loader/__init__.py", line 1, in <module>
    from .loader import Loader, OrderOption
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/loader/loader.py", line 14, in <module>
    from ffcv.fields.base import Field
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/fields/__init__.py", line 1, in <module>
    from .base import Field
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/fields/base.py", line 6, in <module>
    from ..pipeline.operation import Operation
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/pipeline/__init__.py", line 2, in <module>
    from .pipeline_spec import PipelineSpec
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/pipeline/pipeline_spec.py", line 5, in <module>
    from ..transforms.module import ModuleWrapper
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/transforms/__init__.py", line 5, in <module>
    from .random_resized_crop import RandomResizedCrop
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/transforms/random_resized_crop.py", line 5, in <module>
    from .utils import fast_crop
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/transforms/utils/fast_crop.py", line 4, in <module>
    from ...libffcv import ctypes_resize
  File "/home/user/miniconda3/envs/my_env/lib/python3.9/site-packages/ffcv/libffcv.py", line 6, in <module>
    import ffcv._libffcv
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

I tried the workaround proposed by @thom-OrigamiLabs but it is not working in my case.

Note: I guess that this may be related to issue #136

[thom-OrigamiLabs]

@andrewilyas thanks for the response and the library, I'll look to make a proper fix and submit a PR when I get the chance.

@SerezD I think that might be a coincedentally placed different error - mine was about not finding the newly compiled libffcv library, whereas libGL should be a system library AFAIK. You could try the solutions suggested here, particularly the following:

apt-get update && apt-get install libgl1