can't pip install ffcv

[exnx]

Hi, I have an existing conda environment that I'm trying to add ffcv to. Oddly enough, if I create a new conda environment with ffcv in it (via the conda create line given in the docs), I can import ffcv just fine. However, if I want to add the ffcv dependancies to an existing conda environment, ffcv won't install. Here's my work flow.

(Note: I used pip instead of conda install because I get some

• have an existing conda environment w/ python 3.8, cuda 10.2, torch 1.9, torchvision 0.10, numba
• pip install open-cv, cupy, pkgconfig, compiler, pyturbojpeg (no libjpeg-turbo for pip available)
• pip install ffcv

Then I get this exception:

    ERROR: Command errored out with exit status 1:
     command: /home/miniconda3/envs/hippo-ffcv2/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-q89j7v53/ffcv/setup.py'"'"'; __file__='"'"'/tmp/pip-install-q89j7v53/ffcv/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-4qu3jg5l
         cwd: /tmp/pip-install-q89j7v53/ffcv/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-q89j7v53/ffcv/setup.py", line 30, in <module>
        extension_kwargs = pkgconfig('opencv4', extension_kwargs)
      File "/tmp/pip-install-q89j7v53/ffcv/setup.py", line 18, in pkgconfig
        raise Exception()
    Exception
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Note: I used pip because when I tried to use conda for (cupy, pkg-config, compilers, open-cv, etc), I get this error:

Collecting package metadata (current_repodata.json): done
Solving environment: | 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - pytorch/linux-64::torchvision==0.10.0=py38_cu111
  - pytorch/linux-64::torchaudio==0.9.0=py38

Which after a long output of conflicts, it states this:

Your installed version is: 2.27

To get around this, I used pip, and didn't have these conflicts. But again, I can't install ffcv. Anybody have any idea what might be the issue?

Update:

I've also tried install opencv from source via https://docs.opencv.org/4.x/d7/d9f/tutorial_linux_install.html#tutorial_linux_install_quick_build_core, but still got the same exception above when trying to pip install ffcv

Update2:

I cloned the ffcv repo and ran python setup.py, and inspected the extension_kwargs var, and got this:

{'sources': ['./libffcv/libffcv.cpp'], 'include_dirs': ['/home/miniconda3/envs/ffcv_hippo2/include/opencv4', '/home/miniconda3/envs/ffcv_hippo2/include'], 'library_dirs': ['/home/miniconda3/envs/ffcv_hippo2/lib', '/home/miniconda3/envs/ffcv_hippo2/lib'], 'libraries': ['opencv_gapi', 'opencv_stitching', 'opencv_alphamat', 'opencv_aruco', 'opencv_barcode', 'opencv_bgsegm', 'opencv_bioinspired', 'opencv_ccalib', 'opencv_cvv', 'opencv_dnn_objdetect', 'opencv_dnn_superres', 'opencv_dpm', 'opencv_face', 'opencv_freetype', 'opencv_fuzzy', 'opencv_hdf', 'opencv_hfs', 'opencv_img_hash', 'opencv_intensity_transform', 'opencv_line_descriptor', 'opencv_mcc', 'opencv_quality', 'opencv_rapid', 'opencv_reg', 'opencv_rgbd', 'opencv_saliency', 'opencv_stereo', 'opencv_structured_light', 'opencv_phase_unwrapping', 'opencv_superres', 'opencv_optflow', 'opencv_surface_matching', 'opencv_tracking', 'opencv_highgui', 'opencv_datasets', 'opencv_text', 'opencv_plot', 'opencv_videostab', 'opencv_videoio', 'opencv_wechat_qrcode', 'opencv_xfeatures2d', 'opencv_shape', 'opencv_ml', 'opencv_ximgproc', 'opencv_video', 'opencv_dnn', 'opencv_xobjdetect', 'opencv_objdetect', 'opencv_calib3d', 'opencv_imgcodecs', 'opencv_features2d', 'opencv_flann', 'opencv_xphoto', 'opencv_photo', 'opencv_imgproc', 'opencv_core', 'turbojpeg', 'pthread']}

[bibhabasumohapatra]

help wanted ! Collecting ffcv Using cached ffcv-0.0.3.tar.gz (53 kB) Preparing metadata (setup.py) ... error error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\ASUS\AppData\Local\Temp\pip-install-fzrd9cqe\ffcv_549c561025204b5bb7558015a0e2dbcc\setup.py", line 33, in <module>
          extension_kwargs['libraries'].append('pthread')
      KeyError: 'libraries'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

i have probably done everything . . . in fact I have uninstalled anaconda and again installed and tried . . . everytime getting same issue.

I have followed :

1) conda create -n ffcv python=3.9 cupy pkg-config compilers libjpeg-turbo opencv pytorch torchvision cudatoolkit=11.3 numba -c pytorch -c conda-forge

2) conda activate ffcv

3) pip install ffcv 

[GuillaumeLeclerc]

Hi @exnx:

What are the output for

• pkg-config --cflags --libs opencv4
• pkg-config --cflags --libs libturbojpeg

?

[GuillaumeLeclerc]

@bibhabasumohapatra We do not support Windows. There was a pull request that tried to add support but if I recall properly it wasn't really working

[exnx]

@GuillaumeLeclerc

Ah, it can't find opencv. It did find libturbojpeg ok.

pkg-config --cflags --libs opencv4

output:

Package opencv4 was not found in the pkg-config search path.
Perhaps you should add the directory containing opencv4.pc
to the PKG_CONFIG_PATH environment variable
No package 'opencv4' found

pkg-config --cflags --libs libturbojpeg

output:

-I/home/miniconda3/envs/hippo_fccv_clone/include -L/home/miniconda3/envs/hippo_fccv_clone/lib -lturbojpeg

I used pip install for opencv, because for some reason I can't conda install in this existing environment, something to do with inconsistencies. Is conda the only way to install opencv, or I guess the from source route as well?

Thanks!

[GuillaumeLeclerc]

The pip version is a built binary and doesn't contain the header files needed for the compilation of ffcv. There are more simpler ways than from source but they usually depend on your operating system. For example on ubuntu that might be apt install libopencv-dev or something along those lines. In some cases the OS installs it under opencv instead of opencv4 so you might have to either update the setup.py script or update the pc file in your OS folder.

Keep us updated with your progress

[exnx]

ok, my lab mate solved our issue. Our procedure was:

1. make a fresh conda env first on it's own entirely, using python 3.8. For whatever reason, doing this separately instead of creating the environment and installing the ffcv dependancies in step 2 made a difference. Things broke in step 4 later if I didn't start with this.

2. now, conda install the ffcv requirements with this below:

conda install cupy pkg-config compilers libjpeg-turbo opencv pytorch torchvision torchaudio cudatoolkit=11.3 numba -c pytorch -c conda-forge

(This used pytorch 1.10 for me)

3. pip install ffcv

4. finally, pip install my other project's requirements (e.g., pytorch_lightning, etc)

This workflow resolves my issue, no need to install opencv from source etc.

[GuillaumeLeclerc]

Ok sounds good!

[GuillaumeLeclerc]

Note that pytorch_ligthing currently has issues with FFCV as it tries to preload the data again when FFCV has already done everything perfectly. We unfortunately don't recommend using them together as of now :/ There is a temporary solution somewhere on our Slack if you want to try though

[exnx]

Thanks for the heads up, yes I'd love to try this hack. I'll join it now.