search

ys-zong / MEDFAIR

[ICLR 2023 spotlight] MEDFAIR: Benchmarking Fairness for Medical Imaging
https://ys-zong.github.io/MEDFAIR/
55 stars 9 forks source link

Issues Installing Packages from `environments.yml` on macOS ARM and Ubuntu #10

Open MASTER-WM opened 2 weeks ago

MASTER-WM commented 2 weeks ago

Description

I am experiencing several issues while trying to set up my development environment using an environments.yml file on both macOS ARM and Ubuntu. Below are the detailed steps and errors I encountered:

Steps to Reproduce

  1. Attempt to create the environment on macOS ARM using the environments.yml file:

    conda env create -n fair_benchmark -f environment.yml

  2. Encounter the following error:

    PackagesNotFoundError: The following packages are not available from current channels:
  - libstdcxx-ng
  - libgcc-ng
  - _libgcc_mutex

  3. Attempt to create the environment on Ubuntu using the environments.yml file:

    conda env create -n fair_benchmark -f environment.yml

  4. Encounter the following error:

    ERROR: No matching distribution found for oapackage==2.4.4

  5. Remove oapackage from the environments.yml file and attempt to create the environment again. Encounter the following error:

    ERROR: No matching distribution found for torch==1.10.2+cu111

  6. Remove torch from the environments.yml file and attempt to create the environment again. Encounter the following error:

    ERROR: No matching distribution found for torchvision==0.11.3+cu111

  7. Remove torchvision from the environments.yml file and attempt to create the environment again. Encounter the following error:

    ERROR: Cannot install -r /home/amir/PycharmProjects/Medfair/MEDFAIR/condaenv.3zil8mmr.requirements.txt (line 132) and tornado==4.5.3 because these package versions have conflicting dependencies.

  8. Attempt to install packages using pip. Some packages install successfully, but I encounter errors stating that packages are not installed when I try to use them.

Expected Behavior

The environment should be created successfully with all specified packages installed and functional.

Actual Behavior

Encountering multiple errors related to missing packages and conflicting dependencies.

System Information

environments.yml (MEDFAIR)

name: torch11
channels:
  - anaconda
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - backcall=0.2.0=py_0
  - ca-certificates=2020.10.14=0
  - certifi=2020.6.20=py38_0
  - ipython_genutils=0.2.0=py38_0
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - ncurses=6.3=h7f8727e_2
  - openssl=1.1.1m=h7f8727e_0
  - pexpect=4.8.0=py38_0
  - pickleshare=0.7.5=py38_1000
  - pip=21.2.4=py38h06a4308_0
  - python=3.8.12=h12debd9_0
  - readline=8.1.2=h7f8727e_1
  - setuptools=58.0.4=py38h06a4308_0
  - sqlite=3.37.0=hc218d9a_0
  - tk=8.6.11=h1ccaba5_0
  - wcwidth=0.2.5=py_0
  - xz=5.2.5=h7b6447c_0
  - zeromq=4.3.3=he6710b0_3
  - zlib=1.2.11=h7f8727e_4
  - pip:
    - absl-py==1.0.0
    - addict==2.4.0
    - albumentations==0.5.2
    - anyio==3.6.1
    - anykeystore==0.2
    - anyqt==0.1.1
    - argon2-cffi==21.3.0
    - argon2-cffi-bindings==21.2.0
    - arrow==1.2.2
    - asttokens==2.0.5
    - astunparse==1.6.3
    - attrs==21.4.0
    - autopep8==1.6.0
    - av==9.2.0
    - awscli==1.22.77
    - baycomp==1.0.2
    - bayesian-torch==0.2.1
    - beautifulsoup4==4.11.1
    - black==22.1.0
    - bleach==4.1.0
    - botocore==1.24.22
    - bottleneck==1.3.5
    - cachecontrol==0.12.11
    - cachetools==5.0.0
    - cffi==1.15.0
    - chardet==4.0.0
    - charset-normalizer==2.0.12
    - click==8.0.3
    - colorama==0.4.3
    - colorcet==3.0.0
    - commonmark==0.9.1
    - configargparse==1.5.3
    - cryptacular==1.6.2
    - cryptography==37.0.4
    - cycler==0.11.0
    - debugpy==1.5.1
    - decorator==5.1.1
    - defusedxml==0.7.1
    - deprecated==1.2.13
    - dicom2nifti==2.3.3
    - dictdiffer==0.9.0
    - diffdist==0.1
    - dill==0.3.4
    - docker-pycreds==0.4.0
    - docopt==0.6.2
    - docutils==0.15.2
    - dominate==2.6.0
    - efficientnet-pytorch==0.6.3
    - entrypoints==0.4
    - et-xmlfile==1.1.0
    - executing==0.8.2
    - face-alignment==1.3.5
    - filelock==3.7.0
    - fire==0.4.0
    - flatbuffers==2.0
    - fonttools==4.29.1
    - future==0.18.2
    - fvcore==0.1.5.post20220512
    - gast==0.5.3
    - gitdb==4.0.9
    - gitpython==3.1.27
    - google-auth==2.6.2
    - google-auth-oauthlib==0.4.6
    - google-pasta==0.2.0
    - greenlet==1.1.2
    - grpcio==1.44.0
    - h11==0.12.0
    - h5py==3.6.0
    - httpcore==0.15.0
    - httpx==0.23.0
    - humanize==3.14.0
    - hupper==1.10.3
    - idna==2.10
    - idx2numpy==1.2.3
    - imageio==2.16.0
    - imageio-ffmpeg==0.4.5
    - imgaug==0.4.0
    - importlib-metadata==4.11.3
    - importlib-resources==5.4.0
    - imutils==0.5.4
    - iopath==0.1.9
    - ipydatawidgets==4.2.0
    - itk==5.2.1.post1
    - itk-core==5.2.1.post1
    - itk-filtering==5.2.1.post1
    - itk-io==5.2.1.post1
    - itk-meshtopolydata==0.7.1
    - itk-numerics==5.2.1.post1
    - itk-registration==5.2.1.post1
    - itk-segmentation==5.2.1.post1
    - itkwidgets==0.32.1
    - jedi==0.17.2
    - jeepney==0.8.0
    - jinja2==3.0.3
    - jmespath==1.0.0
    - joblib==1.1.0
    - jsonpatch==1.32
    - jsonpickle==1.5.2
    - jsonpointer==2.2
    - jsonschema==4.4.0
    - keyring==23.7.0
    - keyrings-alt==4.1.1
    - kiwisolver==1.3.2
    - lesscpy==0.15.0
    - libclang==13.0.0
    - llvmlite==0.38.0
    - lmdb==1.3.0
    - lockfile==0.12.2
    - lpips==0.1.4
    - markdown==3.3.6
    - markupsafe==2.0.1
    - matplotlib==3.5.1
    - matplotlib-inline==0.1.3
    - mdai==0.11.0
    - mistune==0.8.4
    - mmcls==0.23.0
    - mmcv-full==1.4.2
    - mpi4py==3.1.3
    - msgpack==1.0.4
    - multicoretsne==0.1
    - munch==2.5.0
    - mypy-extensions==0.4.3
    - natsort==8.1.0
    - nbclient==0.5.10
    - nbconvert==6.4.2
    - nbformat==5.1.3
    - nest-asyncio==1.5.4
    - netcal==1.1.2
    - networkx==2.6.3
    - nibabel==3.2.1
    - nilearn==0.9.0
    - notebook==6.4.12
    - numba==0.55.1
    - numpy==1.22.0
    - nvidia-ml-py3==7.352.0
    - oapackage==2.4.4
    - oauthlib==3.2.0
    - opencv-contrib-python==4.5.5.64
    - opencv-python==4.5.5.62
    - opencv-python-headless==4.5.5.64
    - openpyxl==3.0.9
    - openslide-python==1.1.2
    - opentsne==0.6.2
    - opt-einsum==3.3.0
    - orange-canvas-core==0.1.27
    - orange-widget-base==4.17.0
    - orange3==3.32.0
    - packaging==21.3
    - pandas==1.4.0
    - pandocfilters==1.5.0
    - param==1.12.0
    - parameterized==0.8.1
    - parso==0.7.1
    - pastedeploy==2.1.1
    - pathspec==0.9.0
    - pathtools==0.1.2
    - patsy==0.5.2
    - pbkdf2==1.3
    - pillow==9.0.1
    - plaster==1.0
    - plaster-pastedeploy==0.7
    - platformdirs==2.5.0
    - ply==3.11
    - portalocker==2.4.0
    - pretrainedmodels==0.7.4
    - progressbar2==4.0.0
    - prometheus-client==0.13.1
    - promise==2.3
    - prompt-toolkit==3.0.27
    - protobuf==3.19.4
    - psutil==5.9.0
    - ptyprocess==0.7.0
    - pure-eval==0.2.2
    - py-cpuinfo==8.0.0
    - pyasn1==0.4.8
    - pyasn1-modules==0.2.8
    - pycodestyle==2.8.0
    - pycparser==2.21
    - pyct==0.4.8
    - pydicom==2.3.0
    - pyglet==1.5.23
    - pygments==2.11.2
    - pylibjpeg==1.4.0
    - pylibjpeg-libjpeg==1.3.1
    - pynndescent==0.5.7
    - pynvml==11.4.1
    - pyparsing==3.0.7
    - pyqt5==5.15.7
    - pyqt5-qt5==5.15.2
    - pyqt5-sip==12.11.0
    - pyqtgraph==0.12.3
    - pyqtwebengine==5.15.6
    - pyqtwebengine-qt5==5.15.2
    - pyramid==2.0
    - pyramid-mailer==0.15.1
    - pyrsistent==0.18.1
    - python-dateutil==2.8.2
    - python-gdcm==3.0.12
    - python-utils==3.1.0
    - python3-openid==3.2.0
    - pytorchvideo==0.1.5
    - pytz==2021.3
    - pywavelets==1.2.0
    - pyyaml==5.4.1
    - pyzmq==22.3.0
    - qasync==0.23.0
    - qimage2ndarray==1.9.0
    - qtconsole==5.2.2
    - qtpy==2.0.1
    - qudida==0.0.4
    - remote-ikernel==0.4.6
    - repoze-sendmail==4.4.1
    - requests==2.25.1
    - requests-oauthlib==1.3.1
    - retrying==1.3.3
    - rfc3986==1.5.0
    - rsa==4.7.2
    - s3transfer==0.5.2
    - sacred==0.8.2
    - scikit-image==0.19.2
    - scikit-learn==1.0.2
    - scikit-posthocs==0.7.0
    - scipy==1.7.3
    - seaborn==0.11.2
    - secretstorage==3.3.2
    - send2trash==1.8.0
    - sentry-sdk==1.5.6
    - serverfiles==0.3.1
    - setproctitle==1.2.2
    - shortuuid==1.0.8
    - simpleitk==2.1.1
    - six==1.16.0
    - sklearn==0.0
    - smmap==5.0.0
    - sniffio==1.2.0
    - soupsieve==2.3.2.post1
    - sqlalchemy==1.4.37
    - stack-data==0.1.4
    - statannot==0.2.3
    - statsmodels==0.13.2
    - strconv==0.4.2
    - tabulate==0.8.9
    - tensorboard==2.8.0
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.1
    - termcolor==1.1.0
    - terminado==0.13.1
    - testpath==0.5.0
    - tf-estimator-nightly==2.8.0.dev2021122109
    - thop==0.0.31-2005241907
    - threadpoolctl==3.1.0
    - tifffile==2022.2.9
    - toml==0.10.2
    - tomli==2.0.1
    - torch==1.10.2+cu111
    - torchfile==0.1.0
    - torchsampler==0.1.2
    - torchvision==0.11.3+cu111
    - tornado==4.5.3
    - tqdm==4.62.3
    - traitlets==5.1.1
    - traittypes==0.2.1
    - transaction==3.0.1
    - translationstring==1.4
    - typing-extensions==4.0.1
    - umap-learn==0.5.3
    - urllib3==1.26.8
    - velruse==1.1.1
    - venusian==3.0.0
    - visdom==0.1.8.9
    - wandb==0.12.11
    - webencodings==0.5.1
    - webob==1.8.7
    - websocket-client==1.3.1
    - werkzeug==2.0.3
    - wget==3.2
    - wheel==0.37.1
    - widgetsnbextension==3.5.2
    - wrapt==1.13.3
    - wtforms==3.0.1
    - wtforms-recaptcha==0.3.2
    - xlrd==2.0.1
    - xlsxwriter==3.0.3
    - yacs==0.1.8
    - yapf==0.32.0
    - yaspin==2.1.0
    - zipp==3.7.0
    - zope-deprecation==4.4.0
    - zope-interface==5.4.0
    - zope-sqlalchemy==1.6
    - zstandard==0.17.0

Additional Information

I also tried installing some of the packages using pip, and while they seem to install correctly, I receive errors indicating they are not installed when I try to use them in my project.

Request for Help

Thank you for your assistance.

ys-zong commented 2 weeks ago

Hi, thanks for your interest. I lost access to the cluster that I used to run the experiments so maybe it's due to some hardware issues. Can you try to set up the environment using the following commands via pip to see if it works? This should contain a minimal set of packages that you should be able to run most of the experiments.

conda create -n fair_benchmark python==3.10 -y
conda activate fair_benchmark
pip install -r requirements.txt

Inside requirements.txt:

torch
torchvision
pandas
wandb
scikit-learn
h5py
opencv-python
OApackage
Orange3
matplotlib
natsort==8.1.0
bayesian-torch==0.2.1