Unable to install dependencies

[Zone04]

Hello, I have been trying to reinstall CyberBattleSim without any luck until now. I am unable to either install dependencies with pip install -r requirements.dev.txt, run bash init.sh or even building the included docker container with the Dockerfile, indicating that this must be a general problem and not a problem with my specific installation of python/conda. It takes all 3 of these commands about 7 hours to run, with pip taking increasingly long time to determine a compatible version of the requirements, ultimately failing with the following error (also including the last logging information from the installation process):

From pip install -r requirements.dev.txt

Collecting async-lru>=1.0.0
  Using cached async_lru-2.0.3-py3-none-any.whl (6.0 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached async_lru-2.0.2-py3-none-any.whl (5.7 kB)
  Using cached async_lru-2.0.1-py3-none-any.whl (5.4 kB)
  Using cached async_lru-2.0.0-py3-none-any.whl (5.4 kB)
  Using cached async_lru-1.0.3-py3-none-any.whl (4.3 kB)
  Using cached async_lru-1.0.2.tar.gz (5.0 kB)
  Preparing metadata (setup.py) ... done
  Using cached async_lru-1.0.1.tar.gz (4.9 kB)
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of async-lru to determine which version is compatible with other requirements. This could take a while.
  Using cached async_lru-1.0.0.tar.gz (4.9 kB)
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of asttokens to determine which version is compatible with other requirements. This could take a while.
Collecting asttokens>=2.1.0
  Using cached asttokens-2.4.0-py2.py3-none-any.whl (27 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached asttokens-2.3.0-py2.py3-none-any.whl (27 kB)
  Using cached asttokens-2.2.1-py2.py3-none-any.whl (26 kB)
  Using cached asttokens-2.2.0-py2.py3-none-any.whl (26 kB)
  Using cached asttokens-2.1.0-py2.py3-none-any.whl (26 kB)
INFO: pip is looking at multiple versions of anyio to determine which version is compatible with other requirements. This could take a while.
Collecting anyio>=3.1.0
  Using cached anyio-3.7.1-py3-none-any.whl (80 kB)
INFO: pip is looking at multiple versions of asttokens to determine which version is compatible with other requirements. This could take a while.
  Using cached anyio-3.7.0-py3-none-any.whl (80 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
  Using cached anyio-3.6.1-py3-none-any.whl (80 kB)
  Using cached anyio-3.6.0-py3-none-any.whl (80 kB)
  Using cached anyio-3.5.0-py3-none-any.whl (79 kB)
  Using cached anyio-3.4.0-py3-none-any.whl (78 kB)
INFO: pip is looking at multiple versions of anyio to determine which version is compatible with other requirements. This could take a while.
  Using cached anyio-3.3.4-py3-none-any.whl (78 kB)
  Using cached anyio-3.3.3-py3-none-any.whl (78 kB)
  Using cached anyio-3.3.2-py3-none-any.whl (78 kB)
  Using cached anyio-3.3.1-py3-none-any.whl (77 kB)
  Using cached anyio-3.3.0-py3-none-any.whl (77 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached anyio-3.2.1-py3-none-any.whl (75 kB)
  Using cached anyio-3.2.0-py3-none-any.whl (75 kB)
  Using cached anyio-3.1.0-py3-none-any.whl (74 kB)
INFO: pip is looking at multiple versions of wheel to determine which version is compatible with other requirements. This could take a while.
Collecting wheel
  Using cached wheel-0.41.3-py3-none-any.whl (65 kB)
  Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
  Using cached wheel-0.41.1-py3-none-any.whl (64 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 393, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000

From building Dockerfile:

22328.7 Collecting anyio>=3.1.0
22328.7   Downloading anyio-3.7.1-py3-none-any.whl (80 kB)
22328.7 INFO: pip is looking at multiple versions of asttokens to determine which version is compatible with other requirements. This could take a while.
22328.7   Downloading anyio-3.7.0-py3-none-any.whl (80 kB)
22328.7 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
22328.7   Downloading anyio-3.6.2-py3-none-any.whl (80 kB)
22328.7   Downloading anyio-3.6.1-py3-none-any.whl (80 kB)
22328.7   Downloading anyio-3.6.0-py3-none-any.whl (80 kB)
22328.7   Downloading anyio-3.5.0-py3-none-any.whl (79 kB)
22328.7   Downloading anyio-3.4.0-py3-none-any.whl (78 kB)
22328.7 INFO: pip is looking at multiple versions of anyio to determine which version is compatible with other requirements. This could take a while.
22328.7   Downloading anyio-3.3.4-py3-none-any.whl (78 kB)
22328.7   Downloading anyio-3.3.3-py3-none-any.whl (78 kB)
22328.7   Downloading anyio-3.3.2-py3-none-any.whl (78 kB)
22328.7   Downloading anyio-3.3.1-py3-none-any.whl (77 kB)
22328.7   Downloading anyio-3.3.0-py3-none-any.whl (77 kB)
22328.7 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
22328.7   Downloading anyio-3.2.1-py3-none-any.whl (75 kB)
22328.7   Downloading anyio-3.2.0-py3-none-any.whl (75 kB)
22328.7   Downloading anyio-3.1.0-py3-none-any.whl (74 kB)
22328.7 INFO: pip is looking at multiple versions of tinycss2 to determine which version is compatible with other requirements. This could take a while.
22328.7 Collecting tinycss2
22328.7   Downloading tinycss2-1.2.0-py3-none-any.whl (21 kB)
22328.7   Downloading tinycss2-1.1.1-py3-none-any.whl (21 kB)
22328.7   Downloading tinycss2-1.1.0-py3-none-any.whl (21 kB)
22328.7 
22328.7 failed
22328.7 
22328.7 
22328.7 ==> WARNING: A newer version of conda exists. <==
22328.7   current version: 23.9.0
22328.7   latest version: 23.10.0
22328.7 
22328.7 Please update conda by running
22328.7 
22328.7     $ conda update -n base -c conda-forge conda
22328.7 
22328.7 Or to minimize the number of packages updated during conda update use
22328.7 
22328.7      conda install conda=23.10.0
22328.7 
22328.7 
22328.7 Pip subprocess error:
22328.7 ERROR: Exception:
22328.7 Traceback (most recent call last):
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
22328.7     status = run_func(*args)
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
22328.7     return func(self, options, args)
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 338, in run
22328.7     requirement_set = resolver.resolve(
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
22328.7     result = self._result = resolver.resolve(
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
22328.7     state = resolution.resolve(requirements, max_rounds=max_rounds)
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 393, in resolve
22328.7     raise ResolutionTooDeep(max_rounds)
22328.7 pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000
22328.7 
22328.7 
22328.7 CondaEnvException: Pip failed
22328.7 
------
Dockerfile:12
--------------------
  10 |     ADD *.txt ./
  11 |     ADD *.yml ./
  12 | >>> RUN conda env create -f env.yml
  13 |     RUN conda init bash
  14 |     ENV PATH /opt/miniconda/envs/cybersim/bin:$PATH
--------------------
ERROR: failed to solve: process "/bin/sh -c conda env create -f env.yml" did not complete successfully: exit code: 1

[blumu]

@Zone04 Thanks for reporting. Your logs show that your version of conda is 23.9.0. Could you confirm whether you can repro with version 23.5.2?

[Zone04]

@blumu The reported version is from the docker building process, done without modifying the provided Dockerfile. Running bash init.sh on the host with conda 23.9.0 gave the first logs, 23.10.0 gave similar results. I downgraded conda to 23.5.2 as per your request, bash init.sh is currently running but is now stuck on "Installing pip dependencies" and I suspect that it will crash with the same error in a few hours. I will let you know when it ends, either with a succes or a failure.

[blumu]

Thank you for trying it. I'm suspecting a regression in the base docker image. A potential workaround, assuming it's still available, is to use the docker image from the last successful github action run: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04@sha256:c3a0158d43baedab356a634312da5ee3135bb99efea4080147e8d0d5dde17425

[Zone04]

I've stopped the previous attempt of running init.sh, which was still stuck at "Installing pip dependencies" to try rebuilding the Dockerfile with the image you suggested. If this can be of any help, here is an export of the conda environment a colleague successfully created few months ago with the init.sh script. When I tried recreating the environment from this file, I ran into the same error as in #100 because the default version of setuptools in a new environment was version 68, so I had to manually downgrade setuptools in this new environment before updating the environment with the file.

name: cybersim
channels:
  - plotly
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - blas=1.0=mkl
  - brotli-python=1.1.0=py39h3d6467e_1
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2023.08.22=h06a4308_0
  - certifi=2023.7.22=py39h06a4308_0
  - charset-normalizer=3.3.1=pyhd8ed1ab_0
  - gmp=6.2.1=h295c915_3
  - gmpy2=2.1.2=py39heeb90bb_0
  - icu=73.2=h59595ed_0
  - idna=3.4=pyhd8ed1ab_0
  - importlib-metadata=6.8.0=pyha770c72_0
  - importlib_metadata=6.8.0=hd8ed1ab_0
  - intel-openmp=2023.1.0=hdb19cb5_46306
  - joblib=1.2.0=py39h06a4308_0
  - jupyter_client=8.6.0=py39h06a4308_0
  - jupyter_core=5.4.0=py39hf3d152e_0
  - kernel-headers_linux-64=2.6.32=he073ed8_16
  - ld_impl_linux-64=2.40=h41732ed_0
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=13.2.0=h807b86a_2
  - libgfortran-ng=11.2.0=h00389a5_1
  - libgfortran5=11.2.0=h1234567_1
  - libgomp=13.2.0=h807b86a_2
  - libnsl=2.0.1=hd590300_0
  - libsodium=1.0.18=h36c2ea0_1
  - libsqlite=3.43.2=h2797004_0
  - libstdcxx-ng=13.2.0=h7e041cc_2
  - libuuid=2.38.1=h0b41bf4_0
  - libuv=1.46.0=hd590300_0
  - libzlib=1.2.13=hd590300_5
  - mkl=2023.1.0=h213fc3f_46344
  - mkl-service=2.4.0=py39h5eee18b_1
  - mkl_fft=1.3.8=py39h5eee18b_0
  - mkl_random=1.2.4=py39hdb19cb5_0
  - mpc=1.1.0=h10f8cd9_1
  - mpfr=4.0.2=hb69a4c5_1
  - mpmath=1.3.0=py39h06a4308_0
  - nb_conda_kernels=2.3.1=py39hf3d152e_2
  - ncurses=6.4=hcb278e6_0
  - nodeenv=1.8.0=pyhd8ed1ab_0
  - nodejs=20.8.1=h1990674_0
  - numpy-base=1.26.0=py39hb5e798b_0
  - openssl=3.1.4=hd590300_0
  - pip=21.3.1=pyhd8ed1ab_0
  - pixman-cos6-x86_64=0.32.8=h9b0a68f_1105
  - platformdirs=3.11.0=pyhd8ed1ab_0
  - plotly-orca=1.3.1=1
  - pyright=1.1.331=py39hd1e30aa_1
  - pysocks=1.7.1=pyha2e5f31_6
  - python=3.9.18=h0755675_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=4_cp39
  - pyzmq=25.1.1=py39h8c080ef_2
  - readline=8.2=h8228510_1
  - requests=2.31.0=pyhd8ed1ab_0
  - scikit-learn=1.3.0=py39h1128e8f_0
  - scipy=1.11.3=py39h5f9d8c6_0
  - six=1.16.0=pyh6c4a22f_0
  - sympy=1.11.1=py39h06a4308_0
  - sysroot_linux-64=2.12=he073ed8_16
  - tbb=2021.8.0=hdb19cb5_0
  - tenacity=8.2.3=pyhd8ed1ab_0
  - threadpoolctl=2.2.0=pyh0d69192_0
  - tk=8.6.13=h2797004_0
  - tornado=6.3.3=py39hd1e30aa_1
  - traitlets=5.11.2=pyhd8ed1ab_0
  - typing-extensions=4.8.0=hd8ed1ab_0
  - typing_extensions=4.8.0=pyha770c72_0
  - urllib3=2.0.7=pyhd8ed1ab_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xorg-x11-server-common-cos6-x86_64=1.17.4=h9b0a68f_1105
  - xorg-x11-server-xvfb-cos6-x86_64=1.17.4=h9b0a68f_1105
  - xz=5.2.6=h166bdaf_0
  - zeromq=4.3.5=h59595ed_0
  - zlib=1.2.13=hd590300_5
  - pip:
      - absl-py==2.0.0
      - ale-py==0.7.4
      - ansiwrap==0.8.4
      - anyio==4.0.0
      - argon2-cffi==23.1.0
      - argon2-cffi-bindings==21.2.0
      - arrow==1.3.0
      - asciichart==0.1
      - asciichartpy==1.5.25
      - asttokens==2.4.0
      - async-lru==2.0.4
      - attrs==19.2.0
      - autopep8==2.0.4
      - autorom==0.4.2
      - autorom-accept-rom-license==0.6.1
      - babel==2.13.1
      - backcall==0.2.0
      - beautifulsoup4==4.12.2
      - black==23.10.1
      - bleach==6.1.0
      - boolean-py==4.0
      - cachetools==5.3.2
      - cffi==1.16.0
      - cfgv==3.4.0
      - click==8.1.7
      - cloudpickle==3.0.0
      - comm==0.1.4
      - contourpy==1.1.1
      - cycler==0.12.1
      - debugpy==1.8.0
      - decorator==4.3.0
      - defusedxml==0.7.1
      - distlib==0.3.7
      - entrypoints==0.4
      - exceptiongroup==1.1.3
      - executing==2.0.0
      - farama-notifications==0.0.4
      - fastjsonschema==2.18.1
      - filelock==3.12.4
      - flake8==6.1.0
      - fonttools==4.43.1
      - fqdn==1.5.1
      - google-auth==2.23.4
      - google-auth-oauthlib==1.1.0
      - grpcio==1.59.2
      - gym==0.21.0
      - gymnasium==0.29.1
      - identify==2.5.30
      - importlib-resources==6.1.1
      - iniconfig==2.0.0
      - ipykernel==6.6.1
      - ipython==8.16.1
      - ipython-genutils==0.2.0
      - ipywidgets==8.1.1
      - isoduration==20.11.0
      - jedi==0.19.1
      - jinja2==3.1.2
      - json5==0.9.14
      - jsonpointer==2.4
      - jsonschema==4.17.3
      - jupyter==1.0.0
      - jupyter-client==7.4.9
      - jupyter-console==6.4.4
      - jupyter-events==0.6.3
      - jupyter-lsp==2.2.0
      - jupyter-server==2.9.0
      - jupyter-server-terminals==0.4.4
      - jupyterlab==4.0.7
      - jupyterlab-pygments==0.2.2
      - jupyterlab-server==2.24.0
      - jupyterlab-widgets==3.0.9
      - jupytext==1.6.0
      - keyboard==0.13.5
      - kiwisolver==1.4.5
      - markdown==3.5.1
      - markdown-it-py==0.5.8
      - markupsafe==2.1.3
      - matplotlib==3.6.3
      - matplotlib-inline==0.1.6
      - mccabe==0.7.0
      - mistune==3.0.2
      - mypy-extensions==1.0.0
      - nbclient==0.8.0
      - nbconvert==7.9.2
      - nbformat==5.9.2
      - nest-asyncio==1.5.8
      - networkx==2.8.6
      - notebook==7.0.6
      - notebook-shim==0.2.3
      - numpy==1.23.0
      - nvidia-cublas-cu11==11.10.3.66
      - nvidia-cuda-nvrtc-cu11==11.7.99
      - nvidia-cuda-runtime-cu11==11.7.99
      - nvidia-cudnn-cu11==8.5.0.96
      - oauthlib==3.2.2
      - opencv-python==4.8.1.78
      - ordered-set==4.0.2
      - overrides==7.4.0
      - packaging==23.2
      - pandas==2.1.1
      - pandocfilters==1.5.0
      - papermill==2.3.3
      - parso==0.8.3
      - pathspec==0.11.2
      - pep8-naming==0.13.3
      - pexpect==4.8.0
      - pickleshare==0.7.5
      - pillow==10.1.0
      - plotly==4.11.0
      - pluggy==1.3.0
      - pre-commit==3.5.0
      - progressbar2==3.51.4
      - prometheus-client==0.17.1
      - prompt-toolkit==3.0.39
      - protobuf==3.19.6
      - psutil==5.7.2
      - ptyprocess==0.7.0
      - pure-eval==0.2.2
      - py==1.11.0
      - pyasn1==0.5.0
      - pyasn1-modules==0.3.0
      - pycodestyle==2.11.1
      - pycparser==2.21
      - pyflakes==3.1.0
      - pygments==2.16.1
      - pyparsing==3.1.1
      - pyqt5==5.15.10
      - pyqt5-qt5==5.15.2
      - pyqt5-sip==12.13.0
      - pyqtgraph==0.13.3
      - pyrsistent==0.19.3
      - pytest==6.2.5
      - python-json-logger==2.0.7
      - python-utils==3.8.1
      - pytz==2023.3.post1
      - pyyaml==6.0.1
      - qtconsole==5.4.4
      - qtpy==2.4.1
      - requests-oauthlib==1.3.1
      - retrying==1.3.4
      - rfc3339-validator==0.1.4
      - rfc3986-validator==0.1.1
      - rsa==4.9
      - send2trash==1.8.2
      - setuptools==65.5.1
      - shimmy==1.3.0
      - simpy==4.1.1
      - sniffio==1.3.0
      - soupsieve==2.5
      - stable-baselines3==2.2.1
      - stack-data==0.6.3
      - tabulate==0.8.10
      - tensorboard==2.15.1
      - tensorboard-data-server==0.7.2
      - terminado==0.17.1
      - textwrap3==0.9.2
      - tinycss2==1.2.1
      - toml==0.10.2
      - tomli==2.0.1
      - torch==1.13.1
      - tqdm==4.66.1
      - types-python-dateutil==2.8.19.14
      - tzdata==2023.3
      - uri-template==1.3.0
      - virtualenv==20.24.6
      - wcwidth==0.2.8
      - webcolors==1.13
      - webencodings==0.5.1
      - websocket-client==1.6.4
      - werkzeug==3.0.1
      - widgetsnbextension==4.0.9
      - zipp==3.8.1

[Zone04]

The building process from the given image has now been stuck on the pip dependencies for more than 10 minutes, I highly suspect that this will not complete either.

[Zone04]

docker_output.txt As expected, it failed after 7 hours while installing pip dependencies in the conda environment creation step from the Dockerfile. Please find the complete output attached.

[blumu]

@zone04 Just to make sure I understand correctly were you able to make it work using the env.yaml file shared by your colleague? And if yes, did you have to downgrade setuptools in your system python environment prior to running conda env create or by just adding the setuptools depdency to the env.yaml file?

[Zone04]

@blumu Using the env.yml file I linked before, creating the conda env fails when pip installs gym. I then have the "cybersim" environment with only part of the dependencies installed, and among these dependcies I have setuptools version 68. I then manually downgrade to version 65, and update the environment using conda env update --file env.yml, which this time succeeds

[blumu]

@Zone04 Thank you for sharing the logs and info. Could you try branch https://github.com/microsoft/CyberBattleSim/tree/wiblum/dep see if it works for you? https://github.com/microsoft/CyberBattleSim/pull/119/

[Zone04]

Will do tomorrow when I'll be back on my work computer

[Zone04]

The docker container was successfully built and running the docker run command from README worked without any problem. Manually creating the conda environment with the included requirements was also successful, as well as running the init.sh script. All tests passed. Thank you very much

[blumu]

@Zone04 Thank you for confirming, I'll go ahead and merge the PR.