search

allegroai / clearml-agent

ClearML Agent - ML-Ops made easy. ML-Ops scheduler & orchestration solution
https://clear.ml/docs/
Apache License 2.0
236 stars 90 forks source link

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL is ignored when task is resetted and enqueued #104

Open Honzys opened 2 years ago

Honzys commented 2 years ago

Hello everyone!

When using env variable CLEARML_AGENT_SKIP_PIP_VENV_INSTALL for clearml-agent I guess there's a bug because it is ignored when resetting and equeuing the task again.

It works as expected when you submit the task from scatch (meaning just submitting it).

But if you reset the same task and enqueue it again it will try to install all the requirements (which are already installed in the python env). I think that it is not desirable behaviour.

Could you please confirm that it is a bug?

I can create a PR to fix this issue, but I would need some pointers where to look.

Thanks in advance! :)

jkhenning commented 2 years ago

Hi @Honzys,

I think the issue is related to a bug in the CLEARML_AGENT_SKIP_PIP_VENV_INSTALL support which was just fixed yesterday - it's included in the latest v1.2.2 release - can you see if it resolves your issue?

Honzys commented 2 years ago

Hello, I've just tested the new release, but sadly the problem still remains. After resetting and enqueuing the agent still tries to install all the dependencies.

jkhenning commented 2 years ago

Hi @Honzys,

This is very strange... Can you attach the task's log from the run that was successful and the run that failed?

Honzys commented 2 years ago

Hello, sorry for late reponse.

There's the log from fresh start - succesfull one:

1648813249156 gauss_user:4gpu INFO task 692a9a8fa41743b686e5182b7150205e pulled from 009758b9298b4f7f80a67c0a8f767b4c by worker gauss_user:4gpu

1648813249305 gauss_user:4gpu INFO Running Task 692a9a8fa41743b686e5182b7150205e inside docker: *********:5005/ip/solvers/my-project:v3.0 arguments: ['-v', '/srv:/srv', '-v', '/mnt/nas.brno/:/mnt/nas.brno/', '-v', '/home/user/.aws/credentials:/root/.aws/credentials', '-v', '/home/user/.docker/:/root/.docker/', '--mount', 'type=bind,source=/etc/innovatrics/iengine.lic,target=/etc/innovatrics/iengine.lic', '-e', 'GLOG_minloglevel=2', '--shm-size=250g', '-e', 'UNAME=user']

1648813249370 gauss_user:4gpu INFO Executing: ['docker', 'run', '-t', '--gpus', '"device=4,5,6,7"', '-v', '/srv:/srv', '-v', '/mnt/nas.brno/:/mnt/nas.brno/', '-v', '/home/user/.aws/credentials:/root/.aws/credentials', '-v', '/home/user/.docker/:/root/.docker/', '--mount', 'type=bind,source=/etc/innovatrics/iengine.lic,target=/etc/innovatrics/iengine.lic', '-e', 'GLOG_minloglevel=2', '--shm-size=250g', '-e', 'UNAME=user', '-v', '/tmp/ssh-ryCA2ZZ3LX/agent.361431:/tmp/ssh-ryCA2ZZ3LX/agent.361431', '-e', 'SSH_AUTH_SOCK=/tmp/ssh-ryCA2ZZ3LX/agent.361431', '-l', 'clearml-worker-id=gauss_user:4gpu', '-l', 'clearml-parent-worker-id=gauss_user:4gpu', '-e', 'CLEARML_WORKER_ID=gauss_user:4gpu', '-e', 'CLEARML_DOCKER_IMAGE=*********:5005/ip/solvers/my-project:v3.0 -v /srv:/srv -v /mnt/nas.brno/:/mnt/nas.brno/ -v /home/user/.aws/credentials:/root/.aws/credentials -v /home/user/.docker/:/root/.docker/ --mount type=bind,source=/etc/innovatrics/iengine.lic,target=/etc/innovatrics/iengine.lic -e GLOG_minloglevel=2 --shm-size=250g -e UNAME=user', '-e', 'CLEARML_TASK_ID=692a9a8fa41743b686e5182b7150205e', '-e', 'CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/venv/bin/python', '-v', '/home/user/.git-credentials:/root/.git-credentials', '-v', '/home/user/.gitconfig:/root/.gitconfig', '-v', '/tmp/user/.clearml_agent.n0_3qsh4.cfg:/root/clearml.conf', '-v', '/srv/workplace/user/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/srv/workplace/user/.clearml/pip-cache:/root/.cache/pip', '-v', '/srv/workplace/user/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/srv/workplace/user/.clearml/cache:/clearml_agent_cache', '-v', '/srv/workplace/user/.clearml/vcs-cache:/root/.clearml/vcs-cache', '--rm', '*********:5005/ip/solvers/my-project:v3.0', 'bash', '-c', 'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /root/clearml.conf /root/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring  --id 692a9a8fa41743b686e5182b7150205e']

1648813254453 gauss_user:4gpu DEBUG v3.0: Pulling from ip/solvers/my-project
Digest: sha256:25f5417e8977e3d8853f62c40d9e9c4354ba3bd03a80e920444516bd65d9b6d9
Status: Image is up to date for *********:5005/ip/solvers/my-project:v3.0
*********:5005/ip/solvers/my-project:v3.0

1648813259564 gauss_user:4gpu DEBUG /opt/venv/bin/python3.8
pip 22.0.4 from /opt/venv/lib/python3.8/site-packages/pip (python 3.8)
Looking in indexes: *************, https://pypi.org/simple/
Requirement already satisfied: pip in /opt/venv/lib/python3.8/site-packages (22.0.4)

1648813264688 gauss_user:4gpu DEBUG Looking in indexes: *************, https://pypi.org/simple/
Collecting clearml-agent
  Using cached clearml_agent-1.2.2-py3-none-any.whl (371 kB)
Collecting jsonschema<3.3.0,>=2.6.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting PyYAML<5.5.0,>=3.12
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Requirement already satisfied: urllib3<1.27.0,>=1.21.1 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (1.26.9)
Requirement already satisfied: pathlib2<2.4.0,>=2.3.0 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (2.3.7.post1)
Collecting six<1.16.0,>=1.13.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: python-dateutil<2.9.0,>=2.4.2 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (2.8.2)
Requirement already satisfied: furl<2.2.0,>=2.0.0 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (2.1.3)
Collecting pyjwt<2.1.0,>=1.6.4
  Using cached PyJWT-2.0.1-py3-none-any.whl (15 kB)
Collecting requests<2.26.0,>=2.20.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting psutil<5.9.0,>=3.4.2
  Using cached psutil-5.8.0-cp38-cp38-manylinux2010_x86_64.whl (296 kB)
Collecting pyhocon<0.4.0,>=0.3.38
  Using cached pyhocon-0.3.59-py3-none-any.whl
Collecting pyparsing<2.5.0,>=2.0.3
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Requirement already satisfied: virtualenv<21,>=16 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (20.14.0)
Collecting attrs<20.4.0,>=18.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Requirement already satisfied: future<0.19.0,>=0.16.0 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (0.18.2)
Requirement already satisfied: orderedmultidict>=1.0.1 in /opt/venv/lib/python3.8/site-packages (from furl<2.2.0,>=2.0.0->clearml-agent) (1.0.1)
Requirement already satisfied: setuptools in /opt/venv/lib/python3.8/site-packages (from jsonschema<3.3.0,>=2.6.0->clearml-agent) (61.2.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/venv/lib/python3.8/site-packages (from jsonschema<3.3.0,>=2.6.0->clearml-agent) (0.18.1)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/venv/lib/python3.8/site-packages (from requests<2.26.0,>=2.20.0->clearml-agent) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/venv/lib/python3.8/site-packages (from requests<2.26.0,>=2.20.0->clearml-agent) (2021.10.8)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: filelock<4,>=3.2 in /opt/venv/lib/python3.8/site-packages (from virtualenv<21,>=16->clearml-agent) (3.6.0)
Requirement already satisfied: platformdirs<3,>=2 in /opt/venv/lib/python3.8/site-packages (from virtualenv<21,>=16->clearml-agent) (2.5.1)
Requirement already satisfied: distlib<1,>=0.3.1 in /opt/venv/lib/python3.8/site-packages (from virtualenv<21,>=16->clearml-agent) (0.3.4)

1648813269918 gauss_user:4gpu DEBUG Installing collected packages: six, PyYAML, pyparsing, pyjwt, psutil, idna, attrs, requests, pyhocon, jsonschema, clearml-agent
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 6.0
    Uninstalling PyYAML-6.0:
      Successfully uninstalled PyYAML-6.0
  Attempting uninstall: pyparsing
    Found existing installation: pyparsing 3.0.7
    Uninstalling pyparsing-3.0.7:
      Successfully uninstalled pyparsing-3.0.7
  Attempting uninstall: pyjwt
    Found existing installation: PyJWT 2.1.0
    Uninstalling PyJWT-2.1.0:
      Successfully uninstalled PyJWT-2.1.0
  Attempting uninstall: psutil
    Found existing installation: psutil 5.9.0
    Uninstalling psutil-5.9.0:
      Successfully uninstalled psutil-5.9.0
  Attempting uninstall: idna
    Found existing installation: idna 3.3
    Uninstalling idna-3.3:
      Successfully uninstalled idna-3.3
  Attempting uninstall: attrs
    Found existing installation: attrs 21.4.0
    Uninstalling attrs-21.4.0:
      Successfully uninstalled attrs-21.4.0
  Attempting uninstall: requests
    Found existing installation: requests 2.27.1
    Uninstalling requests-2.27.1:
      Successfully uninstalled requests-2.27.1
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 4.4.0
    Uninstalling jsonschema-4.4.0:
      Successfully uninstalled jsonschema-4.4.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
python-benedict 0.25.0 requires pyyaml<7.0,>=6.0, but you have pyyaml 5.4.1 which is incompatible.
python-benedict 0.25.0 requires requests<3.0.0,>=2.26.0, but you have requests 2.25.1 which is incompatible.
Successfully installed PyYAML-5.4.1 attrs-20.3.0 clearml-agent-1.2.2 idna-2.10 jsonschema-3.2.0 psutil-5.8.0 pyhocon-0.3.59 pyjwt-2.0.1 pyparsing-2.4.7 requests-2.25.1 six-1.15.0
Current configuration (clearml_agent v1.2.2, location: /root/clearml.conf):
----------------------
agent.worker_id = gauss_user:4gpu
agent.worker_name = gauss
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version =
agent.package_manager.system_site_packages = true
agent.package_manager.force_upgrade = true
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.torch_nightly = false
agent.package_manager.extra_index_url.0 = *************
agent.package_manager.extra_index_url.1 = https://download.pytorch.org/whl/cu101
agent.package_manager.force_repo_requirements_txt = false
agent.package_manager.conda_env_as_base_docker = false
agent.venvs_dir = /root/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /root/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /root/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = true
agent.docker_pip_cache = /srv/workplace/user/.clearml/pip-cache
agent.docker_apt_cache = /srv/workplace/user/.clearml/apt-cache
agent.docker_force_pull = true
agent.default_docker.image = nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
agent.enable_task_env = false
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = /root/.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache
agent.docker_internal_mounts.venv_build = /root/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script =
agent.git_user = user
agent.default_python = 3.8
agent.docker_install_opencv_libs = false
agent.cuda_version = 113
agent.cudnn_version = 0
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.web_server = **************
api.api_server = **************
api.files_server = **************
api.credentials.access_key = *********************
api.host = **************
sdk.storage.cache.default_base_dir = /clearml_agent_cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false

Executing task id [692a9a8fa41743b686e5182b7150205e]:
repository = *****************
branch = IMAGE-2569-update-dependencies
version_num =
tag =
docker_cmd = ***************** -v /srv:/srv -v /mnt/nas.brno/:/mnt/nas.brno/ -v /home/user/.aws/credentials:/root/.aws/credentials -v /home/user/.docker/:/root/.docker/ -e GLOG_minloglevel=2 --shm-size=250g -e UNAME=user
entry_point = train.py
working_dir = .

Python interpreter /opt/venv/bin/python is set from environment var

Using cached repository in "/root/.clearml/vcs-cache/my-project.git.5929fda2b781e4ac79f698d6bd3e3838/my-project.git"
fatal: unable to write credential store: Device or resource busy
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 7 (delta 1), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (7/7), done.
From *****************
   341b28a..a98632f  master     -> origin/master
 * [new branch]      WIP_IRIS   -> origin/WIP_IRIS
Note: checking out 'origin/IMAGE-2569-update-dependencies'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 2721d99 Change how to get logger in callback [IMAGE-2569]
type: git
url: ****************
branch: HEAD
commit: 2721d992a141ad41db91e53abaf9eba3338409f3
root: /root/.clearml/venvs-builds/task_repository/my-project.git

1648813274842 gauss_user:4gpu DEBUG Looking in indexes: *************, https://pypi.org/simple/
Requirement already satisfied: pip in /opt/venv/lib/python3.8/site-packages (22.0.4)
Looking in indexes: *************, https://pypi.org/simple/

1648813280146 gauss_user:4gpu DEBUG Collecting Cython
  Using cached Cython-0.29.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.28
Adding venv into cache: /root/.clearml/venvs-builds
Running task id [692a9a8fa41743b686e5182b7150205e]:
[.]$ /opt/venv/bin/python -u train.py
Summary - installed python packages:
pip:
- absl-py==1.0.0
- aiobotocore==2.1.2
- aiohttp==3.8.1
- aiohttp-retry==2.4.6
- aioitertools==0.10.0
- aiosignal==1.2.0
- antlr4-python3-runtime==4.8
- appdirs==1.4.4
- arrow==1.2.2
- astropy==5.0.3
- async-timeout==4.0.2
- asyncssh==2.8.1
- atpublic==3.0.1
- attrs==20.3.0
- azure-cognitiveservices-anomalydetector==0.3.0
- azure-cognitiveservices-vision-face==0.6.0
- azure-common==1.1.28
- binaryornot==0.4.4
- black==22.1.0
- bleach==4.1.0
- boto3==1.20.24
- botocore==1.23.24
- cachetools==5.0.0
- cattrs==1.0.0
- certifi==2021.10.8
- cffi==1.15.0
- cfgv==3.3.1
- chardet==4.0.0
- charset-normalizer==2.0.12
- clearml==1.3.2
- click==8.1.0
- colorama==0.4.4
- commonmark==0.9.1
- configobj==5.0.6
- cookiecutter==1.7.3
- coverage==6.3.2
- crc32c==2.2.post0
- cruft==1.4.0
- cryptography==36.0.2
- cycler==0.11.0
- Cython==0.29.28
- dictdiffer==0.9.0
- diskcache==5.4.0
- distlib==0.3.4
- distro==1.7.0
- docutils==0.18.1
- dpath==2.0.6
- dulwich==0.20.35
- dvc==2.9.5
- examples==1.0.2
- falcon==2.0.0
- filelock==3.6.0
- flake8==4.0.1
- flake8-gl-codeclimate==0.1.5
- flatten-dict==0.4.2
- flufl.lock==6.0
- fonttools==4.31.2
- freezegun==1.2.1
- frozenlist==1.3.0
- fsspec==2022.2.0
- ftfy==6.1.1
- funcy==1.17
- furl==2.1.3
- future==0.18.2
- gitdb==4.0.9
- GitPython==3.1.27
- google-auth==2.6.2
- google-auth-oauthlib==0.4.6
- grandalf==0.6
- grpcio==1.44.0
- h5py==3.6.0
- hug==2.6.1
- hydra-core==1.1.1
- identify==2.4.12
- idna==2.10
- imagecorruptions==1.1.2
- imageio==2.16.1
- imantics==0.1.12
- imgaug==0.4.0
- importlib-metadata==4.11.3
- importlib-resources==5.6.0
- iniconfig==1.1.1
- inno-common==0.3.0
- inno-db==1.9.0
- inno-df==0.9.0
- inno-dsr==0.10.2
- inno-dvc==1.44.1
- inno-eval==3.0.0
- inno-nn==0.9.1
- inno-primitives==0.4.2
- inno-training==0.5.1
- inno.evaluation==0.3.2
- inno.iface-dev-nightly==4.16.0.0
- inno.iface-nightly==4.16.0.0
- inno.image-vector-nightly==4.0.0
- isodate==0.6.1
- jeepney==0.7.1
- Jinja2==3.1.1
- jinja2-time==0.2.0
- jmespath==0.10.0
- joblib==1.1.0
- jsonschema==3.2.0
- keyring==23.5.0
- kiwisolver==1.4.2
- lxml==4.8.0
- mailchecker==4.1.14
- Markdown==3.3.6
- MarkupSafe==2.1.1
- matplotlib==3.5.1
- mccabe==0.6.1
- msrest==0.6.21
- multidict==6.0.2
- mypy==0.942
- mypy-extensions==0.4.3
- nanotime==0.5.2
- networkx==2.7.1
- nodeenv==1.6.0
- numexpr==2.8.1
- numpy==1.22.3
- nvsmi==0.4.2
- oauthlib==3.2.0
- omegaconf==2.1.1
- opencv-python==4.5.5.64
- opencv-python-headless==4.5.5.64
- orderedmultidict==1.0.1
- packaging==21.3
- pandas==1.4.1
- pathlib2==2.3.7.post1
- pathspec==0.9.0
- phonenumbers==8.12.45
- Pillow==9.0.1
- pkg_resources==0.0.0
- pkginfo==1.8.2
- platformdirs==2.5.1
- plotly==5.6.0
- pluggy==1.0.0
- ply==3.11
- poyo==0.5.0
- pre-commit==2.17.0
- protobuf==3.19.4
- psutil==5.8.0
- py==1.11.0
- pyasn1==0.4.8
- pyasn1-modules==0.2.8
- pycodestyle==2.8.0
- pycparser==2.21
- pydantic==1.9.0
- pyDeprecate==0.3.2
- pydot==1.4.2
- pyerfa==2.0.0.1
- pyfastcopy==1.0.3
- pyflakes==2.4.0
- pygit2==1.9.1
- Pygments==2.11.2
- pygtrie==2.4.2
- pyhocon==0.3.59
- PyJWT==2.0.1
- pyparsing==2.4.7
- pyrsistent==0.18.1
- pytest==7.1.1
- pytest-cov==3.0.0
- python-benedict==0.25.0
- python-dateutil==2.8.2
- python-fsutil==0.6.0
- python-slugify==6.1.1
- pytorch-lightning==1.6.0
- pytz==2022.1
- PyWavelets==1.3.0
- PyYAML==5.4.1
- readme-renderer==34.0
- requests==2.25.1
- requests-oauthlib==1.3.1
- requests-toolbelt==0.9.1
- rfc3986==2.0.0
- rich==12.0.1
- rsa==4.8
- ruamel.yaml==0.17.21
- ruamel.yaml.clib==0.2.6
- s3fs==2022.2.0
- s3transfer==0.5.2
- scikit-image==0.19.2
- scikit-learn==0.24.2
- scipy==1.8.0
- scmrepo==0.0.7
- SecretStorage==3.3.1
- setproctitle==1.2.2
- Shapely==1.8.1.post1
- shortuuid==1.0.8
- shtab==1.5.3
- six==1.15.0
- smmap==5.0.0
- tables==3.7.0
- tabulate==0.8.9
- tenacity==8.0.1
- tensorboard==2.8.0
- tensorboard-data-server==0.6.1
- tensorboard-plugin-wit==1.8.1
- terminaltables==3.1.10
- text-unidecode==1.3
- threadpoolctl==3.1.0
- tifffile==2022.3.25
- toml==0.10.2
- tomli==2.0.1
- torch==1.11.0+cu113
- torchmetrics==0.7.3
- torchvision==0.12.0+cu113
- tqdm==4.63.1
- twine==3.8.0
- typing-inspect==0.7.1
- typing_extensions==4.1.1
- urllib3==1.26.9
- virtualenv==20.14.0
- voluptuous==0.13.0
- wcwidth==0.2.5
- webencodings==0.5.1
- Werkzeug==2.1.0
- wrapt==1.14.0
- xmljson==0.2.1
- xmltodict==0.12.0
- yarl==1.7.2
- zc.lockfile==2.0
- zipp==3.7.0

Environment setup completed successfully

Starting Task Execution:

FROM HERE THE TRAINING STARTS OK

There's the log started by Reset -> Enqueue:

1649163679221 gauss_user:4gpu INFO task 692a9a8fa41743b686e5182b7150205e pulled from 009758b9298b4f7f80a67c0a8f767b4c by worker gauss_user:4gpu

1649163679328 gauss_user:4gpu INFO Running Task 692a9a8fa41743b686e5182b7150205e inside docker: ************/ip/solvers/my-project:v3.0 arguments: ['-v', '/srv:/srv', '-v', '/mnt/nas.brno/:/mnt/nas.brno/', '-v', '/home/user/.aws/credentials:/root/.aws/credentials', '-v', '/home/user/.docker/:/root/.docker/', '-e', 'GLOG_minloglevel=2', '--shm-size=250g', '-e', 'UNAME=user']

1649163679378 gauss_user:4gpu INFO Executing: ['docker', 'run', '-t', '--gpus', '"device=4,5,6,7"', '-v', '/srv:/srv', '-v', '/mnt/nas.brno/:/mnt/nas.brno/', '-v', '/home/user/.aws/credentials:/root/.aws/credentials', '-v', '/home/user/.docker/:/root/.docker/', '-e', 'GLOG_minloglevel=2', '--shm-size=250g', '-e', 'UNAME=user', '-v', '/tmp/ssh-ryCA2ZZ3LX/agent.361431:/tmp/ssh-ryCA2ZZ3LX/agent.361431', '-e', 'SSH_AUTH_SOCK=/tmp/ssh-ryCA2ZZ3LX/agent.361431', '-l', 'clearml-worker-id=gauss_user:4gpu', '-l', 'clearml-parent-worker-id=gauss_user:4gpu', '-e', 'CLEARML_WORKER_ID=gauss_user:4gpu', '-e', 'CLEARML_DOCKER_IMAGE=************/ip/solvers/my-project:v3.0 -v /srv:/srv -v /mnt/nas.brno/:/mnt/nas.brno/ -v /home/user/.aws/credentials:/root/.aws/credentials -v /home/user/.docker/:/root/.docker/ -e GLOG_minloglevel=2 --shm-size=250g -e UNAME=user', '-e', 'CLEARML_TASK_ID=692a9a8fa41743b686e5182b7150205e', '-e', 'CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/opt/venv/bin/python', '-v', '/home/user/.git-credentials:/root/.git-credentials', '-v', '/home/user/.gitconfig:/root/.gitconfig', '-v', '/tmp/user/.clearml_agent.n0_3qsh4.cfg:/root/clearml.conf', '-v', '/srv/workplace/user/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/srv/workplace/user/.clearml/pip-cache:/root/.cache/pip', '-v', '/srv/workplace/user/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/srv/workplace/user/.clearml/cache:/clearml_agent_cache', '-v', '/srv/workplace/user/.clearml/vcs-cache:/root/.clearml/vcs-cache', '--rm', '************/ip/solvers/my-project:v3.0', 'bash', '-c', 'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /root/clearml.conf /root/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring  --id 692a9a8fa41743b686e5182b7150205e']

1649163684463 gauss_user:4gpu DEBUG v3.0: Pulling from ip/solvers/my-project
Digest: sha256:25f5417e8977e3d8853f62c40d9e9c4354ba3bd03a80e920444516bd65d9b6d9
Status: Image is up to date for ************/ip/solvers/my-project:v3.0
************/ip/solvers/my-project:v3.0

1649163689547 gauss_user:4gpu DEBUG /opt/venv/bin/python3.8
pip 22.0.4 from /opt/venv/lib/python3.8/site-packages/pip (python 3.8)
Looking in indexes: ******************, https://pypi.org/simple/
Requirement already satisfied: pip in /opt/venv/lib/python3.8/site-packages (22.0.4)

1649163694657 gauss_user:4gpu DEBUG Looking in indexes: ******************, https://pypi.org/simple/
Collecting clearml-agent
  Using cached clearml_agent-1.2.2-py3-none-any.whl (371 kB)
Collecting jsonschema<3.3.0,>=2.6.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting six<1.16.0,>=1.13.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: virtualenv<21,>=16 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (20.14.0)
Collecting psutil<5.9.0,>=3.4.2
  Using cached psutil-5.8.0-cp38-cp38-manylinux2010_x86_64.whl (296 kB)
Requirement already satisfied: python-dateutil<2.9.0,>=2.4.2 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (2.8.2)
Collecting PyYAML<5.5.0,>=3.12
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Collecting pyparsing<2.5.0,>=2.0.3
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting requests<2.26.0,>=2.20.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Requirement already satisfied: pathlib2<2.4.0,>=2.3.0 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (2.3.7.post1)
Collecting pyhocon<0.4.0,>=0.3.38
  Using cached pyhocon-0.3.59-py3-none-any.whl
Requirement already satisfied: urllib3<1.27.0,>=1.21.1 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (1.26.9)
Collecting pyjwt<2.1.0,>=1.6.4
  Using cached PyJWT-2.0.1-py3-none-any.whl (15 kB)
Requirement already satisfied: furl<2.2.0,>=2.0.0 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (2.1.3)
Requirement already satisfied: future<0.19.0,>=0.16.0 in /opt/venv/lib/python3.8/site-packages (from clearml-agent) (0.18.2)
Collecting attrs<20.4.0,>=18.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Requirement already satisfied: orderedmultidict>=1.0.1 in /opt/venv/lib/python3.8/site-packages (from furl<2.2.0,>=2.0.0->clearml-agent) (1.0.1)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/venv/lib/python3.8/site-packages (from jsonschema<3.3.0,>=2.6.0->clearml-agent) (0.18.1)
Requirement already satisfied: setuptools in /opt/venv/lib/python3.8/site-packages (from jsonschema<3.3.0,>=2.6.0->clearml-agent) (61.2.0)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/venv/lib/python3.8/site-packages (from requests<2.26.0,>=2.20.0->clearml-agent) (4.0.0)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: certifi>=2017.4.17 in /opt/venv/lib/python3.8/site-packages (from requests<2.26.0,>=2.20.0->clearml-agent) (2021.10.8)
Requirement already satisfied: platformdirs<3,>=2 in /opt/venv/lib/python3.8/site-packages (from virtualenv<21,>=16->clearml-agent) (2.5.1)
Requirement already satisfied: distlib<1,>=0.3.1 in /opt/venv/lib/python3.8/site-packages (from virtualenv<21,>=16->clearml-agent) (0.3.4)
Requirement already satisfied: filelock<4,>=3.2 in /opt/venv/lib/python3.8/site-packages (from virtualenv<21,>=16->clearml-agent) (3.6.0)
Installing collected packages: six, PyYAML, pyparsing, pyjwt, psutil, idna, attrs, requests, pyhocon, jsonschema, clearml-agent
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:

1649163699861 gauss_user:4gpu DEBUG       Successfully uninstalled six-1.16.0
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 6.0
    Uninstalling PyYAML-6.0:
      Successfully uninstalled PyYAML-6.0
  Attempting uninstall: pyparsing
    Found existing installation: pyparsing 3.0.7
    Uninstalling pyparsing-3.0.7:
      Successfully uninstalled pyparsing-3.0.7
  Attempting uninstall: pyjwt
    Found existing installation: PyJWT 2.1.0
    Uninstalling PyJWT-2.1.0:
      Successfully uninstalled PyJWT-2.1.0
  Attempting uninstall: psutil
    Found existing installation: psutil 5.9.0
    Uninstalling psutil-5.9.0:
      Successfully uninstalled psutil-5.9.0
  Attempting uninstall: idna
    Found existing installation: idna 3.3
    Uninstalling idna-3.3:
      Successfully uninstalled idna-3.3
  Attempting uninstall: attrs
    Found existing installation: attrs 21.4.0
    Uninstalling attrs-21.4.0:
      Successfully uninstalled attrs-21.4.0
  Attempting uninstall: requests
    Found existing installation: requests 2.27.1
    Uninstalling requests-2.27.1:
      Successfully uninstalled requests-2.27.1
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 4.4.0
    Uninstalling jsonschema-4.4.0:
      Successfully uninstalled jsonschema-4.4.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
python-benedict 0.25.0 requires pyyaml<7.0,>=6.0, but you have pyyaml 5.4.1 which is incompatible.
python-benedict 0.25.0 requires requests<3.0.0,>=2.26.0, but you have requests 2.25.1 which is incompatible.
Successfully installed PyYAML-5.4.1 attrs-20.3.0 clearml-agent-1.2.2 idna-2.10 jsonschema-3.2.0 psutil-5.8.0 pyhocon-0.3.59 pyjwt-2.0.1 pyparsing-2.4.7 requests-2.25.1 six-1.15.0
Current configuration (clearml_agent v1.2.2, location: /root/clearml.conf):
----------------------
agent.worker_id = gauss_user:4gpu
agent.worker_name = gauss
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version =
agent.package_manager.system_site_packages = true
agent.package_manager.force_upgrade = true
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.torch_nightly = false
agent.package_manager.extra_index_url.0 = ******************
agent.package_manager.extra_index_url.1 = https://download.pytorch.org/whl/cu101
agent.package_manager.force_repo_requirements_txt = false
agent.package_manager.conda_env_as_base_docker = false
agent.venvs_dir = /root/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /root/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /root/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = true
agent.docker_pip_cache = /srv/workplace/user/.clearml/pip-cache
agent.docker_apt_cache = /srv/workplace/user/.clearml/apt-cache
agent.docker_force_pull = true
agent.default_docker.image = nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
agent.enable_task_env = false
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = /root/.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache
agent.docker_internal_mounts.venv_build = /root/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script =
agent.git_user = user
agent.default_python = 3.8
agent.docker_install_opencv_libs = false
agent.cuda_version = 113
agent.cudnn_version = 0
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.web_server = **************
api.api_server = **************
api.files_server = **************
api.credentials.access_key = *********************
api.host = **************
sdk.storage.cache.default_base_dir = /clearml_agent_cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false

Executing task id [692a9a8fa41743b686e5182b7150205e]:
repository = **************
branch = IMAGE-2569-update-dependencies
version_num = 2721d992a141ad41db91e53abaf9eba3338409f3
tag =
docker_cmd = ************/ip/solvers/my-project:v3.0 -v /srv:/srv -v /mnt/nas.brno/:/mnt/nas.brno/ -v /home/user/.aws/credentials:/root/.aws/credentials -v /home/user/.docker/:/root/.docker/ -e GLOG_minloglevel=2 --shm-size=250g -e UNAME=user
entry_point = train.py
working_dir = .

Python interpreter /opt/venv/bin/python is set from environment var

Using cached repository in "/root/.clearml/vcs-cache/my-project.git.5929fda2b781e4ac79f698d6bd3e3838/my-project.git"
fatal: unable to write credential store: Device or resource busy
Note: checking out '2721d992a141ad41db91e53abaf9eba3338409f3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 2721d99 Change how to get logger in callback [IMAGE-2569]
type: git
url: **************
branch: HEAD
commit: 2721d992a141ad41db91e53abaf9eba3338409f3
root: /root/.clearml/venvs-builds/task_repository/my-project.git

Looking in indexes: ******************, https://pypi.org/simple/
Requirement already satisfied: pip in /opt/venv/lib/python3.8/site-packages (22.0.4)

1649163704758 gauss_user:4gpu DEBUG Looking in indexes: ******************, https://pypi.org/simple/
Collecting Cython
  Using cached Cython-0.29.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)

1649163709824 gauss_user:4gpu DEBUG Installing collected packages: Cython
Successfully installed Cython-0.29.28
Looking in indexes: ******************, https://pypi.org/simple/
Requirement already satisfied: Cython==0.29.28 in /opt/venv/lib/python3.8/site-packages (0.29.28)

1649163714897 gauss_user:4gpu DEBUG Looking in indexes: ******************, https://pypi.org/simple/
Requirement already satisfied: numpy==1.22.3 in /opt/venv/lib/python3.8/site-packages (1.22.3)
/opt/venv/lib/python3.8/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

1649163719968 gauss_user:4gpu DEBUG Torch CUDA 113 download page found
Found PyTorch version torch==1.11.0 matching CUDA version 113

1649163725042 gauss_user:4gpu DEBUG Found PyTorch version torchvision==0.12.0 matching CUDA version 113
Looking in indexes: ******************, https://pypi.org/simple/, ******************, https://download.pytorch.org/whl/cu101
Collecting torch==1.11.0+cu113
  File was already downloaded /root/.clearml/pip-download-cache/cu113/torch-1.11.0+cu113-cp38-cp38-linux_x86_64.whl
Successfully downloaded torch

1649163730112 gauss_user:4gpu DEBUG Looking in indexes: ******************, https://pypi.org/simple/, ******************, https://download.pytorch.org/whl/cu101
Collecting torchvision==0.12.0+cu113
  File was already downloaded /root/.clearml/pip-download-cache/cu113/torchvision-0.12.0+cu113-cp38-cp38-linux_x86_64.whl
Successfully downloaded torchvision
Looking in indexes: ******************, https://pypi.org/simple/, ******************, https://download.pytorch.org/whl/cu101
Processing /root/.clearml/pip-download-cache/cu113/torch-1.11.0+cu113-cp38-cp38-linux_x86_64.whl
Processing /root/.clearml/pip-download-cache/cu113/torchvision-0.12.0+cu113-cp38-cp38-linux_x86_64.whl
Requirement already satisfied: absl-py==1.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 1)) (1.0.0)
Requirement already satisfied: aiobotocore==2.1.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 2)) (2.1.2)
Requirement already satisfied: aiohttp==3.8.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 3)) (3.8.1)
Requirement already satisfied: aiohttp-retry==2.4.6 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 4)) (2.4.6)
Requirement already satisfied: aioitertools==0.10.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 5)) (0.10.0)
Requirement already satisfied: aiosignal==1.2.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 6)) (1.2.0)

1649163735201 gauss_user:4gpu DEBUG Requirement already satisfied: antlr4-python3-runtime==4.8 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 7)) (4.8)
Requirement already satisfied: appdirs==1.4.4 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 8)) (1.4.4)
Requirement already satisfied: arrow==1.2.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 9)) (1.2.2)
Requirement already satisfied: astropy==5.0.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 10)) (5.0.3)
Requirement already satisfied: async-timeout==4.0.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 11)) (4.0.2)
Requirement already satisfied: asyncssh==2.8.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 12)) (2.8.1)
Requirement already satisfied: atpublic==3.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 13)) (3.0.1)
Requirement already satisfied: attrs==20.3.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 14)) (20.3.0)
Requirement already satisfied: azure-cognitiveservices-anomalydetector==0.3.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 15)) (0.3.0)

1649163740374 gauss_user:4gpu DEBUG Requirement already satisfied: azure-cognitiveservices-vision-face==0.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 16)) (0.6.0)
Requirement already satisfied: azure-common==1.1.28 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 17)) (1.1.28)
Requirement already satisfied: binaryornot==0.4.4 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 18)) (0.4.4)
Requirement already satisfied: black==22.1.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 19)) (22.1.0)
Requirement already satisfied: bleach==4.1.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 20)) (4.1.0)
Requirement already satisfied: boto3==1.20.24 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 21)) (1.20.24)
Requirement already satisfied: botocore==1.23.24 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 22)) (1.23.24)
Requirement already satisfied: cachetools==5.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 23)) (5.0.0)

1649163745445 gauss_user:4gpu DEBUG Requirement already satisfied: cattrs==1.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 24)) (1.0.0)
Requirement already satisfied: certifi==2021.10.8 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 25)) (2021.10.8)
Requirement already satisfied: cffi==1.15.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 26)) (1.15.0)
Requirement already satisfied: cfgv==3.3.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 27)) (3.3.1)
Requirement already satisfied: chardet==4.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 28)) (4.0.0)
Requirement already satisfied: charset-normalizer==2.0.12 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 29)) (2.0.12)
Requirement already satisfied: click==8.1.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 31)) (8.1.0)
Requirement already satisfied: colorama==0.4.4 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 32)) (0.4.4)
Requirement already satisfied: commonmark==0.9.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 33)) (0.9.1)

1649163750514 gauss_user:4gpu DEBUG Requirement already satisfied: configobj==5.0.6 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 34)) (5.0.6)
Requirement already satisfied: cookiecutter==1.7.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 35)) (1.7.3)
Requirement already satisfied: coverage==6.3.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 36)) (6.3.2)
Requirement already satisfied: crc32c==2.2.post0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 37)) (2.2.post0)
Requirement already satisfied: cruft==1.4.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 38)) (1.4.0)
Requirement already satisfied: cryptography==36.0.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 39)) (36.0.2)
Requirement already satisfied: cycler==0.11.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 40)) (0.11.0)
Requirement already satisfied: Cython==0.29.28 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 41)) (0.29.28)

1649163755576 gauss_user:4gpu DEBUG Requirement already satisfied: dictdiffer==0.9.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 42)) (0.9.0)
Requirement already satisfied: diskcache==5.4.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 43)) (5.4.0)
Requirement already satisfied: distlib==0.3.4 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 44)) (0.3.4)
Requirement already satisfied: distro==1.7.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 45)) (1.7.0)
Requirement already satisfied: docutils==0.18.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 46)) (0.18.1)
Requirement already satisfied: dpath==2.0.6 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 47)) (2.0.6)
Requirement already satisfied: dulwich==0.20.35 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 48)) (0.20.35)
Requirement already satisfied: dvc==2.9.5 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 49)) (2.9.5)
Requirement already satisfied: examples==1.0.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 50)) (1.0.2)

1649163760651 gauss_user:4gpu DEBUG Requirement already satisfied: falcon==2.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 51)) (2.0.0)
Requirement already satisfied: filelock==3.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 52)) (3.6.0)
Requirement already satisfied: flake8==4.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 53)) (4.0.1)
Requirement already satisfied: flake8-gl-codeclimate==0.1.5 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 54)) (0.1.5)
Requirement already satisfied: flatten-dict==0.4.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 55)) (0.4.2)
Requirement already satisfied: flufl.lock==6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 56)) (6.0)
Requirement already satisfied: fonttools==4.31.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 57)) (4.31.2)
Requirement already satisfied: freezegun==1.2.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 58)) (1.2.1)
Requirement already satisfied: frozenlist==1.3.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 59)) (1.3.0)
Requirement already satisfied: fsspec==2022.2.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 60)) (2022.2.0)

1649163765722 gauss_user:4gpu DEBUG Requirement already satisfied: ftfy==6.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 61)) (6.1.1)
Requirement already satisfied: funcy==1.17 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 62)) (1.17)
Requirement already satisfied: furl==2.1.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 63)) (2.1.3)
Requirement already satisfied: future==0.18.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 64)) (0.18.2)
Requirement already satisfied: gitdb==4.0.9 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 65)) (4.0.9)
Requirement already satisfied: GitPython==3.1.27 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 66)) (3.1.27)
Requirement already satisfied: google-auth==2.6.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 67)) (2.6.2)
Requirement already satisfied: google-auth-oauthlib==0.4.6 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 68)) (0.4.6)
Requirement already satisfied: grandalf==0.6 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 69)) (0.6)
Requirement already satisfied: grpcio==1.44.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 70)) (1.44.0)

1649163770794 gauss_user:4gpu DEBUG Requirement already satisfied: h5py==3.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 71)) (3.6.0)
Requirement already satisfied: hug==2.6.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 72)) (2.6.1)
Requirement already satisfied: hydra-core==1.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 73)) (1.1.1)
Requirement already satisfied: identify==2.4.12 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 74)) (2.4.12)
Requirement already satisfied: idna==2.10 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 75)) (2.10)
Requirement already satisfied: imagecorruptions==1.1.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 76)) (1.1.2)
Requirement already satisfied: imageio==2.16.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 77)) (2.16.1)
Requirement already satisfied: imantics==0.1.12 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 78)) (0.1.12)

1649163775859 gauss_user:4gpu DEBUG Requirement already satisfied: imgaug==0.4.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 79)) (0.4.0)
Requirement already satisfied: importlib-metadata==4.11.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 80)) (4.11.3)
Requirement already satisfied: importlib-resources==5.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 81)) (5.6.0)
Requirement already satisfied: iniconfig==1.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 82)) (1.1.1)
Requirement already satisfied: inno-common==0.3.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 83)) (0.3.0)
Requirement already satisfied: inno-db==1.9.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 84)) (1.9.0)
Requirement already satisfied: inno-df==0.9.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 85)) (0.9.0)
Requirement already satisfied: inno-dsr==0.10.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 86)) (0.10.2)
Requirement already satisfied: inno-dvc==1.44.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 87)) (1.44.1)
Requirement already satisfied: inno-eval==3.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 88)) (3.0.0)

1649163780937 gauss_user:4gpu DEBUG Requirement already satisfied: inno-nn==0.9.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 89)) (0.9.1)
Requirement already satisfied: inno-primitives==0.4.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 90)) (0.4.2)
Requirement already satisfied: inno-training==0.5.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 91)) (0.5.1)
Requirement already satisfied: inno.evaluation==0.3.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 92)) (0.3.2)
Requirement already satisfied: inno.iface-dev-nightly==4.16.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 93)) (4.16.0.0)
Requirement already satisfied: inno.iface-nightly==4.16.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 94)) (4.16.0.0)
Requirement already satisfied: inno.image-vector-nightly==4.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 95)) (4.0.0)
Requirement already satisfied: isodate==0.6.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 96)) (0.6.1)
Requirement already satisfied: jeepney==0.7.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 97)) (0.7.1)

1649163786022 gauss_user:4gpu DEBUG Requirement already satisfied: Jinja2==3.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 98)) (3.1.1)
Requirement already satisfied: jinja2-time==0.2.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 99)) (0.2.0)
Requirement already satisfied: jmespath==0.10.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 100)) (0.10.0)
Requirement already satisfied: joblib==1.1.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 101)) (1.1.0)
Requirement already satisfied: jsonschema==3.2.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 102)) (3.2.0)
Requirement already satisfied: keyring==23.5.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 103)) (23.5.0)
Requirement already satisfied: kiwisolver==1.4.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 104)) (1.4.2)
Requirement already satisfied: lxml==4.8.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 105)) (4.8.0)
Requirement already satisfied: mailchecker==4.1.14 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 106)) (4.1.14)

1649163791112 gauss_user:4gpu DEBUG Requirement already satisfied: Markdown==3.3.6 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 107)) (3.3.6)
Requirement already satisfied: MarkupSafe==2.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 108)) (2.1.1)
Requirement already satisfied: matplotlib==3.5.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 109)) (3.5.1)
Requirement already satisfied: mccabe==0.6.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 110)) (0.6.1)
Requirement already satisfied: msrest==0.6.21 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 111)) (0.6.21)
Requirement already satisfied: multidict==6.0.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 112)) (6.0.2)
Requirement already satisfied: mypy==0.942 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 113)) (0.942)
Requirement already satisfied: mypy-extensions==0.4.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 114)) (0.4.3)
Requirement already satisfied: nanotime==0.5.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 115)) (0.5.2)

1649163796180 gauss_user:4gpu DEBUG Requirement already satisfied: networkx==2.7.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 116)) (2.7.1)
Requirement already satisfied: nodeenv==1.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 117)) (1.6.0)
Requirement already satisfied: numexpr==2.8.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 118)) (2.8.1)
Requirement already satisfied: numpy==1.22.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 119)) (1.22.3)
Requirement already satisfied: nvsmi==0.4.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 120)) (0.4.2)
Requirement already satisfied: oauthlib==3.2.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 121)) (3.2.0)
Requirement already satisfied: omegaconf==2.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 122)) (2.1.1)
Requirement already satisfied: opencv-python==4.5.5.64 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 123)) (4.5.5.64)
Requirement already satisfied: opencv-python-headless==4.5.5.64 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 124)) (4.5.5.64)

1649163801251 gauss_user:4gpu DEBUG Requirement already satisfied: orderedmultidict==1.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 125)) (1.0.1)
Requirement already satisfied: packaging==21.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 126)) (21.3)
Requirement already satisfied: pandas==1.4.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 127)) (1.4.1)
Requirement already satisfied: pathlib2==2.3.7.post1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 128)) (2.3.7.post1)
Requirement already satisfied: pathspec==0.9.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 129)) (0.9.0)
Requirement already satisfied: phonenumbers==8.12.45 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 130)) (8.12.45)
Requirement already satisfied: Pillow==9.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 131)) (9.0.1)
Requirement already satisfied: pkg_resources==0.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 132)) (0.0.0)

1649163806335 gauss_user:4gpu DEBUG Requirement already satisfied: pkginfo==1.8.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 133)) (1.8.2)
Requirement already satisfied: platformdirs==2.5.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 134)) (2.5.1)
Requirement already satisfied: plotly==5.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 135)) (5.6.0)
Requirement already satisfied: pluggy==1.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 136)) (1.0.0)
Requirement already satisfied: ply==3.11 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 137)) (3.11)
Requirement already satisfied: poyo==0.5.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 138)) (0.5.0)
Requirement already satisfied: pre-commit==2.17.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 139)) (2.17.0)
Requirement already satisfied: protobuf==3.19.4 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 140)) (3.19.4)
Requirement already satisfied: psutil==5.8.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 141)) (5.8.0)
Requirement already satisfied: py==1.11.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 142)) (1.11.0)

1649163811411 gauss_user:4gpu DEBUG Requirement already satisfied: pyasn1==0.4.8 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 143)) (0.4.8)
Requirement already satisfied: pyasn1-modules==0.2.8 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 144)) (0.2.8)
Requirement already satisfied: pycodestyle==2.8.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 145)) (2.8.0)
Requirement already satisfied: pycparser==2.21 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 146)) (2.21)
Requirement already satisfied: pydantic==1.9.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 147)) (1.9.0)
Requirement already satisfied: pyDeprecate==0.3.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 148)) (0.3.2)
Requirement already satisfied: pydot==1.4.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 149)) (1.4.2)
Requirement already satisfied: pyerfa==2.0.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 150)) (2.0.0.1)
Requirement already satisfied: pyfastcopy==1.0.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 151)) (1.0.3)
Requirement already satisfied: pyflakes==2.4.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 152)) (2.4.0)

1649163816480 gauss_user:4gpu DEBUG Requirement already satisfied: pygit2==1.9.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 153)) (1.9.1)
Requirement already satisfied: Pygments==2.11.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 154)) (2.11.2)
Requirement already satisfied: pygtrie==2.4.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 155)) (2.4.2)
Requirement already satisfied: pyhocon==0.3.59 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 156)) (0.3.59)
Requirement already satisfied: PyJWT==2.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 157)) (2.0.1)
Requirement already satisfied: pyparsing==2.4.7 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 158)) (2.4.7)
Requirement already satisfied: pyrsistent==0.18.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 159)) (0.18.1)
Requirement already satisfied: pytest==7.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 160)) (7.1.1)
Requirement already satisfied: pytest-cov==3.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 161)) (3.0.0)
Requirement already satisfied: python-benedict==0.25.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 162)) (0.25.0)

1649163821538 gauss_user:4gpu DEBUG Requirement already satisfied: python-dateutil==2.8.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 163)) (2.8.2)
Requirement already satisfied: python-fsutil==0.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 164)) (0.6.0)
Requirement already satisfied: python-slugify==6.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 165)) (6.1.1)
Requirement already satisfied: pytorch-lightning==1.6.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 166)) (1.6.0)
Requirement already satisfied: pytz==2022.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 167)) (2022.1)
Requirement already satisfied: PyWavelets==1.3.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 168)) (1.3.0)
Requirement already satisfied: PyYAML==5.4.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 169)) (5.4.1)
Requirement already satisfied: readme-renderer==34.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 170)) (34.0)
Requirement already satisfied: requests==2.25.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 171)) (2.25.1)
Requirement already satisfied: requests-oauthlib==1.3.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 172)) (1.3.1)
Requirement already satisfied: requests-toolbelt==0.9.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 173)) (0.9.1)

1649163826601 gauss_user:4gpu DEBUG Requirement already satisfied: rfc3986==2.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 174)) (2.0.0)
Requirement already satisfied: rich==12.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 175)) (12.0.1)
Requirement already satisfied: rsa==4.8 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 176)) (4.8)
Requirement already satisfied: ruamel.yaml==0.17.21 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 177)) (0.17.21)
Requirement already satisfied: ruamel.yaml.clib==0.2.6 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 178)) (0.2.6)
Requirement already satisfied: s3fs==2022.2.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 179)) (2022.2.0)
Requirement already satisfied: s3transfer==0.5.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 180)) (0.5.2)
Requirement already satisfied: scikit-image==0.19.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 181)) (0.19.2)

1649163831663 gauss_user:4gpu DEBUG Requirement already satisfied: scikit-learn==0.24.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 182)) (0.24.2)
Requirement already satisfied: scipy==1.8.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 183)) (1.8.0)
Requirement already satisfied: scmrepo==0.0.7 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 184)) (0.0.7)
Requirement already satisfied: SecretStorage==3.3.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 185)) (3.3.1)
Requirement already satisfied: setproctitle==1.2.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 186)) (1.2.2)
Requirement already satisfied: Shapely==1.8.1.post1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 187)) (1.8.1.post1)
Requirement already satisfied: shortuuid==1.0.8 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 188)) (1.0.8)
Requirement already satisfied: shtab==1.5.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 189)) (1.5.3)
Requirement already satisfied: six==1.15.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 190)) (1.15.0)

1649163836725 gauss_user:4gpu DEBUG Requirement already satisfied: smmap==5.0.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 191)) (5.0.0)
Requirement already satisfied: tables==3.7.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 192)) (3.7.0)
Requirement already satisfied: tabulate==0.8.9 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 193)) (0.8.9)
Requirement already satisfied: tenacity==8.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 194)) (8.0.1)
Requirement already satisfied: tensorboard==2.8.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 195)) (2.8.0)
Requirement already satisfied: tensorboard-data-server==0.6.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 196)) (0.6.1)
Requirement already satisfied: tensorboard-plugin-wit==1.8.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 197)) (1.8.1)
Requirement already satisfied: terminaltables==3.1.10 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 198)) (3.1.10)
Requirement already satisfied: text-unidecode==1.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 199)) (1.3)

1649163841790 gauss_user:4gpu DEBUG Requirement already satisfied: threadpoolctl==3.1.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 200)) (3.1.0)
Requirement already satisfied: tifffile==2022.3.25 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 201)) (2022.3.25)
Requirement already satisfied: toml==0.10.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 202)) (0.10.2)
Requirement already satisfied: tomli==2.0.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 203)) (2.0.1)
Requirement already satisfied: torchmetrics==0.7.3 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 205)) (0.7.3)
Requirement already satisfied: tqdm==4.63.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 207)) (4.63.1)
Requirement already satisfied: twine==3.8.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 208)) (3.8.0)
Requirement already satisfied: typing-inspect==0.7.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 209)) (0.7.1)
Requirement already satisfied: typing_extensions==4.1.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 210)) (4.1.1)
Requirement already satisfied: urllib3==1.26.9 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 211)) (1.26.9)

1649163846919 gauss_user:4gpu DEBUG Requirement already satisfied: virtualenv==20.14.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 212)) (20.14.0)
Requirement already satisfied: voluptuous==0.13.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 213)) (0.13.0)
Requirement already satisfied: wcwidth==0.2.5 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 214)) (0.2.5)
Requirement already satisfied: webencodings==0.5.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 215)) (0.5.1)
Requirement already satisfied: Werkzeug==2.1.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 216)) (2.1.0)
Requirement already satisfied: wrapt==1.14.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 217)) (1.14.0)
Requirement already satisfied: xmljson==0.2.1 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 218)) (0.2.1)
Requirement already satisfied: xmltodict==0.12.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 219)) (0.12.0)
Requirement already satisfied: yarl==1.7.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 220)) (1.7.2)
Requirement already satisfied: zc.lockfile==2.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 221)) (2.0)

1649163851994 gauss_user:4gpu DEBUG Requirement already satisfied: zipp==3.7.0 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 222)) (3.7.0)
Requirement already satisfied: clearml==1.3.2 in /opt/venv/lib/python3.8/site-packages (from -r /tmp/cached-reqsr9tlzb6r.txt (line 223)) (1.3.2)
Collecting fsspec[http]>=2021.10.1
  Using cached fsspec-2022.3.0-py3-none-any.whl (136 kB)
Collecting dvc[s3]<3.0.0,>=1.11.15
  Using cached dvc-2.10.0-py3-none-any.whl (400 kB)
Requirement already satisfied: setuptools in /opt/venv/lib/python3.8/site-packages (from jsonschema==3.2.0->-r /tmp/cached-reqsr9tlzb6r.txt (line 102)) (61.2.0)

1649163857087 gauss_user:4gpu DEBUG Collecting python-benedict==0.25.0
  Using cached python_benedict-0.25.0-py3-none-any.whl (40 kB)
INFO: pip is looking at multiple versions of pytest-cov to determine which version is compatible with other requirements. This could take a while.
Collecting pytest-cov==3.0.0
  Using cached pytest_cov-3.0.0-py3-none-any.whl (20 kB)
INFO: pip is looking at multiple versions of pytest to determine which version is compatible with other requirements. This could take a while.
Collecting pytest==7.1.1
  Using cached pytest-7.1.1-py3-none-any.whl (297 kB)
INFO: pip is looking at multiple versions of pyrsistent to determine which version is compatible with other requirements. This could take a while.
Collecting pyrsistent==0.18.1
  Using cached pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (119 kB)
INFO: pip is looking at multiple versions of pyparsing to determine which version is compatible with other requirements. This could take a while.
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
INFO: pip is looking at multiple versions of pyjwt to determine which version is compatible with other requirements. This could take a while.
Collecting PyJWT==2.0.1
  Using cached PyJWT-2.0.1-py3-none-any.whl (15 kB)
INFO: pip is looking at multiple versions of pyhocon to determine which version is compatible with other requirements. This could take a while.
Collecting pyhocon==0.3.59
  Using cached pyhocon-0.3.59-py3-none-any.whl
INFO: pip is looking at multiple versions of pygtrie to determine which version is compatible with other requirements. This could take a while.
Collecting pygtrie==2.4.2
  Using cached pygtrie-2.4.2.tar.gz (35 kB)
  Preparing metadata (setup.py) ... [?25l- \ | done
[?25hINFO: pip is looking at multiple versions of pygments to determine which version is compatible with other requirements. This could take a while.
Collecting Pygments==2.11.2
  Using cached Pygments-2.11.2-py3-none-any.whl (1.1 MB)
INFO: pip is looking at multiple versions of pygit2 to determine which version is compatible with other requirements. This could take a while.
Collecting pygit2==1.9.1
  Using cached pygit2-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
INFO: pip is looking at multiple versions of pyflakes to determine which version is compatible with other requirements. This could take a while.
Collecting pyflakes==2.4.0
  Using cached pyflakes-2.4.0-py2.py3-none-any.whl (69 kB)
INFO: pip is looking at multiple versions of pyfastcopy to determine which version is compatible with other requirements. This could take a while.
Collecting pyfastcopy==1.0.3
  Using cached pyfastcopy-1.0.3.tar.gz (4.4 kB)
  Preparing metadata (setup.py) ... [?25l- \

1649163862202 gauss_user:4gpu DEBUG  | / done
[?25hINFO: pip is looking at multiple versions of pyerfa to determine which version is compatible with other requirements. This could take a while.
Collecting pyerfa==2.0.0.1
  Using cached pyerfa-2.0.0.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (775 kB)
INFO: pip is looking at multiple versions of pydot to determine which version is compatible with other requirements. This could take a while.
Collecting pydot==1.4.2
  Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
INFO: pip is looking at multiple versions of pydeprecate to determine which version is compatible with other requirements. This could take a while.
Collecting pyDeprecate==0.3.2
  Using cached pyDeprecate-0.3.2-py3-none-any.whl (10 kB)
INFO: pip is looking at multiple versions of pydantic to determine which version is compatible with other requirements. This could take a while.
Collecting pydantic==1.9.0
  Using cached pydantic-1.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.6 MB)
INFO: pip is looking at multiple versions of pycparser to determine which version is compatible with other requirements. This could take a while.
Collecting pycparser==2.21
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
INFO: pip is looking at multiple versions of pycodestyle to determine which version is compatible with other requirements. This could take a while.
Collecting pycodestyle==2.8.0
  Using cached pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
INFO: pip is looking at multiple versions of pyasn1-modules to determine which version is compatible with other requirements. This could take a while.
Collecting pyasn1-modules==0.2.8
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
INFO: pip is looking at multiple versions of pyasn1 to determine which version is compatible with other requirements. This could take a while.
Collecting pyasn1==0.4.8
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
INFO: pip is looking at multiple versions of py to determine which version is compatible with other requirements. This could take a while.
Collecting py==1.11.0
  Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
INFO: pip is looking at multiple versions of psutil to determine which version is compatible with other requirements. This could take a while.
Collecting psutil==5.8.0
  Using cached psutil-5.8.0-cp38-cp38-manylinux2010_x86_64.whl (296 kB)
INFO: pip is looking at multiple versions of protobuf to determine which version is compatible with other requirements. This could take a while.
Collecting protobuf==3.19.4
  Using cached protobuf-3.19.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
INFO: pip is looking at multiple versions of pre-commit to determine which version is compatible with other requirements. This could take a while.
Collecting pre-commit==2.17.0
  Using cached pre_commit-2.17.0-py2.py3-none-any.whl (195 kB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of poyo to determine which version is compatible with other requirements. This could take a while.
Collecting poyo==0.5.0
  Using cached poyo-0.5.0-py2.py3-none-any.whl (10 kB)
INFO: pip is looking at multiple versions of ply to determine which version is compatible with other requirements. This could take a while.
Collecting ply==3.11
  Using cached ply-3.11-py2.py3-none-any.whl (49 kB)
INFO: pip is looking at multiple versions of pluggy to determine which version is compatible with other requirements. This could take a while.
Collecting pluggy==1.0.0
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
INFO: pip is looking at multiple versions of pre-commit to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of plotly to determine which version is compatible with other requirements. This could take a while.
Collecting plotly==5.6.0
  Using cached plotly-5.6.0-py2.py3-none-any.whl (27.7 MB)
INFO: pip is looking at multiple versions of platformdirs to determine which version is compatible with other requirements. This could take a while.
Collecting platformdirs==2.5.1
  Using cached platformdirs-2.5.1-py3-none-any.whl (14 kB)
INFO: pip is looking at multiple versions of pkginfo to determine which version is compatible with other requirements. This could take a while.
Collecting pkginfo==1.8.2
  Using cached pkginfo-1.8.2-py2.py3-none-any.whl (26 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of pkg-resources to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pillow to determine which version is compatible with other requirements. This could take a while.
Collecting Pillow==9.0.1
  Using cached Pillow-9.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of phonenumbers to determine which version is compatible with other requirements. This could take a while.
Collecting phonenumbers==8.12.45
  Using cached phonenumbers-8.12.45-py2.py3-none-any.whl (2.6 MB)
INFO: pip is looking at multiple versions of pathspec to determine which version is compatible with other requirements. This could take a while.
Collecting pathspec==0.9.0
  Using cached pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
INFO: pip is looking at multiple versions of pathlib2 to determine which version is compatible with other requirements. This could take a while.
Collecting pathlib2==2.3.7.post1
  Using cached pathlib2-2.3.7.post1-py2.py3-none-any.whl (18 kB)
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
Collecting pandas==1.4.1
  Using cached pandas-1.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
INFO: pip is looking at multiple versions of packaging to determine which version is compatible with other requirements. This could take a while.
Collecting packaging==21.3
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of orderedmultidict to determine which version is compatible with other requirements. This could take a while.
Collecting orderedmultidict==1.0.1
  Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
INFO: pip is looking at multiple versions of opencv-python-headless to determine which version is compatible with other requirements. This could take a while.
Collecting opencv-python-headless==4.5.5.64
  Using cached opencv_python_headless-4.5.5.64-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.8 MB)
INFO: pip is looking at multiple versions of opencv-python to determine which version is compatible with other requirements. This could take a while.
Collecting opencv-python==4.5.5.64
  Using cached opencv_python-4.5.5.64-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.5 MB)

1649163867203 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of omegaconf to determine which version is compatible with other requirements. This could take a while.
Collecting omegaconf==2.1.1
  Using cached omegaconf-2.1.1-py3-none-any.whl (74 kB)
INFO: pip is looking at multiple versions of oauthlib to determine which version is compatible with other requirements. This could take a while.
Collecting oauthlib==3.2.0
  Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
INFO: pip is looking at multiple versions of nvsmi to determine which version is compatible with other requirements. This could take a while.
Collecting nvsmi==0.4.2
  Using cached nvsmi-0.4.2-py3-none-any.whl (5.5 kB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
Collecting numpy==1.22.3
  Using cached numpy-1.22.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
INFO: pip is looking at multiple versions of omegaconf to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of numexpr to determine which version is compatible with other requirements. This could take a while.
Collecting numexpr==2.8.1
  Using cached numexpr-2.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (381 kB)
INFO: pip is looking at multiple versions of nodeenv to determine which version is compatible with other requirements. This could take a while.
Collecting nodeenv==1.6.0
  Using cached nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)
INFO: pip is looking at multiple versions of networkx to determine which version is compatible with other requirements. This could take a while.
Collecting networkx==2.7.1
  Using cached networkx-2.7.1-py3-none-any.whl (2.0 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of nanotime to determine which version is compatible with other requirements. This could take a while.
Collecting nanotime==0.5.2
  Using cached nanotime-0.5.2.tar.gz (3.2 kB)
  Preparing metadata (setup.py) ... [?25l- \ | / done
[?25hINFO: pip is looking at multiple versions of mypy-extensions to determine which version is compatible with other requirements. This could take a while.
Collecting mypy-extensions==0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)

1649163872267 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of mypy to determine which version is compatible with other requirements. This could take a while.
Collecting mypy==0.942
  Using cached mypy-0.942-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (16.8 MB)
INFO: pip is looking at multiple versions of multidict to determine which version is compatible with other requirements. This could take a while.
Collecting multidict==6.0.2
  Using cached multidict-6.0.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (121 kB)
INFO: pip is looking at multiple versions of msrest to determine which version is compatible with other requirements. This could take a while.
Collecting msrest==0.6.21
  Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
INFO: pip is looking at multiple versions of mccabe to determine which version is compatible with other requirements. This could take a while.
Collecting mccabe==0.6.1
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
INFO: pip is looking at multiple versions of matplotlib to determine which version is compatible with other requirements. This could take a while.
Collecting matplotlib==3.5.1
  Using cached matplotlib-3.5.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.3 MB)
INFO: pip is looking at multiple versions of markupsafe to determine which version is compatible with other requirements. This could take a while.
Collecting MarkupSafe==2.1.1
  Using cached MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
INFO: pip is looking at multiple versions of markdown to determine which version is compatible with other requirements. This could take a while.
Collecting Markdown==3.3.6
  Using cached Markdown-3.3.6-py3-none-any.whl (97 kB)
INFO: pip is looking at multiple versions of mailchecker to determine which version is compatible with other requirements. This could take a while.
Collecting mailchecker==4.1.14
  Using cached mailchecker-4.1.14.tar.gz (230 kB)
  Preparing metadata (setup.py) ... [?25l- \ | / done
[?25hINFO: pip is looking at multiple versions of lxml to determine which version is compatible with other requirements. This could take a while.
Collecting lxml==4.8.0
  Using cached lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.9 MB)

1649163877338 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of kiwisolver to determine which version is compatible with other requirements. This could take a while.
Collecting kiwisolver==1.4.2
  Using cached kiwisolver-1.4.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
INFO: pip is looking at multiple versions of keyring to determine which version is compatible with other requirements. This could take a while.
Collecting keyring==23.5.0
  Using cached keyring-23.5.0-py3-none-any.whl (33 kB)
INFO: pip is looking at multiple versions of jsonschema to determine which version is compatible with other requirements. This could take a while.
Collecting jsonschema==3.2.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
INFO: pip is looking at multiple versions of joblib to determine which version is compatible with other requirements. This could take a while.
Collecting joblib==1.1.0
  Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
INFO: pip is looking at multiple versions of jmespath to determine which version is compatible with other requirements. This could take a while.
Collecting jmespath==0.10.0
  Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
INFO: pip is looking at multiple versions of jinja2-time to determine which version is compatible with other requirements. This could take a while.
Collecting jinja2-time==0.2.0
  Using cached jinja2_time-0.2.0-py2.py3-none-any.whl (6.4 kB)
INFO: pip is looking at multiple versions of jinja2 to determine which version is compatible with other requirements. This could take a while.
Collecting Jinja2==3.1.1
  Using cached Jinja2-3.1.1-py3-none-any.whl (132 kB)
INFO: pip is looking at multiple versions of jeepney to determine which version is compatible with other requirements. This could take a while.
Collecting jeepney==0.7.1
  Using cached jeepney-0.7.1-py3-none-any.whl (54 kB)
INFO: pip is looking at multiple versions of isodate to determine which version is compatible with other requirements. This could take a while.
Collecting isodate==0.6.1
  Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
INFO: pip is looking at multiple versions of inno-image-vector-nightly to determine which version is compatible with other requirements. This could take a while.
Collecting inno.image-vector-nightly==4.0.0
  Using cached ********************/packages/inno-image-vector-nightly/4.0.0/inno.image_vector_nightly-4.0.0-py3-none-linux_x86_64.whl (2.8 MB)
INFO: pip is looking at multiple versions of inno-iface-nightly to determine which version is compatible with other requirements. This could take a while.
Collecting inno.iface-nightly==4.16.0.0

1649163892435 gauss_user:4gpu DEBUG   Using cached ********************/packages/inno-iface-nightly/4.16.0.0/inno.iface_nightly-4.16.0.0-py3-none-linux_x86_64.whl (942.2 MB)

1649163897513 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of inno-iface-dev-nightly to determine which version is compatible with other requirements. This could take a while.
Collecting inno.iface-dev-nightly==4.16.0.0
  Using cached ********************/packages/inno-iface-dev-nightly/4.16.0.0/inno.iface_dev_nightly-4.16.0.0-py3-none-any.whl (12 kB)
INFO: pip is looking at multiple versions of inno-evaluation to determine which version is compatible with other requirements. This could take a while.
Collecting inno.evaluation==0.3.2
  Using cached ********************/packages/inno-evaluation/0.3.2/inno.evaluation-0.3.2-py3-none-any.whl (206 kB)
INFO: pip is looking at multiple versions of inno-training to determine which version is compatible with other requirements. This could take a while.
Collecting inno-training==0.5.1
  Using cached ********************/packages/inno-training/0.5.1/inno_training-0.5.1-py3-none-any.whl (368 kB)
INFO: pip is looking at multiple versions of inno-primitives to determine which version is compatible with other requirements. This could take a while.
Collecting inno-primitives==0.4.2
  Using cached ********************/packages/inno-primitives/0.4.2/inno_primitives-0.4.2-py3-none-any.whl (27 kB)

1649163902583 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of inno-evaluation to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of inno-nn to determine which version is compatible with other requirements. This could take a while.
Collecting inno-nn==0.9.1
  Using cached ********************/packages/inno-nn/0.9.1/inno_nn-0.9.1-py3-none-any.whl (13 kB)
INFO: pip is looking at multiple versions of inno-eval to determine which version is compatible with other requirements. This could take a while.
Collecting inno-eval==3.0.0
  Using cached ********************/packages/inno-eval/3.0.0/inno_eval-3.0.0-py3-none-any.whl (62 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of inno-training to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of inno-dvc to determine which version is compatible with other requirements. This could take a while.
Collecting inno-dvc==1.44.1
  Using cached ********************/packages/inno-dvc/1.44.1/inno_dvc-1.44.1-py3-none-any.whl (100 kB)

1649163907656 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of inno-dsr to determine which version is compatible with other requirements. This could take a while.
Collecting inno-dsr==0.10.2
  Using cached ********************/packages/inno-dsr/0.10.2/inno_dsr-0.10.2-py3-none-any.whl (87 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

1649163912717 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of inno-eval to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of inno-df to determine which version is compatible with other requirements. This could take a while.
Collecting inno-df==0.9.0
  Using cached ********************/packages/inno-df/0.9.0/inno_df-0.9.0-py3-none-any.whl (92 kB)

1649163917793 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of inno-db to determine which version is compatible with other requirements. This could take a while.
Collecting inno-db==1.9.0
  Using cached ********************/packages/inno-db/1.9.0/inno_db-1.9.0-py3-none-any.whl (95 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

1649163922854 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of inno-dvc to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of inno-common to determine which version is compatible with other requirements. This could take a while.
Collecting inno-common==0.3.0
  Using cached ********************/packages/inno-common/0.3.0/inno_common-0.3.0-py3-none-any.whl (17 kB)

1649163927920 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of iniconfig to determine which version is compatible with other requirements. This could take a while.
Collecting iniconfig==1.1.1
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)

1649163932992 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of importlib-resources to determine which version is compatible with other requirements. This could take a while.
Collecting importlib-resources==5.6.0
  Using cached importlib_resources-5.6.0-py3-none-any.whl (28 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

1649163938053 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of importlib-metadata to determine which version is compatible with other requirements. This could take a while.
Collecting importlib-metadata==4.11.3
  Using cached importlib_metadata-4.11.3-py3-none-any.whl (18 kB)

1649163943132 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of imgaug to determine which version is compatible with other requirements. This could take a while.
Collecting imgaug==0.4.0
  Using cached imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
INFO: pip is looking at multiple versions of imantics to determine which version is compatible with other requirements. This could take a while.
Collecting imantics==0.1.12
  Using cached imantics-0.1.12.tar.gz (13 kB)

1649163948195 gauss_user:4gpu DEBUG   Preparing metadata (setup.py) ... [?25l- \ | / done

1649163953267 gauss_user:4gpu DEBUG [?25hINFO: pip is looking at multiple versions of imageio to determine which version is compatible with other requirements. This could take a while.
Collecting imageio==2.16.1
  Using cached imageio-2.16.1-py3-none-any.whl (3.3 MB)

1649163958336 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of imagecorruptions to determine which version is compatible with other requirements. This could take a while.
Collecting imagecorruptions==1.1.2
  Using cached imagecorruptions-1.1.2-py3-none-any.whl (2.1 MB)

1649163963405 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while.
Collecting idna==2.10
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)

1649163968465 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of identify to determine which version is compatible with other requirements. This could take a while.
Collecting identify==2.4.12
  Using cached identify-2.4.12-py2.py3-none-any.whl (98 kB)

1649163973533 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of hydra-core to determine which version is compatible with other requirements. This could take a while.
Collecting hydra-core==1.1.1
  Using cached hydra_core-1.1.1-py3-none-any.whl (145 kB)

1649163978613 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of hug to determine which version is compatible with other requirements. This could take a while.
Collecting hug==2.6.1
  Using cached hug-2.6.1-py2.py3-none-any.whl (75 kB)

1649163983678 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of h5py to determine which version is compatible with other requirements. This could take a while.
Collecting h5py==3.6.0
  Using cached h5py-3.6.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)

1649163988747 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of grpcio to determine which version is compatible with other requirements. This could take a while.
Collecting grpcio==1.44.0
  Using cached grpcio-1.44.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
INFO: pip is looking at multiple versions of grandalf to determine which version is compatible with other requirements. This could take a while.
Collecting grandalf==0.6
  Using cached grandalf-0.6-py3-none-any.whl (31 kB)

1649163993828 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of google-auth-oauthlib to determine which version is compatible with other requirements. This could take a while.
Collecting google-auth-oauthlib==0.4.6
  Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)

1649163998946 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.
Collecting google-auth==2.6.2
  Using cached google_auth-2.6.2-py2.py3-none-any.whl (156 kB)

1649164004035 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of gitpython to determine which version is compatible with other requirements. This could take a while.
Collecting GitPython==3.1.27
  Using cached GitPython-3.1.27-py3-none-any.whl (181 kB)

1649164009123 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of gitdb to determine which version is compatible with other requirements. This could take a while.
Collecting gitdb==4.0.9
  Using cached gitdb-4.0.9-py3-none-any.whl (63 kB)

1649164014215 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of future to determine which version is compatible with other requirements. This could take a while.
Collecting future==0.18.2
  Using cached future-0.18.2.tar.gz (829 kB)
  Preparing metadata (setup.py) ... [?25l- \ |

1649164019286 gauss_user:4gpu DEBUG  / - done
[?25hINFO: pip is looking at multiple versions of furl to determine which version is compatible with other requirements. This could take a while.
Collecting furl==2.1.3
  Using cached furl-2.1.3-py2.py3-none-any.whl (20 kB)

1649164024356 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of funcy to determine which version is compatible with other requirements. This could take a while.
Collecting funcy==1.17
  Using cached funcy-1.17-py2.py3-none-any.whl (33 kB)

1649164029440 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of ftfy to determine which version is compatible with other requirements. This could take a while.
Collecting ftfy==6.1.1
  Using cached ftfy-6.1.1-py3-none-any.whl (53 kB)

1649164034518 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of fsspec to determine which version is compatible with other requirements. This could take a while.
Collecting fsspec==2022.2.0
  Using cached fsspec-2022.2.0-py3-none-any.whl (134 kB)

1649164039583 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of frozenlist to determine which version is compatible with other requirements. This could take a while.
Collecting frozenlist==1.3.0
  Using cached frozenlist-1.3.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)

1649164044647 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of freezegun to determine which version is compatible with other requirements. This could take a while.
Collecting freezegun==1.2.1
  Using cached freezegun-1.2.1-py3-none-any.whl (16 kB)

1649164049708 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of fonttools to determine which version is compatible with other requirements. This could take a while.
Collecting fonttools==4.31.2
  Using cached fonttools-4.31.2-py3-none-any.whl (899 kB)

1649164054790 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of flufl-lock to determine which version is compatible with other requirements. This could take a while.
Collecting flufl.lock==6.0
  Using cached flufl.lock-6.0.tar.gz (30 kB)
  Preparing metadata (setup.py) ... [?25l- \ | / - done

1649164059892 gauss_user:4gpu DEBUG [?25hINFO: pip is looking at multiple versions of flatten-dict to determine which version is compatible with other requirements. This could take a while.
Collecting flatten-dict==0.4.2
  Using cached flatten_dict-0.4.2-py2.py3-none-any.whl (9.7 kB)

1649164064987 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of flake8-gl-codeclimate to determine which version is compatible with other requirements. This could take a while.
Collecting flake8-gl-codeclimate==0.1.5
  Using cached flake8-gl-codeclimate-0.1.5.tar.gz (4.9 kB)

1649164070071 gauss_user:4gpu DEBUG   Preparing metadata (setup.py) ... [?25l- \ | / - done

1649164075168 gauss_user:4gpu DEBUG [?25hINFO: pip is looking at multiple versions of flake8 to determine which version is compatible with other requirements. This could take a while.
Collecting flake8==4.0.1
  Using cached flake8-4.0.1-py2.py3-none-any.whl (64 kB)

1649164080259 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of filelock to determine which version is compatible with other requirements. This could take a while.
Collecting filelock==3.6.0
  Using cached filelock-3.6.0-py3-none-any.whl (10.0 kB)

1649164085378 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of falcon to determine which version is compatible with other requirements. This could take a while.
Collecting falcon==2.0.0
  Using cached falcon-2.0.0-py2.py3-none-any.whl (163 kB)
INFO: pip is looking at multiple versions of examples to determine which version is compatible with other requirements. This could take a while.
Collecting examples==1.0.2
  Using cached examples-1.0.2-py3-none-any.whl (8.2 kB)

1649164090464 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of dvc to determine which version is compatible with other requirements. This could take a while.
Collecting dvc==2.9.5
  Using cached dvc-2.9.5-py3-none-any.whl (410 kB)

1649164095604 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of dulwich to determine which version is compatible with other requirements. This could take a while.
Collecting dulwich==0.20.35
  Using cached dulwich-0.20.35-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (552 kB)

1649164100695 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of dpath to determine which version is compatible with other requirements. This could take a while.
Collecting dpath==2.0.6
  Using cached dpath-2.0.6-py3-none-any.whl (15 kB)

1649164105776 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of docutils to determine which version is compatible with other requirements. This could take a while.
Collecting docutils==0.18.1
  Using cached docutils-0.18.1-py2.py3-none-any.whl (570 kB)

1649164110871 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of distro to determine which version is compatible with other requirements. This could take a while.
Collecting distro==1.7.0
  Using cached distro-1.7.0-py3-none-any.whl (20 kB)

1649164115988 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of distlib to determine which version is compatible with other requirements. This could take a while.
Collecting distlib==0.3.4
  Using cached distlib-0.3.4-py2.py3-none-any.whl (461 kB)

1649164121127 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of diskcache to determine which version is compatible with other requirements. This could take a while.
Collecting diskcache==5.4.0
  Using cached diskcache-5.4.0-py3-none-any.whl (44 kB)

1649164126223 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of dictdiffer to determine which version is compatible with other requirements. This could take a while.
Collecting dictdiffer==0.9.0
  Using cached dictdiffer-0.9.0-py2.py3-none-any.whl (16 kB)
INFO: pip is looking at multiple versions of cython to determine which version is compatible with other requirements. This could take a while.
Collecting Cython==0.29.28
  Using cached Cython-0.29.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)

1649164131310 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cycler to determine which version is compatible with other requirements. This could take a while.
Collecting cycler==0.11.0
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)

1649164136398 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cryptography to determine which version is compatible with other requirements. This could take a while.
Collecting cryptography==36.0.2
  Using cached cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)

1649164141495 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cruft to determine which version is compatible with other requirements. This could take a while.
Collecting cruft==1.4.0
  Using cached cruft-1.4.0-py3-none-any.whl (14 kB)

1649164146573 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of crc32c to determine which version is compatible with other requirements. This could take a while.
Collecting crc32c==2.2.post0
  Using cached crc32c-2.2.post0-cp38-cp38-manylinux2010_x86_64.whl (48 kB)

1649164151637 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of coverage to determine which version is compatible with other requirements. This could take a while.
Collecting coverage==6.3.2
  Using cached coverage-6.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)

1649164156714 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cookiecutter to determine which version is compatible with other requirements. This could take a while.
Collecting cookiecutter==1.7.3
  Using cached cookiecutter-1.7.3-py2.py3-none-any.whl (34 kB)

1649164161786 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of configobj to determine which version is compatible with other requirements. This could take a while.
Collecting configobj==5.0.6
  Using cached configobj-5.0.6.tar.gz (33 kB)
  Preparing metadata (setup.py) ... [?25l- \ | / done

1649164166885 gauss_user:4gpu DEBUG [?25hINFO: pip is looking at multiple versions of commonmark to determine which version is compatible with other requirements. This could take a while.
Collecting commonmark==0.9.1
  Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)

1649164171957 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of colorama to determine which version is compatible with other requirements. This could take a while.
Collecting colorama==0.4.4
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)

1649164177131 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of click to determine which version is compatible with other requirements. This could take a while.
Collecting click==8.1.0
  Using cached click-8.1.0-py3-none-any.whl (96 kB)
INFO: pip is looking at multiple versions of charset-normalizer to determine which version is compatible with other requirements. This could take a while.
Collecting charset-normalizer==2.0.12
  Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)

1649164182204 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while.
Collecting chardet==4.0.0
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)

1649164187277 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cfgv to determine which version is compatible with other requirements. This could take a while.
Collecting cfgv==3.3.1
  Using cached cfgv-3.3.1-py2.py3-none-any.whl (7.3 kB)

1649164192357 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cffi to determine which version is compatible with other requirements. This could take a while.
Collecting cffi==1.15.0
  Using cached cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)

1649164197433 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while.
Collecting certifi==2021.10.8
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)

1649164202510 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cattrs to determine which version is compatible with other requirements. This could take a while.
Collecting cattrs==1.0.0
  Using cached cattrs-1.0.0-py2.py3-none-any.whl (14 kB)

1649164207586 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of cachetools to determine which version is compatible with other requirements. This could take a while.
Collecting cachetools==5.0.0
  Using cached cachetools-5.0.0-py3-none-any.whl (9.1 kB)

1649164212666 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of botocore to determine which version is compatible with other requirements. This could take a while.
Collecting botocore==1.23.24
  Using cached botocore-1.23.24-py3-none-any.whl (8.4 MB)
INFO: pip is looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while.
Collecting boto3==1.20.24
  Using cached boto3-1.20.24-py3-none-any.whl (131 kB)

1649164217726 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of bleach to determine which version is compatible with other requirements. This could take a while.
Collecting bleach==4.1.0
  Using cached bleach-4.1.0-py2.py3-none-any.whl (157 kB)

1649164222785 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of black to determine which version is compatible with other requirements. This could take a while.
Collecting black==22.1.0
  Using cached black-22.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)

1649164227859 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of binaryornot to determine which version is compatible with other requirements. This could take a while.
Collecting binaryornot==0.4.4
  Using cached binaryornot-0.4.4-py2.py3-none-any.whl (9.0 kB)

1649164232927 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of azure-common to determine which version is compatible with other requirements. This could take a while.
Collecting azure-common==1.1.28
  Using cached azure_common-1.1.28-py2.py3-none-any.whl (14 kB)

1649164237999 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of azure-cognitiveservices-vision-face to determine which version is compatible with other requirements. This could take a while.
Collecting azure-cognitiveservices-vision-face==0.6.0
  Using cached azure_cognitiveservices_vision_face-0.6.0-py2.py3-none-any.whl (67 kB)
INFO: pip is looking at multiple versions of azure-cognitiveservices-anomalydetector to determine which version is compatible with other requirements. This could take a while.
Collecting azure-cognitiveservices-anomalydetector==0.3.0
  Using cached azure_cognitiveservices_anomalydetector-0.3.0-py2.py3-none-any.whl (15 kB)

1649164243071 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Collecting attrs==20.3.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)

1649164248134 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of atpublic to determine which version is compatible with other requirements. This could take a while.
Collecting atpublic==3.0.1
  Using cached atpublic-3.0.1-py3-none-any.whl (4.8 kB)

1649164253194 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of asyncssh to determine which version is compatible with other requirements. This could take a while.
Collecting asyncssh==2.8.1
  Using cached asyncssh-2.8.1-py3-none-any.whl (287 kB)

1649164258276 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of async-timeout to determine which version is compatible with other requirements. This could take a while.
Collecting async-timeout==4.0.2
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)

1649164263340 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of astropy to determine which version is compatible with other requirements. This could take a while.
Collecting astropy==5.0.3
  Using cached astropy-5.0.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (11.2 MB)

1649164268412 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of arrow to determine which version is compatible with other requirements. This could take a while.
Collecting arrow==1.2.2
  Using cached arrow-1.2.2-py3-none-any.whl (64 kB)

1649164273474 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of appdirs to determine which version is compatible with other requirements. This could take a while.
Collecting appdirs==1.4.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)

1649164283572 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of antlr4-python3-runtime to determine which version is compatible with other requirements. This could take a while.
Collecting antlr4-python3-runtime==4.8
  Using cached antlr4-python3-runtime-4.8.tar.gz (112 kB)
  Preparing metadata (setup.py) ... [?25l- \ | / done

1649164293664 gauss_user:4gpu DEBUG [?25hINFO: pip is looking at multiple versions of astropy to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of aiosignal to determine which version is compatible with other requirements. This could take a while.
Collecting aiosignal==1.2.0
  Using cached aiosignal-1.2.0-py3-none-any.whl (8.2 kB)

1649164303758 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of aioitertools to determine which version is compatible with other requirements. This could take a while.
Collecting aioitertools==0.10.0
  Using cached aioitertools-0.10.0-py3-none-any.whl (23 kB)

1649164308818 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of aiohttp-retry to determine which version is compatible with other requirements. This could take a while.
Collecting aiohttp-retry==2.4.6
  Using cached aiohttp_retry-2.4.6-py3-none-any.whl (7.7 kB)

1649164313887 gauss_user:4gpu DEBUG INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

1649164318962 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of aiohttp to determine which version is compatible with other requirements. This could take a while.
Collecting aiohttp==3.8.1
  Using cached aiohttp-3.8.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.3 MB)

1649164329052 gauss_user:4gpu DEBUG INFO: pip is looking at multiple versions of aiobotocore to determine which version is compatible with other requirements. This could take a while.
Collecting aiobotocore==2.1.2
  Using cached aiobotocore-2.1.2.tar.gz (58 kB)
  Preparing metadata (setup.py) ... [?25l- \ | / done

1649164339134 gauss_user:4gpu DEBUG [?25hINFO: pip is looking at multiple versions of absl-py to determine which version is compatible with other requirements. This could take a while.
Collecting absl-py==1.0.0
  Using cached absl_py-1.0.0-py3-none-any.whl (126 kB)

1649164344223 gauss_user:4gpu DEBUG ERROR: Cannot install -r /tmp/cached-reqsr9tlzb6r.txt (line 10), -r /tmp/cached-reqsr9tlzb6r.txt (line 122), -r /tmp/cached-reqsr9tlzb6r.txt (line 139), -r /tmp/cached-reqsr9tlzb6r.txt (line 162), -r /tmp/cached-reqsr9tlzb6r.txt (line 87), -r /tmp/cached-reqsr9tlzb6r.txt (line 88), -r /tmp/cached-reqsr9tlzb6r.txt (line 91), -r /tmp/cached-reqsr9tlzb6r.txt (line 92), PyYAML==5.4.1 and python-benedict==0.25.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested PyYAML==5.4.1
    astropy 5.0.3 depends on PyYAML>=3.13
    inno-dvc 1.44.1 depends on PyYaml
    inno-eval 3.0.0 depends on pyyaml
    inno-training 0.5.1 depends on PyYAML
    inno-evaluation 0.3.2 depends on PyYAML
    omegaconf 2.1.1 depends on PyYAML>=5.1.0
    pre-commit 2.17.0 depends on pyyaml>=5.1
    python-benedict 0.25.0 depends on pyyaml<7.0 and >=6.0
    The user requested PyYAML==5.4.1
    astropy 5.0.3 depends on PyYAML>=3.13
    inno-dvc 1.44.1 depends on PyYaml
    inno-eval 3.0.0 depends on pyyaml
    inno-training 0.5.1 depends on PyYAML
    inno-evaluation 0.3.2 depends on PyYAML
    omegaconf 2.1.1 depends on PyYAML>=5.1.0
    pre-commit 2.17.0 depends on pyyaml>=5.1
    python-benedict 0.25.0 depends on pyyaml<7.0 and >=6.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts


1649164349267 gauss_user:4gpu DEBUG
clearml_agent: ERROR: Could not install task requirements!
Command '['/opt/venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'install', '-r', '/tmp/cached-reqsr9tlzb6r.txt', '--extra-index-url', '******************', '--extra-index-url', 'https://download.pytorch.org/whl/cu101', '--upgrade']' returned non-zero exit status 1.

1649164349307 gauss_user:4gpu DEBUG Process failed, exit code 1

I've "censored" some informations, but I hope that everything is clear from the log.

As you can see in the second log, the agent tries to install all dependencies (which he already has, because it's the same container).

jkhenning commented 2 years ago

Hi @Honzys,

Again, sorry, I missed this 🙁

This difference seems very strange. Are you sure you clear the INSTALLED PACKAGES section in the reset?

elazarcoh commented 5 months ago

I can confirm that manually clearing the INSTALLED PACKAGES section fixes the problem for me.

I guess the "Reset" does not clear the installed packages by design? If not, maybe consider clearing it also with the task reset.