I have a magical problem

afterimagex commented 2 years ago

I try to use clearml to train yolov5. Its code repository is here When I train locally, everything is normal. When I choose remote training the program will be blocked in the docker after the pip installation is executed, and there is nothing Error, but it will not continue training, it will only run indefinitely someone can help me.

jkhenning commented 2 years ago

Hi @afterimagex,

Can you share your training code? How exactly are you running it remotely? Are you using a ClearML Agent? How are you setting it up?

afterimagex commented 2 years ago

My clearml-agent execution command is：

clearml-agent daemon --gpus 0,1 --queue yolov5 --detached --docker pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime

I just fork the source code of yolov5, and then add Task.init, my training code is：

def main():
    training_loop:
    ···

def parse_opt(known=False):
    task = Task.init(project_name="Detection", task_name="yolov5-train", reuse_last_task_id=False)
    task.set_base_docker("pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime")
    parser = argparse.ArgumentParser()
    ···
    opt = parser.parse_known_args()[0] if known else parser.parse_args()
    task.execute_remotely(queue_name="yolov5")

if __name__ == "__main__":
    opt = parse_opt(True)
    main(opt)

and the code repository can be found here: https://github.com/afterimagex/yolov5/blob/clearml/train.py change line is https://github.com/afterimagex/yolov5/blob/ea68ec2a0353c1954fe881ffc856b29db1e32879/train.py#L455

It is worth mentioning that in the ~ / clearml.conf file, i set system_ site_ packages: true

afterimagex commented 2 years ago

here is my log:

1637513593647 Blanks-MacBook-Pro.local info ClearML Task: created new task id=25d37e907179476f9b2849c8d293d477
ClearML results page: http://192.168.100.44:8080/projects/5247bc72258a4675bdac765fda89d3ed/experiments/25d37e907179476f9b2849c8d293d477/output/log
2021-11-22 00:53:14,582 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
1637513595829 Blanks-MacBook-Pro.local info 2021-11-22 00:53:15,829 - clearml.Task - INFO - Finished repository detection and package analysis
1637513598498 MS-7A93:gpu0,1 INFO task 25d37e907179476f9b2849c8d293d477 pulled from 863ad79f53074662a7bc3aedfd9935aa by worker MS-7A93:gpu0,1

1637513598536 MS-7A93:gpu0,1 INFO Running Task 25d37e907179476f9b2849c8d293d477 inside default docker: pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime arguments: []

1637513598565 MS-7A93:gpu0,1 INFO Executing: ['docker', 'run', '-t', '--gpus', '"device=0,1"', '-l', 'clearml-worker-id=MS-7A93:gpu0,1', '-l', 'clearml-parent-worker-id=MS-7A93:gpu0,1', '-e', 'CLEARML_WORKER_ID=MS-7A93:gpu0,1', '-e', 'CLEARML_DOCKER_IMAGE=pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime', '-v', '/home/pc/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent.eu2yas7g.cfg:/root/clearml.conf', '-v', '/tmp/clearml_agent.ssh.jnivxzb9:/root/.ssh', '-v', '/home/pc/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/pc/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/pc/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/pc/.clearml/cache:/clearml_agent_cache', '-v', '/home/pc/.clearml/vcs-cache:/root/.clearml/vcs-cache', '--rm', 'pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime', 'bash', '-c', 'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL libsm6 libxext6 libxrender-dev libglib2.0-0" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; for i in {10..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update && apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /root/clearml.conf /root/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring  --id 25d37e907179476f9b2849c8d293d477']

1637513603608 MS-7A93:gpu0,1 DEBUG WARNING: Error loading config file: /home/pc/.docker/config.json: open /home/pc/.docker/config.json: permission denied
/opt/conda/bin/python3.7
pip 21.0.1 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2430 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
23% [7 Packages 56.0 kB/1344 kB 4%] [Waiting for headers]

1637513608636 MS-7A93:gpu0,1 DEBUG Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1442 kB]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [666 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [26.8 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
40% [11 Packages 940 kB/11.3 MB 8%]                                962 kB/s 17s

1637513613668 MS-7A93:gpu0,1 DEBUG 56% [11 Packages 5571 kB/11.3 MB 49%]                              724 kB/s 16s

1637513618708 MS-7A93:gpu0,1 DEBUG Get:12 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2867 kB]
81% [11 Packages store 0 B] [13 Packages 1586 kB/2867 kB 55%]      1203 kB/s 3s

1637513623835 MS-7A93:gpu0,1 DEBUG Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2222 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [699 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [34.4 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Fetched 23.8 MB in 21s (1133 kB/s)

The following additional packages will be installed:
  git-man krb5-locales less libasn1-8-heimdal libbsd0 libcurl3-gnutls libedit2
  liberror-perl libexpat1 libgdbm-compat4 libgdbm5 libglib2.0-data
  libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
  libheimntlm0-heimdal libhx509-5-heimdal libice6 libicu60 libk5crypto3
  libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2
  libldap-common libnghttp2-14 libperl5.26 libpsl5 libpthread-stubs0-dev
  libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db
  libsqlite3-0 libssl1.0.0 libwind0-heimdal libx11-6 libx11-data libx11-dev
  libx11-doc libxau-dev libxau6 libxcb1 libxcb1-dev libxdmcp-dev libxdmcp6
  libxml2 libxmuu1 libxrender1 netbase openssh-client patch perl
  perl-modules-5.26 publicsuffix shared-mime-info x11-common x11proto-core-dev
  x11proto-dev xauth xdg-user-dirs xorg-sgml-doctools xtrans-dev
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
  git-gui gitk gitweb git-cvs git-mediawiki git-svn gdbm-l10n krb5-doc
  krb5-user libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal
  libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql libxcb-doc
  keychain libpam-ssh monkeysphere ssh-askpass ed diffutils-doc perl-doc
  libterm-readline-gnu-perl | libterm-readline-perl-perl make
The following NEW packages will be installed:
  git git-man krb5-locales less libasn1-8-heimdal libbsd0 libcurl3-gnutls
  libedit2 liberror-perl libexpat1 libgdbm-compat4 libgdbm5 libglib2.0-0
  libglib2.0-data libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal
  libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libice6
  libicu60 libk5crypto3 libkeyutils1 libkrb5-26-heimdal libkrb5-3
  libkrb5support0 libldap-2.4-2 libldap-common libnghttp2-14 libperl5.26
  libpsl5 libpthread-stubs0-dev libroken18-heimdal librtmp1 libsasl2-2
  libsasl2-modules libsasl2-modules-db libsm6 libsqlite3-0 libssl1.0.0
  libwind0-heimdal libx11-6 libx11-data libx11-dev libx11-doc libxau-dev
  libxau6 libxcb1 libxcb1-dev libxdmcp-dev libxdmcp6 libxext6 libxml2 libxmuu1
  libxrender-dev libxrender1 netbase openssh-client patch perl
  perl-modules-5.26 publicsuffix shared-mime-info x11-common x11proto-core-dev
  x11proto-dev xauth xdg-user-dirs xorg-sgml-doctools xtrans-dev
0 upgraded, 71 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/30.5 MB of archives.
After this operation, 154 MB of additional disk space will be used.
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libxau6:amd64.
(Reading database ... 5946 files and directories currently installed.)
Preparing to unpack .../00-libxau6_1%3a1.0.8-1ubuntu1_amd64.deb ...
Unpacking libxau6:amd64 (1:1.0.8-1ubuntu1) ...
Selecting previously unselected package libbsd0:amd64.
Preparing to unpack .../01-libbsd0_0.8.7-1ubuntu0.1_amd64.deb ...
Unpacking libbsd0:amd64 (0.8.7-1ubuntu0.1) ...
Selecting previously unselected package libxdmcp6:amd64.
Preparing to unpack .../02-libxdmcp6_1%3a1.1.2-3_amd64.deb ...
Unpacking libxdmcp6:amd64 (1:1.1.2-3) ...
Selecting previously unselected package libxcb1:amd64.
Preparing to unpack .../03-libxcb1_1.13-2~ubuntu18.04_amd64.deb ...
Unpacking libxcb1:amd64 (1.13-2~ubuntu18.04) ...
Selecting previously unselected package libx11-data.
Preparing to unpack .../04-libx11-data_2%3a1.6.4-3ubuntu0.4_all.deb ...
Unpacking libx11-data (2:1.6.4-3ubuntu0.4) ...
Selecting previously unselected package libx11-6:amd64.
Preparing to unpack .../05-libx11-6_2%3a1.6.4-3ubuntu0.4_amd64.deb ...
Unpacking libx11-6:amd64 (2:1.6.4-3ubuntu0.4) ...
Selecting previously unselected package libxext6:amd64.
Preparing to unpack .../06-libxext6_2%3a1.3.3-1_amd64.deb ...
Unpacking libxext6:amd64 (2:1.3.3-1) ...
Selecting previously unselected package x11-common.
Preparing to unpack .../07-x11-common_1%3a7.7+19ubuntu7.1_all.deb ...
dpkg-query: no packages found matching nux-tools
Unpacking x11-common (1:7.7+19ubuntu7.1) ...
Selecting previously unselected package libice6:amd64.
Preparing to unpack .../08-libice6_2%3a1.0.9-2_amd64.deb ...
Unpacking libice6:amd64 (2:1.0.9-2) ...
Selecting previously unselected package libsm6:amd64.
Preparing to unpack .../09-libsm6_2%3a1.2.2-1_amd64.deb ...
Unpacking libsm6:amd64 (2:1.2.2-1) ...

1637513628885 MS-7A93:gpu0,1 DEBUG Selecting previously unselected package perl-modules-5.26.
Preparing to unpack .../10-perl-modules-5.26_5.26.1-6ubuntu0.5_all.deb ...
Unpacking perl-modules-5.26 (5.26.1-6ubuntu0.5) ...
Selecting previously unselected package libgdbm5:amd64.
Preparing to unpack .../11-libgdbm5_1.14.1-6_amd64.deb ...
Unpacking libgdbm5:amd64 (1.14.1-6) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../12-libgdbm-compat4_1.14.1-6_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.14.1-6) ...
Selecting previously unselected package libperl5.26:amd64.
Preparing to unpack .../13-libperl5.26_5.26.1-6ubuntu0.5_amd64.deb ...
Unpacking libperl5.26:amd64 (5.26.1-6ubuntu0.5) ...
Selecting previously unselected package perl.
Preparing to unpack .../14-perl_5.26.1-6ubuntu0.5_amd64.deb ...
Unpacking perl (5.26.1-6ubuntu0.5) ...
Selecting previously unselected package less.
Preparing to unpack .../15-less_487-0.1_amd64.deb ...
Unpacking less (487-0.1) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../16-libexpat1_2.2.5-3ubuntu0.2_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.5-3ubuntu0.2) ...
Selecting previously unselected package libglib2.0-0:amd64.
Preparing to unpack .../17-libglib2.0-0_2.56.4-0ubuntu0.18.04.8_amd64.deb ...
Unpacking libglib2.0-0:amd64 (2.56.4-0ubuntu0.18.04.8) ...
Selecting previously unselected package libglib2.0-data.
Preparing to unpack .../18-libglib2.0-data_2.56.4-0ubuntu0.18.04.8_all.deb ...
Unpacking libglib2.0-data (2.56.4-0ubuntu0.18.04.8) ...
Selecting previously unselected package libicu60:amd64.
Preparing to unpack .../19-libicu60_60.2-3ubuntu3.2_amd64.deb ...
Unpacking libicu60:amd64 (60.2-3ubuntu3.2) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../20-libsqlite3-0_3.22.0-1ubuntu0.4_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.22.0-1ubuntu0.4) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../21-libxml2_2.9.4+dfsg1-6.1ubuntu1.4_amd64.deb ...
Unpacking libxml2:amd64 (2.9.4+dfsg1-6.1ubuntu1.4) ...
Selecting previously unselected package netbase.
Preparing to unpack .../22-netbase_5.4_all.deb ...
Unpacking netbase (5.4) ...
Selecting previously unselected package shared-mime-info.
Preparing to unpack .../23-shared-mime-info_1.9-2_amd64.deb ...
Unpacking shared-mime-info (1.9-2) ...
Selecting previously unselected package xdg-user-dirs.
Preparing to unpack .../24-xdg-user-dirs_0.17-1ubuntu1_amd64.deb ...
Unpacking xdg-user-dirs (0.17-1ubuntu1) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../25-krb5-locales_1.16-2ubuntu0.2_all.deb ...
Unpacking krb5-locales (1.16-2ubuntu0.2) ...
Selecting previously unselected package libedit2:amd64.
Preparing to unpack .../26-libedit2_3.1-20170329-1_amd64.deb ...
Unpacking libedit2:amd64 (3.1-20170329-1) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../27-libkrb5support0_1.16-2ubuntu0.2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.16-2ubuntu0.2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../28-libk5crypto3_1.16-2ubuntu0.2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.16-2ubuntu0.2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../29-libkeyutils1_1.5.9-9.2ubuntu2_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.5.9-9.2ubuntu2) ...

1637513633964 MS-7A93:gpu0,1 DEBUG Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../30-libkrb5-3_1.16-2ubuntu0.2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.16-2ubuntu0.2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../31-libgssapi-krb5-2_1.16-2ubuntu0.2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.16-2ubuntu0.2) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../32-libpsl5_0.19.1-5build1_amd64.deb ...
Unpacking libpsl5:amd64 (0.19.1-5build1) ...
Selecting previously unselected package libssl1.0.0:amd64.
Preparing to unpack .../33-libssl1.0.0_1.0.2n-1ubuntu5.7_amd64.deb ...
Unpacking libssl1.0.0:amd64 (1.0.2n-1ubuntu5.7) ...
Selecting previously unselected package libxmuu1:amd64.
Preparing to unpack .../34-libxmuu1_2%3a1.1.2-2_amd64.deb ...
Unpacking libxmuu1:amd64 (2:1.1.2-2) ...
Selecting previously unselected package openssh-client.
Preparing to unpack .../35-openssh-client_1%3a7.6p1-4ubuntu0.5_amd64.deb ...
Unpacking openssh-client (1:7.6p1-4ubuntu0.5) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../36-publicsuffix_20180223.1310-1_all.deb ...
Unpacking publicsuffix (20180223.1310-1) ...
Selecting previously unselected package xauth.
Preparing to unpack .../37-xauth_1%3a1.0.10-1_amd64.deb ...
Unpacking xauth (1:1.0.10-1) ...
Selecting previously unselected package libroken18-heimdal:amd64.
Preparing to unpack .../38-libroken18-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libroken18-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libasn1-8-heimdal:amd64.
Preparing to unpack .../39-libasn1-8-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libasn1-8-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libheimbase1-heimdal:amd64.
Preparing to unpack .../40-libheimbase1-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libheimbase1-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libhcrypto4-heimdal:amd64.
Preparing to unpack .../41-libhcrypto4-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libhcrypto4-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libwind0-heimdal:amd64.
Preparing to unpack .../42-libwind0-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libwind0-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libhx509-5-heimdal:amd64.
Preparing to unpack .../43-libhx509-5-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libhx509-5-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libkrb5-26-heimdal:amd64.
Preparing to unpack .../44-libkrb5-26-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libkrb5-26-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libheimntlm0-heimdal:amd64.
Preparing to unpack .../45-libheimntlm0-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libheimntlm0-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libgssapi3-heimdal:amd64.
Preparing to unpack .../46-libgssapi3-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libgssapi3-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../47-libsasl2-modules-db_2.1.27~101-g0780600+dfsg-3ubuntu2.3_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2.3) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../48-libsasl2-2_2.1.27~101-g0780600+dfsg-3ubuntu2.3_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2.3) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../49-libldap-common_2.4.45+dfsg-1ubuntu1.10_all.deb ...
Unpacking libldap-common (2.4.45+dfsg-1ubuntu1.10) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../50-libldap-2.4-2_2.4.45+dfsg-1ubuntu1.10_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.45+dfsg-1ubuntu1.10) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../51-libnghttp2-14_1.30.0-1ubuntu1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.30.0-1ubuntu1) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../52-librtmp1_2.4+20151223.gitfa8646d.1-1_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-1) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../53-libcurl3-gnutls_7.58.0-2ubuntu3.16_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.16) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../54-liberror-perl_0.17025-1_all.deb ...
Unpacking liberror-perl (0.17025-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../55-git-man_1%3a2.17.1-1ubuntu0.9_all.deb ...
Unpacking git-man (1:2.17.1-1ubuntu0.9) ...

1637513639068 MS-7A93:gpu0,1 DEBUG Selecting previously unselected package git.
Preparing to unpack .../56-git_1%3a2.17.1-1ubuntu0.9_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.9) ...
Selecting previously unselected package libpthread-stubs0-dev:amd64.
Preparing to unpack .../57-libpthread-stubs0-dev_0.3-4_amd64.deb ...
Unpacking libpthread-stubs0-dev:amd64 (0.3-4) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../58-libsasl2-modules_2.1.27~101-g0780600+dfsg-3ubuntu2.3_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2.3) ...
Selecting previously unselected package xorg-sgml-doctools.
Preparing to unpack .../59-xorg-sgml-doctools_1%3a1.11-1_all.deb ...
Unpacking xorg-sgml-doctools (1:1.11-1) ...
Selecting previously unselected package x11proto-dev.
Preparing to unpack .../60-x11proto-dev_2018.4-4_all.deb ...
Unpacking x11proto-dev (2018.4-4) ...
Selecting previously unselected package x11proto-core-dev.
Preparing to unpack .../61-x11proto-core-dev_2018.4-4_all.deb ...
Unpacking x11proto-core-dev (2018.4-4) ...
Selecting previously unselected package libxau-dev:amd64.
Preparing to unpack .../62-libxau-dev_1%3a1.0.8-1ubuntu1_amd64.deb ...
Unpacking libxau-dev:amd64 (1:1.0.8-1ubuntu1) ...
Selecting previously unselected package libxdmcp-dev:amd64.
Preparing to unpack .../63-libxdmcp-dev_1%3a1.1.2-3_amd64.deb ...
Unpacking libxdmcp-dev:amd64 (1:1.1.2-3) ...
Selecting previously unselected package xtrans-dev.
Preparing to unpack .../64-xtrans-dev_1.3.5-1_all.deb ...
Unpacking xtrans-dev (1.3.5-1) ...
Selecting previously unselected package libxcb1-dev:amd64.
Preparing to unpack .../65-libxcb1-dev_1.13-2~ubuntu18.04_amd64.deb ...
Unpacking libxcb1-dev:amd64 (1.13-2~ubuntu18.04) ...
Selecting previously unselected package libx11-dev:amd64.
Preparing to unpack .../66-libx11-dev_2%3a1.6.4-3ubuntu0.4_amd64.deb ...
Unpacking libx11-dev:amd64 (2:1.6.4-3ubuntu0.4) ...
Selecting previously unselected package libx11-doc.
Preparing to unpack .../67-libx11-doc_2%3a1.6.4-3ubuntu0.4_all.deb ...
Unpacking libx11-doc (2:1.6.4-3ubuntu0.4) ...
Selecting previously unselected package libxrender1:amd64.
Preparing to unpack .../68-libxrender1_1%3a0.9.10-1_amd64.deb ...
Unpacking libxrender1:amd64 (1:0.9.10-1) ...
Selecting previously unselected package libxrender-dev:amd64.
Preparing to unpack .../69-libxrender-dev_1%3a0.9.10-1_amd64.deb ...
Unpacking libxrender-dev:amd64 (1:0.9.10-1) ...
Selecting previously unselected package patch.
Preparing to unpack .../70-patch_2.7.6-2ubuntu1.1_amd64.deb ...
Unpacking patch (2.7.6-2ubuntu1.1) ...
Setting up libedit2:amd64 (3.1-20170329-1) ...
Setting up git-man (1:2.17.1-1ubuntu0.9) ...
Setting up libexpat1:amd64 (2.2.5-3ubuntu0.2) ...
Setting up libicu60:amd64 (60.2-3ubuntu3.2) ...
Setting up less (487-0.1) ...
Setting up libssl1.0.0:amd64 (1.0.2n-1ubuntu5.7) ...
Setting up libnghttp2-14:amd64 (1.30.0-1ubuntu1) ...
Setting up libldap-common (2.4.45+dfsg-1ubuntu1.10) ...
Setting up libpthread-stubs0-dev:amd64 (0.3-4) ...
Setting up libpsl5:amd64 (0.19.1-5build1) ...
Setting up libglib2.0-0:amd64 (2.56.4-0ubuntu0.18.04.8) ...
No schema files found: doing nothing.
Setting up xorg-sgml-doctools (1:1.11-1) ...
Setting up libsasl2-modules-db:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2.3) ...
Setting up libsasl2-2:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2.3) ...
Setting up libroken18-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-1) ...
Setting up perl-modules-5.26 (5.26.1-6ubuntu0.5) ...
Setting up libgdbm5:amd64 (1.14.1-6) ...
Setting up libbsd0:amd64 (0.8.7-1ubuntu0.1) ...
Setting up libkrb5support0:amd64 (1.16-2ubuntu0.2) ...
Setting up libxml2:amd64 (2.9.4+dfsg1-6.1ubuntu1.4) ...
Setting up x11proto-dev (2018.4-4) ...
Setting up xtrans-dev (1.3.5-1) ...
Setting up patch (2.7.6-2ubuntu1.1) ...
Setting up libglib2.0-data (2.56.4-0ubuntu0.18.04.8) ...
Setting up krb5-locales (1.16-2ubuntu0.2) ...
Setting up publicsuffix (20180223.1310-1) ...
Setting up libheimbase1-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libsqlite3-0:amd64 (3.22.0-1ubuntu0.4) ...
Setting up libx11-doc (2:1.6.4-3ubuntu0.4) ...
Setting up shared-mime-info (1.9-2) ...
Setting up libxdmcp6:amd64 (1:1.1.2-3) ...
Setting up libgdbm-compat4:amd64 (1.14.1-6) ...
Setting up libkeyutils1:amd64 (1.5.9-9.2ubuntu2) ...
Setting up libsasl2-modules:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2.3) ...
Setting up x11-common (1:7.7+19ubuntu7.1) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up xdg-user-dirs (0.17-1ubuntu1) ...
Setting up libx11-data (2:1.6.4-3ubuntu0.4) ...
Setting up libxau6:amd64 (1:1.0.8-1ubuntu1) ...
Setting up netbase (5.4) ...
Setting up libk5crypto3:amd64 (1.16-2ubuntu0.2) ...
Setting up x11proto-core-dev (2018.4-4) ...
Setting up libwind0-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libasn1-8-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libhcrypto4-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libxau-dev:amd64 (1:1.0.8-1ubuntu1) ...
Setting up libhx509-5-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libxdmcp-dev:amd64 (1:1.1.2-3) ...
Setting up libperl5.26:amd64 (5.26.1-6ubuntu0.5) ...
Setting up libice6:amd64 (2:1.0.9-2) ...
Setting up libkrb5-3:amd64 (1.16-2ubuntu0.2) ...

1637513644029 MS-7A93:gpu0,1 DEBUG Setting up libkrb5-26-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libxcb1:amd64 (1.13-2~ubuntu18.04) ...
Setting up libheimntlm0-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libsm6:amd64 (2:1.2.2-1) ...
Setting up libx11-6:amd64 (2:1.6.4-3ubuntu0.4) ...
Setting up libxmuu1:amd64 (2:1.1.2-2) ...
Setting up libgssapi-krb5-2:amd64 (1.16-2ubuntu0.2) ...
Setting up perl (5.26.1-6ubuntu0.5) ...
Setting up libxrender1:amd64 (1:0.9.10-1) ...
Setting up libxcb1-dev:amd64 (1.13-2~ubuntu18.04) ...
Setting up libx11-dev:amd64 (2:1.6.4-3ubuntu0.4) ...
Setting up libgssapi3-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up openssh-client (1:7.6p1-4ubuntu0.5) ...
Setting up libxext6:amd64 (2:1.3.3-1) ...
Setting up liberror-perl (0.17025-1) ...
Setting up libxrender-dev:amd64 (1:0.9.10-1) ...
Setting up xauth (1:1.0.10-1) ...
Setting up libldap-2.4-2:amd64 (2.4.45+dfsg-1ubuntu1.10) ...
Setting up libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.16) ...
Setting up git (1:2.17.1-1ubuntu0.9) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Collecting pip<20.2
  Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.0.1
    Uninstalling pip-21.0.1:
      Successfully uninstalled pip-21.0.1

1637513649099 MS-7A93:gpu0,1 DEBUG Successfully installed pip-20.1.1
Collecting clearml-agent
  Using cached clearml_agent-1.1.1-py3-none-any.whl (360 kB)
Processing /root/.cache/pip/wheels/cb/20/f9/ff360765ce6f9fc078d6599c10a8f36496e5b5011a29df1ae3/pyhocon-0.3.58-py3-none-any.whl
Collecting pathlib2<2.4.0,>=2.3.0
  Using cached pathlib2-2.3.6-py2.py3-none-any.whl (17 kB)
Collecting six<1.16.0,>=1.11.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied, skipping upgrade: PyYAML<5.5.0,>=3.12 in /opt/conda/lib/python3.7/site-packages (from clearml-agent) (5.4.1)
Collecting pyparsing<2.5.0,>=2.0.3
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting furl<2.2.0,>=2.0.0
  Using cached furl-2.1.3-py2.py3-none-any.whl (20 kB)
Collecting python-dateutil<2.9.0,>=2.4.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Requirement already satisfied, skipping upgrade: psutil<5.9.0,>=3.4.2 in /opt/conda/lib/python3.7/site-packages (from clearml-agent) (5.8.0)
Collecting virtualenv<21,>=16
  Using cached virtualenv-20.10.0-py2.py3-none-any.whl (5.6 MB)
Collecting pyjwt<2.1.0,>=1.6.4
  Using cached PyJWT-2.0.1-py3-none-any.whl (15 kB)
Processing /root/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl
Requirement already satisfied, skipping upgrade: urllib3<1.27.0,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from clearml-agent) (1.26.6)
Collecting jsonschema<3.3.0,>=2.6.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting attrs<20.4.0,>=18.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Processing /root/.cache/pip/wheels/35/f3/15/01aa6571f0a72ee6ae7b827c1491c37a1f72d686fd22b43b0e/typing-3.7.4.3-py3-none-any.whl
Requirement already satisfied, skipping upgrade: requests<2.26.0,>=2.20.0 in /opt/conda/lib/python3.7/site-packages (from clearml-agent) (2.25.1)
Collecting orderedmultidict>=1.0.1
  Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
Collecting filelock<4,>=3.2
  Using cached filelock-3.4.0-py3-none-any.whl (9.8 kB)
Collecting backports.entry-points-selectable>=1.0.4
  Using cached backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl (6.2 kB)
Collecting importlib-metadata>=0.12; python_version < "3.8"
  Using cached importlib_metadata-4.8.2-py3-none-any.whl (17 kB)
Collecting platformdirs<3,>=2
  Using cached platformdirs-2.4.0-py3-none-any.whl (14 kB)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.3-py2.py3-none-any.whl (496 kB)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from jsonschema<3.3.0,>=2.6.0->clearml-agent) (58.0.4)
Collecting pyrsistent>=0.14.0
  Using cached pyrsistent-0.18.0-cp37-cp37m-manylinux1_x86_64.whl (119 kB)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<2.26.0,>=2.20.0->clearml-agent) (2.10)
Requirement already satisfied, skipping upgrade: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<2.26.0,>=2.20.0->clearml-agent) (4.0.0)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<2.26.0,>=2.20.0->clearml-agent) (2021.10.8)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from importlib-metadata>=0.12; python_version < "3.8"->virtualenv<21,>=16->clearml-agent) (3.10.0.2)
Collecting zipp>=0.5
  Using cached zipp-3.6.0-py3-none-any.whl (5.3 kB)
Installing collected packages: pyparsing, pyhocon, six, pathlib2, orderedmultidict, furl, python-dateutil, filelock, zipp, importlib-metadata, backports.entry-points-selectable, platformdirs, distlib, virtualenv, pyjwt, future, attrs, pyrsistent, jsonschema, typing, clearml-agent

1637513654104 MS-7A93:gpu0,1 DEBUG   Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
  Attempting uninstall: filelock
    Found existing installation: filelock 3.0.12
    Uninstalling filelock-3.0.12:
      Successfully uninstalled filelock-3.0.12
Successfully installed attrs-20.3.0 backports.entry-points-selectable-1.1.1 clearml-agent-1.1.1 distlib-0.3.3 filelock-3.4.0 furl-2.1.3 future-0.18.2 importlib-metadata-4.8.2 jsonschema-3.2.0 orderedmultidict-1.0.1 pathlib2-2.3.6 platformdirs-2.4.0 pyhocon-0.3.58 pyjwt-2.0.1 pyparsing-2.4.7 pyrsistent-0.18.0 python-dateutil-2.8.2 six-1.15.0 typing-3.7.4.3 virtualenv-20.10.0 zipp-3.6.0
[33mWARNING: You are using pip version 20.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/opt/conda/bin/python3.7 -m pip install --upgrade pip' command.[0m

jkhenning commented 2 years ago

Hi @afterimagex,

The only suspicious thing I see is:

1637513603608 MS-7A93:gpu0,1 DEBUG WARNING: Error loading config file: /home/pc/.docker/config.json: open /home/pc/.docker/config.json: permission denied

It might suggest this dir/file was created in previous runs with different privileges, do you have access to this file?

afterimagex commented 2 years ago

Use the root user to start the agent to solve the problem

starsky commented 1 year ago

Use the root user to start the agent to solve the problem

@afterimagex Please :) It is not a solution.

allegroai / clearml

I have a magical problem #495