search

openml / automlbenchmark

OpenML AutoML Benchmarking Framework
https://openml.github.io/automlbenchmark
MIT License
391 stars 130 forks source link

Logs are not saved to S3 in aws mode when job fails during cloud-init #505

Open Innixma opened 1 year ago

Innixma commented 1 year ago

Below is the end of a log that was never saved to S3. I was able to get it by parsing the benchmark run main instance logs on local disk. It appears that instances will not save their logs to S3 if they fail during cloud-init section. This makes debugging issues very hard.

I'd like to request that logs be saved to S3 even if the instance fails during cloud-init.

Beyond that, I'd also like to understand if the error being shown in the log can be fixed in some way. I believe it is due to spinning up many jobs at once, and by chance a few of them trigger this kind of error. Perhaps an automatic retry of some sort? Or should I switch to using Docker?

[i-03047957e2facc42d]>[   46.594400] cloud-init[1807]: Reading package lists...
[i-03047957e2facc42d]>[   46.739390] cloud-init[1807]: Building dependency tree...
[i-03047957e2facc42d]>[   46.740317] cloud-init[1807]: Reading state information...
[i-03047957e2facc42d]>[   46.867845] cloud-init[1807]: The following additional packages will be installed:
[i-03047957e2facc42d]>[   46.869107] cloud-init[1807]:   apt apt-utils binutils binutils-common binutils-x86-64-linux-gnu
[i-03047957e2facc42d]>[   46.870131] cloud-init[1807]:   build-essential cpp cpp-7 dh-python dpkg-dev fakeroot g++ g++-7 gcc gcc-7
[i-03047957e2facc42d]>[   46.870708] cloud-init[1807]:   gcc-7-base gcc-8-base libalgorithm-diff-perl libalgorithm-diff-xs-perl
[i-03047957e2facc42d]>[   46.870861] cloud-init[1807]:   libalgorithm-merge-perl libapt-inst2.0 libapt-pkg5.0 libasan4 libatomic1
[i-03047957e2facc42d]>[   46.871035] cloud-init[1807]:   libbinutils libc-dev-bin libc6 libc6-dev libcc1-0 libcilkrts5 libdpkg-perl
[i-03047957e2facc42d]>[   46.871228] cloud-init[1807]:   libexpat1 libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-7-dev
[i-03047957e2facc42d]>[   46.871414] cloud-init[1807]:   libgcc1 libgomp1 libisl19 libitm1 liblsan0 libmpc3 libmpx2 libpython3-dev
[i-03047957e2facc42d]>[   46.871599] cloud-init[1807]:   libpython3-stdlib libpython3.6 libpython3.6-dev libpython3.6-minimal
[i-03047957e2facc42d]>[   46.871776] cloud-init[1807]:   libpython3.6-stdlib libpython3.7 libpython3.7-dev libpython3.7-minimal
[i-03047957e2facc42d]>[   46.871968] cloud-init[1807]:   libpython3.7-stdlib libquadmath0 libstdc++-7-dev libstdc++6 libtsan0
[i-03047957e2facc42d]>[   46.872336] cloud-init[1807]:   libubsan0 linux-libc-dev make manpages-dev python-pip-whl python3
[i-03047957e2facc42d]>[   46.872505] cloud-init[1807]:   python3-crypto python3-dev python3-distutils python3-keyring
[i-03047957e2facc42d]>[   46.872683] cloud-init[1807]:   python3-keyrings.alt python3-lib2to3 python3-minimal python3-secretstorage
[i-03047957e2facc42d]>[   46.872869] cloud-init[1807]:   python3-setuptools python3-wheel python3-xdg python3.6 python3.6-dev
[i-03047957e2facc42d]>[   46.873063] cloud-init[1807]:   python3.6-minimal python3.7-minimal
[i-03047957e2facc42d]>[   46.873253] cloud-init[1807]: Suggested packages:
[i-03047957e2facc42d]>[   46.873432] cloud-init[1807]:   apt-doc aptitude | synaptic | wajig binutils-doc cpp-doc gcc-7-locales
[i-03047957e2facc42d]>[   46.873614] cloud-init[1807]:   debian-keyring g++-multilib g++-7-multilib gcc-7-doc libstdc++6-7-dbg
[i-03047957e2facc42d]>[   46.873802] cloud-init[1807]:   gcc-multilib autoconf automake libtool flex bison gdb gcc-doc gcc-7-multilib
[i-03047957e2facc42d]>[   46.873990] cloud-init[1807]:   libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan4-dbg
[i-03047957e2facc42d]>[   46.874186] cloud-init[1807]:   liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libmpx2-dbg
[i-03047957e2facc42d]>[   46.874360] cloud-init[1807]:   libquadmath0-dbg glibc-doc bzr libstdc++-7-doc make-doc python3-doc
[i-03047957e2facc42d]>[   46.874543] cloud-init[1807]:   python3-tk python3-venv python3-apt-dbg python-apt-doc python-crypto-doc
[i-03047957e2facc42d]>[   46.874713] cloud-init[1807]:   gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0
[i-03047957e2facc42d]>[   46.874904] cloud-init[1807]:   python-secretstorage-doc python-setuptools-doc python3.6-venv python3.6-doc
[i-03047957e2facc42d]>[   46.875080] cloud-init[1807]:   binfmt-support
[i-03047957e2facc42d]>[   46.925896] cloud-init[1807]: The following NEW packages will be installed:
[i-03047957e2facc42d]>[   46.926995] cloud-init[1807]:   binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-7
[i-03047957e2facc42d]>[   46.928416] cloud-init[1807]:   dh-python dpkg-dev fakeroot g++ g++-7 gcc gcc-7 gcc-7-base
[i-03047957e2facc42d]>[   46.928505] cloud-init[1807]:   libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
[i-03047957e2facc42d]>[   46.928569] cloud-init[1807]:   libasan4 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcilkrts5
[i-03047957e2facc42d]>[   46.928624] cloud-init[1807]:   libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-7-dev
[i-03047957e2facc42d]>[   46.928677] cloud-init[1807]:   libgomp1 libisl19 libitm1 liblsan0 libmpc3 libmpx2 libpython3-dev
[i-03047957e2facc42d]>[   46.928726] cloud-init[1807]:   libpython3.6-dev libpython3.7 libpython3.7-dev libpython3.7-minimal
[i-03047957e2facc42d]>[   46.928775] cloud-init[1807]:   libpython3.7-stdlib libquadmath0 libstdc++-7-dev libtsan0 libubsan0
[i-03047957e2facc42d]>[   46.928819] cloud-init[1807]:   linux-libc-dev make manpages-dev python-pip-whl python3-crypto python3-dev
[i-03047957e2facc42d]>[   46.928862] cloud-init[1807]:   python3-distutils python3-keyring python3-keyrings.alt python3-lib2to3
[i-03047957e2facc42d]>[   46.928903] cloud-init[1807]:   python3-pip python3-secretstorage python3-setuptools python3-wheel
[i-03047957e2facc42d]>[   46.928947] cloud-init[1807]:   python3-xdg python3.6-dev python3.7 python3.7-dev python3.7-minimal
[i-03047957e2facc42d]>[   46.928988] cloud-init[1807]:   python3.7-venv
[i-03047957e2facc42d]>[   46.929029] cloud-init[1807]: The following packages will be upgraded:
[i-03047957e2facc42d]>[   46.929067] cloud-init[1807]:   apt apt-utils gcc-8-base libapt-inst2.0 libapt-pkg5.0 libc6 libexpat1
[i-03047957e2facc42d]>[   46.929108] cloud-init[1807]:   libgcc1 libpython3-stdlib libpython3.6 libpython3.6-minimal
[i-03047957e2facc42d]>[   46.929148] cloud-init[1807]:   libpython3.6-stdlib libstdc++6 python3 python3-apt python3-minimal python3.6
[i-03047957e2facc42d]>[   46.929189] cloud-init[1807]:   python3.6-minimal
[i-03047957e2facc42d]>[   46.957462] cloud-init[1807]: 18 upgraded, 65 newly installed, 0 to remove and 274 not upgraded.
[i-03047957e2facc42d]>[   46.957950] cloud-init[1807]: Need to get 154 MB of archives.
[i-03047957e2facc42d]>[   46.959033] cloud-init[1807]: After this operation, 345 MB of additional disk space will be used.
[i-03047957e2facc42d]>[   46.959764] cloud-init[1807]: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 gcc-8-base amd64 8.4.0-1ubuntu1~18.04 [18.7 kB]
[i-03047957e2facc42d]>[   46.962967] cloud-init[1807]: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libgcc1 amd64 1:8.4.0-1ubuntu1~18.04 [40.6 kB]
[i-03047957e2facc42d]>[   46.967680] cloud-init[1807]: Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libstdc++6 amd64 8.4.0-1ubuntu1~18.04 [400 kB]
[i-03047957e2facc42d]>[   46.987808] cloud-init[1807]: Get:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libc6 amd64 2.27-3ubuntu1.6 [2831 kB]
[i-03047957e2facc42d]>[   47.045765] cloud-init[1807]: Get:5 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.14 [809 kB]
[i-03047957e2facc42d]>[   47.059805] cloud-init[1807]: Get:6 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-inst2.0 amd64 1.6.14 [57.3 kB]
[i-03047957e2facc42d]>[   47.066355] cloud-init[1807]: Get:7 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.14 [1207 kB]
[i-03047957e2facc42d]>[   47.087696] cloud-init[1807]: Get:8 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.14 [209 kB]
[i-03047957e2facc42d]>[   47.095773] cloud-init[1807]: Get:9 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3.6 amd64 3.6.9-1~18.04ubuntu1.8 [203 kB]
[i-03047957e2facc42d]>[   47.101194] cloud-init[1807]: Get:10 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libexpat1 amd64 2.2.5-3ubuntu0.7 [82.6 kB]
[i-03047957e2facc42d]>[   47.107648] cloud-init[1807]: Get:11 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6 amd64 3.6.9-1~18.04ubuntu1.8 [1414 kB]
[i-03047957e2facc42d]>[   47.117657] cloud-init[1807]: Get:12 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 libpython3.7-minimal amd64 3.7.15-1+bionic1 [589 kB]
[i-03047957e2facc42d]>[   47.129917] cloud-init[1807]: Get:13 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6-stdlib amd64 3.6.9-1~18.04ubuntu1.8 [1711 kB]
[i-03047957e2facc42d]>[   47.155257] cloud-init[1807]: Get:14 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3.6-minimal amd64 3.6.9-1~18.04ubuntu1.8 [1611 kB]
[i-03047957e2facc42d]>[   47.179204] cloud-init[1807]: Get:15 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6-minimal amd64 3.6.9-1~18.04ubuntu1.8 [533 kB]
[i-03047957e2facc42d]>[   47.191610] cloud-init[1807]: Get:16 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-minimal amd64 3.6.7-1~18.04 [23.7 kB]
[i-03047957e2facc42d]>[   47.195934] cloud-init[1807]: Get:17 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3 amd64 3.6.7-1~18.04 [47.2 kB]
[i-03047957e2facc42d]>[   47.200519] cloud-init[1807]: Get:18 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3-stdlib amd64 3.6.7-1~18.04 [7240 B]
[i-03047957e2facc42d]>[   47.204722] cloud-init[1807]: Get:19 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-apt amd64 1.6.5ubuntu0.7 [149 kB]
[i-03047957e2facc42d]>[   47.210768] cloud-init[1807]: Get:20 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 binutils-common amd64 2.30-21ubuntu1~18.04.7 [197 kB]
[i-03047957e2facc42d]>[   47.217531] cloud-init[1807]: Get:21 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libbinutils amd64 2.30-21ubuntu1~18.04.7 [489 kB]
[i-03047957e2facc42d]>[   47.228248] cloud-init[1807]: Get:22 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 binutils-x86-64-linux-gnu amd64 2.30-21ubuntu1~18.04.7 [1839 kB]
[i-03047957e2facc42d]>[   47.254529] cloud-init[1807]: Get:23 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 binutils amd64 2.30-21ubuntu1~18.04.7 [3388 B]
[i-03047957e2facc42d]>[   47.257237] cloud-init[1807]: Get:24 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libc-dev-bin amd64 2.27-3ubuntu1.6 [71.9 kB]
[i-03047957e2facc42d]>[   47.263745] cloud-init[1807]: Get:25 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-libc-dev amd64 4.15.0-197.208 [997 kB]
[i-03047957e2facc42d]>[   47.279133] cloud-init[1807]: Get:26 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libc6-dev amd64 2.27-3ubuntu1.6 [2587 kB]
[i-03047957e2facc42d]>[   47.343330] cloud-init[1807]: Get:27 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 gcc-7-base amd64 7.5.0-3ubuntu1~18.04 [18.3 kB]
[i-03047957e2facc42d]>[   47.347790] cloud-init[1807]: Get:28 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libisl19 amd64 0.19-1 [551 kB]
[i-03047957e2facc42d]>[   47.359057] cloud-init[1807]: Get:29 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libmpc3 amd64 1.1.0-1 [40.8 kB]
[i-03047957e2facc42d]>[   47.365125] cloud-init[1807]: Get:30 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 cpp-7 amd64 7.5.0-3ubuntu1~18.04 [8591 kB]
[i-03047957e2facc42d]>[   47.573840] cloud-init[1807]: Get:31 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 cpp amd64 4:7.4.0-1ubuntu2.3 [27.7 kB]
[i-03047957e2facc42d]>[   47.577546] cloud-init[1807]: Get:32 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcc1-0 amd64 8.4.0-1ubuntu1~18.04 [39.4 kB]
[i-03047957e2facc42d]>[   47.583118] cloud-init[1807]: Get:33 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libgomp1 amd64 8.4.0-1ubuntu1~18.04 [76.5 kB]
[i-03047957e2facc42d]>[   47.589100] cloud-init[1807]: Get:34 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libitm1 amd64 8.4.0-1ubuntu1~18.04 [27.9 kB]
[i-03047957e2facc42d]>[   47.593690] cloud-init[1807]: Get:35 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libatomic1 amd64 8.4.0-1ubuntu1~18.04 [9192 B]
[i-03047957e2facc42d]>[   47.598000] cloud-init[1807]: Get:36 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libasan4 amd64 7.5.0-3ubuntu1~18.04 [358 kB]
[i-03047957e2facc42d]>[   47.604880] cloud-init[1807]: Get:37 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 liblsan0 amd64 8.4.0-1ubuntu1~18.04 [133 kB]
[i-03047957e2facc42d]>[   47.610085] cloud-init[1807]: Get:38 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libtsan0 amd64 8.4.0-1ubuntu1~18.04 [288 kB]
[i-03047957e2facc42d]>[   47.617489] cloud-init[1807]: Get:39 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libubsan0 amd64 7.5.0-3ubuntu1~18.04 [126 kB]
[i-03047957e2facc42d]>[   47.623476] cloud-init[1807]: Get:40 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcilkrts5 amd64 7.5.0-3ubuntu1~18.04 [42.5 kB]
[i-03047957e2facc42d]>[   47.628193] cloud-init[1807]: Get:41 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmpx2 amd64 8.4.0-1ubuntu1~18.04 [11.6 kB]
[i-03047957e2facc42d]>[   47.632838] cloud-init[1807]: Get:42 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libquadmath0 amd64 8.4.0-1ubuntu1~18.04 [134 kB]
[i-03047957e2facc42d]>[   47.638298] cloud-init[1807]: Get:43 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libgcc-7-dev amd64 7.5.0-3ubuntu1~18.04 [2378 kB]
[i-03047957e2facc42d]>[   47.787896] cloud-init[1807]: Get:44 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 gcc-7 amd64 7.5.0-3ubuntu1~18.04 [9381 kB]
[i-03047957e2facc42d]>[   47.991881] cloud-init[1807]: Get:45 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 gcc amd64 4:7.4.0-1ubuntu2.3 [5184 B]
[i-03047957e2facc42d]>[   47.994799] cloud-init[1807]: Get:46 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libstdc++-7-dev amd64 7.5.0-3ubuntu1~18.04 [1471 kB]
[i-03047957e2facc42d]>[   48.016317] cloud-init[1807]: Get:47 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 g++-7 amd64 7.5.0-3ubuntu1~18.04 [9697 kB]
[i-03047957e2facc42d]>[   48.257659] cloud-init[1807]: Get:48 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 g++ amd64 4:7.4.0-1ubuntu2.3 [1568 B]
[i-03047957e2facc42d]>[   48.261362] cloud-init[1807]: Get:49 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 make amd64 4.1-9.1ubuntu1 [154 kB]
[i-03047957e2facc42d]>[   48.266006] cloud-init[1807]: Get:50 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libdpkg-perl all 1.19.0.5ubuntu2.4 [212 kB]
[i-03047957e2facc42d]>[   48.272858] cloud-init[1807]: Get:51 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 dpkg-dev all 1.19.0.5ubuntu2.4 [607 kB]
[i-03047957e2facc42d]>[   48.284358] cloud-init[1807]: Get:52 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 build-essential amd64 12.4ubuntu1 [4758 B]
[i-03047957e2facc42d]>[   48.289129] cloud-init[1807]: Get:53 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lib2to3 all 3.6.9-1~18.04 [77.4 kB]
[i-03047957e2facc42d]>[   48.294783] cloud-init[1807]: Get:54 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-distutils all 3.6.9-1~18.04 [144 kB]
[i-03047957e2facc42d]>[   48.299455] cloud-init[1807]: Get:55 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 dh-python all 3.20180325ubuntu2 [89.2 kB]
[i-03047957e2facc42d]>[   48.304606] cloud-init[1807]: Get:56 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libfakeroot amd64 1.22-2ubuntu1 [25.9 kB]
[i-03047957e2facc42d]>[   48.309568] cloud-init[1807]: Get:57 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 fakeroot amd64 1.22-2ubuntu1 [62.3 kB]
[i-03047957e2facc42d]>[   48.313800] cloud-init[1807]: Get:58 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libalgorithm-diff-perl all 1.19.03-1 [47.6 kB]
[i-03047957e2facc42d]>[   48.319081] cloud-init[1807]: Get:59 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libalgorithm-diff-xs-perl amd64 0.04-5 [11.1 kB]
[i-03047957e2facc42d]>[   48.322059] cloud-init[1807]: Get:60 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libalgorithm-merge-perl all 0.08-3 [12.0 kB]
[i-03047957e2facc42d]>[   48.327328] cloud-init[1807]: Get:61 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libexpat1-dev amd64 2.2.5-3ubuntu0.7 [124 kB]
[i-03047957e2facc42d]>[   48.334559] cloud-init[1807]: Get:62 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libfile-fcntllock-perl amd64 0.22-3build2 [33.2 kB]
[i-03047957e2facc42d]>[   48.340623] cloud-init[1807]: Get:63 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6-dev amd64 3.6.9-1~18.04ubuntu1.8 [44.9 MB]
[i-03047957e2facc42d]>[   49.866039] cloud-init[1807]: Get:64 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3-dev amd64 3.6.7-1~18.04 [7328 B]
[i-03047957e2facc42d]>[   49.870251] cloud-init[1807]: Get:65 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 manpages-dev all 4.15-1 [2217 kB]
[i-03047957e2facc42d]>[   49.941411] cloud-init[1807]: Get:66 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python-pip-whl all 9.0.1-2.3~ubuntu1.18.04.5 [1653 kB]
[i-03047957e2facc42d]>[   49.964579] cloud-init[1807]: Get:67 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 python3-crypto amd64 2.6.1-8ubuntu2 [244 kB]
[i-03047957e2facc42d]>[   49.972374] cloud-init[1807]: Get:68 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3.6-dev amd64 3.6.9-1~18.04ubuntu1.8 [511 kB]
[i-03047957e2facc42d]>[   49.982719] cloud-init[1807]: Get:69 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-dev amd64 3.6.7-1~18.04 [1288 B]
[i-03047957e2facc42d]>[   49.987143] cloud-init[1807]: Get:70 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 python3-secretstorage all 2.3.1-2 [12.1 kB]
[i-03047957e2facc42d]>[   49.991780] cloud-init[1807]: Get:71 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 python3-keyring all 10.6.0-1 [26.7 kB]
[i-03047957e2facc42d]>[   49.996685] cloud-init[1807]: Get:72 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 python3-keyrings.alt all 3.0-1 [16.6 kB]
[i-03047957e2facc42d]>[   49.999983] cloud-init[1807]: Get:73 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-pip all 9.0.1-2.3~ubuntu1.18.04.5 [114 kB]
[i-03047957e2facc42d]>[   50.006490] cloud-init[1807]: Get:74 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB]
[i-03047957e2facc42d]>[   50.014423] cloud-init[1807]: Get:75 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/universe amd64 python3-wheel all 0.30.0-0.2 [36.5 kB]
[i-03047957e2facc42d]>[   50.019141] cloud-init[1807]: Get:76 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-xdg all 0.25-4ubuntu1.1 [31.3 kB]
[i-03047957e2facc42d]>[   77.790118] cloud-init[1807]: Err:77 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 python3.7-minimal amd64 3.7.15-1+bionic1
[i-03047957e2facc42d]>[   77.791721] cloud-init[1807]:   Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) Could not connect to ppa.launchpad.net:80 (185.125.190.52), connection timed out [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.792739] cloud-init[1807]: Err:78 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 libpython3.7-stdlib amd64 3.7.15-1+bionic1
[i-03047957e2facc42d]>[   77.792893] cloud-init[1807]:   Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.793087] cloud-init[1807]: Err:79 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 libpython3.7 amd64 3.7.15-1+bionic1
[i-03047957e2facc42d]>[   77.793252] cloud-init[1807]:   Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.793429] cloud-init[1807]: Err:80 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 libpython3.7-dev amd64 3.7.15-1+bionic1
[i-03047957e2facc42d]>[   77.793664] cloud-init[1807]:   Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.793794] cloud-init[1807]: Err:81 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 python3.7 amd64 3.7.15-1+bionic1
[i-03047957e2facc42d]>[   77.793977] cloud-init[1807]:   Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.794157] cloud-init[1807]: Err:82 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 python3.7-dev amd64 3.7.15-1+bionic1
[i-03047957e2facc42d]>[   77.794345] cloud-init[1807]:   Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.794528] cloud-init[1807]: Err:83 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 python3.7-venv amd64 3.7.15-1+bionic1
[i-03047957e2facc42d]>[   77.794735] cloud-init[1807]:   Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.795969] cloud-init[1807]: Fetched 103 MB in 31s (3344 kB/s)
[i-03047957e2facc42d]>[   77.796371] cloud-init[1807]: E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/pool/main/p/python3.7/python3.7-minimal_3.7.15-1+bionic1_amd64.deb  Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) Could not connect to ppa.launchpad.net:80 (185.125.190.52), connection timed out [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.796542] cloud-init[1807]: E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/pool/main/p/python3.7/libpython3.7-stdlib_3.7.15-1+bionic1_amd64.deb  Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.796715] cloud-init[1807]: E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/pool/main/p/python3.7/libpython3.7_3.7.15-1+bionic1_amd64.deb  Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.796904] cloud-init[1807]: E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/pool/main/p/python3.7/libpython3.7-dev_3.7.15-1+bionic1_amd64.deb  Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.797081] cloud-init[1807]: E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/pool/main/p/python3.7/python3.7_3.7.15-1+bionic1_amd64.deb  Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.797262] cloud-init[1807]: E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/pool/main/p/python3.7/python3.7-dev_3.7.15-1+bionic1_amd64.deb  Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.797446] cloud-init[1807]: E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/pool/main/p/python3.7/python3.7-venv_3.7.15-1+bionic1_amd64.deb  Cannot initiate the connection to ppa.launchpad.net:80 (2620:2d:4000:1::3e). - connect (101: Network is unreachable) [IP: 185.125.190.52 80]
[i-03047957e2facc42d]>[   77.797647] cloud-init[1807]: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
[i-03047957e2facc42d]>[   77.841562] cloud-init[1807]: Cloning into '.'...
[i-03047957e2facc42d]>[   78.311523] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 16: /var/lib/cloud/instance/scripts/runcmd: python3.7: not found
[i-03047957e2facc42d]>[   78.311671] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 17: /var/lib/cloud/instance/scripts/runcmd: python3.7: not found
[i-03047957e2facc42d]>[   78.311903] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 21: /var/lib/cloud/instance/scripts/runcmd: /repo/venv/bin/python3: not found
[i-03047957e2facc42d]>[   78.314570] cloud-init[1807]: xargs: /repo/venv/bin/python3: No such file or directory
[i-03047957e2facc42d]>[   78.314785] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 23: /var/lib/cloud/instance/scripts/runcmd: aws: not found
[i-03047957e2facc42d]>[   78.314835] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 24: /var/lib/cloud/instance/scripts/runcmd: aws: not found
[i-03047957e2facc42d]>[   78.315017] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 25: /var/lib/cloud/instance/scripts/runcmd: /repo/venv/bin/python3: not found
[i-03047957e2facc42d]>[   78.315198] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 26: /var/lib/cloud/instance/scripts/runcmd: /repo/venv/bin/python3: not found
[i-03047957e2facc42d]>[   78.315379] cloud-init[1807]: /var/lib/cloud/instance/scripts/runcmd: 27: /var/lib/cloud/instance/scripts/runcmd: aws: not found
[i-03047957e2facc42d]>[   78.315831] cloud-init[1807]: Cloud-init v. 18.3-9-g2e62cb8a-0ubuntu1~18.04.2 running 'modules:final' at Tue, 15 Nov 2022 16:27:52 +0000. Up 25.08 seconds.
[i-03047957e2facc42d]>[   78.315959] cloud-init[1807]: 2022-11-15 16:28:45,428 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [127]
[i-03047957e2facc42d]>[   78.318562] cloud-init[1807]: 2022-11-15 16:28:45,431 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
[i-03047957e2facc42d]>[   78.318789] cloud-init[1807]: 2022-11-15 16:28:45,431 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
[i-03047957e2facc42d]>[   78.348494] cloud-init[1807]: ci-info: no authorized ssh keys fingerprints found for user ubuntu.
[i-03047957e2facc42d]>[   78.348735] cloud-init[1807]: AutoML benchmark aws.ag.1h8c.kddcup09-upselling.3.autogluon_mq completed after 78.34 s
[i-03047957e2facc42d]>[FAILED] Failed to start Execute cloud user/final scripts.
[i-03047957e2facc42d]>[  OK  ] Reached target Cloud-init target.