search

dask / dask-cloudprovider

Cloud provider cluster managers for Dask. Supports AWS, Google Cloud Azure and more...
https://cloudprovider.dask.org
BSD 3-Clause "New" or "Revised" License
134 stars 110 forks source link

EC2Cluster stays in hang state #345

Closed arbazk007 closed 2 years ago

arbazk007 commented 2 years ago

I am able to create EC2 instance (running state with failed instance) using below command but the cluster stays in hang state:

Environment: Dask version: 2022.02.1 Dask Cloud Provider version: 2022.4.0 Python version: 3.9 Operating System: Ubuntu

Ports 8787 and 8786 are open in security groups.

from dask_cloudprovider.aws import EC2Cluster

instance_profile_dict = {
    "Arn": <<IAM_ROLE>>
}
cluster = EC2Cluster(ami = 'ami-0892d3c7ee96c0bf7',
                     key_name=<<Key_name>>,
                     docker_image="daskdev/dask:latest",
                     vpc = <<test_vpc>>,
                     region = 'us-west-2',
                     iam_instance_profile=instance_profile_dict,
                     security_groups=<<security_groups>>,
                     instance_type = 't3.2xlarge',
                     subnet_id = <<subnet>>,
                     security=False,
                    debug=True)
Logs ``` f7 window] [ 2.063330] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.069017] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.074192] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.079461] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.084006] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.088842] pci 0000:00:03.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] [ 2.096468] PCI: CLS 0 bytes, default 64 [ 2.100086] Trying to unpack rootfs image as initramfs... [ 2.107460] Freeing initrd memory: 4532K [ 2.111177] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.115853] software IO TLB: mapped [mem 0x00000000bbfe9000-0x00000000bffe9000] (64MB) [ 2.123238] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x240937b9988, max_idle_ns: 440795218083 ns [ 2.131182] clocksource: Switched to clocksource tsc [ 2.135266] check: Scanning for low memory corruption every 60 seconds [ 2.140628] Initialise system trusted keyrings [ 2.144207] Key type blacklist registered [ 2.147881] workingset: timestamp_bits=36 max_order=23 bucket_order=0 [ 2.153607] zbud: loaded [ 2.156677] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 2.161204] fuse: init (API version 7.33) [ 2.165380] integrity: Platform Keyring initialized [ 2.176857] Key type asymmetric registered [ 2.180780] Asymmetric key parser 'x509' registered [ 2.184699] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243) [ 2.191541] io scheduler mq-deadline registered [ 2.196038] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.200817] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.207589] ACPI: Power Button [PWRF] [ 2.211226] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1 [ 2.217895] ACPI: Sleep Button [SLPF] [ 2.222113] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.254070] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.261298] Linux agpgart interface v0.103 [ 2.395717] loop: module loaded [ 2.399358] nvme nvme0: pci function 0000:00:04.0 [ 2.403293] PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 2.407198] libphy: Fixed MDIO Bus: probed [ 2.410631] tun: Universal TUN/TAP device driver, 1.6 [ 2.414462] PPP generic driver version 2.4.2 [ 2.419781] VFIO - User Level meta-driver version: 0.3 [ 2.420868] nvme nvme0: 2/0/0 default/read/poll queues [ 2.424228] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.432230] ehci-pci: EHCI PCI platform driver [ 2.432234] nvme0n1: p1 [ 2.435690] ehci-platform: EHCI generic platform driver [ 2.443062] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.447167] ohci-pci: OHCI PCI platform driver [ 2.450607] ohci-platform: OHCI generic platform driver [ 2.454339] uhci_hcd: USB Universal Host Controller Interface driver [ 2.458566] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.465100] i8042: Warning: Keylock active [ 2.469265] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.473462] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.477770] mousedev: PS/2 mouse device common for all mice [ 2.482468] rtc_cmos 00:00: RTC can wake from S4 [ 2.487576] rtc_cmos 00:00: registered as rtc0 [ 2.493461] rtc_cmos 00:00: setting system clock to 2022-05-04T07:24:55 UTC (1651649095) [ 2.500693] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 2.505581] i2c /dev entries driver [ 2.509060] device-mapper: uevent: version 1.0.3 [ 2.513012] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com [ 2.520293] platform eisa.0: Probing EISA bus 0 [ 2.524146] platform eisa.0: EISA: Cannot allocate resource for mainboard [ 2.528884] platform eisa.0: Cannot allocate resource for EISA slot 1 [ 2.533478] platform eisa.0: Cannot allocate resource for EISA slot 2 [ 2.538262] platform eisa.0: Cannot allocate resource for EISA slot 3 [ 2.542898] platform eisa.0: Cannot allocate resource for EISA slot 4 [ 2.548113] platform eisa.0: Cannot allocate resource for EISA slot 5 [ 2.552600] platform eisa.0: Cannot allocate resource for EISA slot 6 [ 2.557688] platform eisa.0: Cannot allocate resource for EISA slot 7 [ 2.562337] platform eisa.0: Cannot allocate resource for EISA slot 8 [ 2.567065] platform eisa.0: EISA: Detected 0 cards [ 2.570940] intel_pstate: CPU model not supported [ 2.575537] ledtrig-cpu: registered to indicate activity on CPUs [ 2.579754] drop_monitor: Initializing network drop monitor service [ 2.584086] NET: Registered protocol family 10 [ 2.588717] Segment Routing with IPv6 [ 2.592572] NET: Registered protocol family 17 [ 2.596740] Key type dns_resolver registered [ 2.601408] No MBM correction factor available [ 2.605291] IPI shorthand broadcast: enabled [ 2.609168] sched_clock: Marking stable (1795726210, 813420534)->(2756437377, -147290633) [ 2.616618] registered taskstats version 1 [ 2.620234] Loading compiled-in X.509 certificates [ 2.625769] Loaded X.509 cert 'Build time autogenerated kernel key: ecb0788dc66cd3f3e8014485463fe429aa2b1eba' [ 2.639339] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969' [ 2.651319] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19' [ 2.659552] blacklist: Loading compiled-in revocation X.509 certificates [ 2.666509] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing: 61482aa2830d0ab2ad5af10b7250da9033ddcef0' [ 2.679308] zswap: loaded using pool lzo/zbud [ 2.683593] Key type ._fscrypt registered [ 2.687386] Key type .fscrypt registered [ 2.691110] Key type fscrypt-provisioning registered [ 2.696260] Key type encrypted registered [ 2.700147] AppArmor: AppArmor sha1 policy hashing enabled [ 2.704363] ima: No TPM chip found, activating TPM-bypass! [ 2.709635] ima: Allocated hash algorithm: sha1 [ 2.713924] ima: No architecture policies found [ 2.718706] evm: Initialising EVM extended attributes: [ 2.723132] evm: security.selinux [ 2.726494] evm: security.SMACK64 [ 2.729956] evm: security.SMACK64EXEC [ 2.734973] evm: security.SMACK64TRANSMUTE [ 2.738835] evm: security.SMACK64MMAP [ 2.742264] evm: security.apparmor [ 2.745610] evm: security.ima [ 2.748883] evm: security.capability [ 2.752242] evm: HMAC attrs: 0x1 [ 2.755840] PM: Magic number: 6:408:419 [ 2.760519] RAS: Correctable Errors collector initialized. [ 3.004423] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 [ 3.013128] md: Waiting for all devices to be available before autodetect [ 3.018209] md: If you don't use raid, use raid=noautodetect [ 3.022511] md: Autodetecting RAID arrays. [ 3.026006] md: autorun ... [ 3.029242] md: ... autorun DONE. [ 3.041599] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. [ 3.048699] VFS: Mounted root (ext4 filesystem) readonly on device 259:1. [ 3.053231] devtmpfs: mounted [ 3.057123] Freeing unused decrypted memory: 2036K [ 3.061318] Freeing unused kernel image (initmem) memory: 2688K [ 3.067764] Write protecting the kernel read-only data: 30720k [ 3.072418] Freeing unused kernel image (text/rodata gap) memory: 2036K [ 3.077263] Freeing unused kernel image (rodata/data gap) memory: 1916K [ 3.158167] x86/mm: Checked W+X mappings: passed, no W+X pages found. [ 3.162430] x86/mm: Checking user space page tables [ 3.240417] x86/mm: Checked W+X mappings: passed, no W+X pages found. [ 3.244647] Run /sbin/init as init process [ 3.856100] systemd[1]: Inserted module 'autofs4' [ 3.906327] systemd[1]: systemd 245.4-4ubuntu3.13 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid) [ 3.921600] systemd[1]: Detected virtualization kvm. [ 3.925253] systemd[1]: Detected architecture x86-64. Welcome to Ubuntu 20.04.3 LTS! [ 3.950734] systemd[1]: Set hostname to . [ 3.975941] systemd[1]: Initializing machine ID from KVM UUID. [ 4.093440] systemd[1]: Installed transient /etc/machine-id file. [ 4.488900] systemd[1]: Created slice system-modprobe.slice. [ OK ] Created slice system-modprobe.slice. [ 4.494534] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 4.500490] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ 4.505796] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password R…uests to Wall Directory Watch. [ 4.514130] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. [ OK ] Set up automount Arbitrary…s File System Automount Point. [ 4.522712] systemd[1]: Reached target User and Group Name Lookups. [ OK ] Reached target User and Group Name Lookups. [ 4.528117] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.533070] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.537625] systemd[1]: Listening on Device-mapper event daemon FIFOs. [ OK ] Listening on Device-mapper event daemon FIFOs. [ 4.543462] systemd[1]: Listening on LVM2 poll daemon socket. [ OK ] Listening on LVM2 poll daemon socket. [ 4.549197] systemd[1]: Listening on multipathd control socket. [ OK ] Listening on multipathd control socket. [ 4.554759] systemd[1]: Listening on Syslog Socket. [ OK ] Listening on Syslog Socket. [ 4.559596] systemd[1]: Listening on fsck to fsckd communication Socket. [ OK ] Listening on fsck to fsckd communication Socket. [ 4.565514] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 4.571392] systemd[1]: Listening on Journal Audit Socket. [ OK ] Listening on Journal Audit Socket. [ 4.576901] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.582511] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.587605] systemd[1]: Listening on Network Service Netlink Socket. [ OK ] Listening on Network Service Netlink Socket. [ 4.593212] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.598534] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.604582] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ 4.610438] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ 4.617101] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ 4.622804] systemd[1]: Mounting Kernel Trace File System... Mounting Kernel Trace File System... [ 4.629720] systemd[1]: Starting Journal Service... Starting Journal Service... [ 4.635400] systemd[1]: Starting Set the console keyboard layout... Starting Set the console keyboard layout... [ 4.641835] systemd[1]: Starting Create list of static device nodes for the current kernel... Starting Create list of st…odes for the current kernel... [ 4.650734] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... Starting Monitoring of LVM…meventd or progress polling... [ 4.660696] systemd[1]: Starting Load Kernel Module drm... Starting Load Kernel Module drm... [ 4.665914] systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped. [ 4.673765] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped. [ 4.682105] systemd[1]: Starting File System Check on Root Device... Starting File System Check on Root Device... [ 4.688920] systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... [ 4.694656] systemd[1]: Starting udev Coldplug all Devices... Starting udev Coldplug all Devices... [ 4.700392] systemd[1]: Starting Uncomplicated firewall... Starting Uncomplicated firewall... [ 4.706878] systemd[1]: Mounted Huge Pages File System. [ OK ] Mounted Huge Pages File System. [ 4.712545] systemd[1]: Mounted POSIX Message Queue File System. [ OK ] Mounted POSIX Message Queue File System. [ 4.718279] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Kernel Trace File System. [ OK ] Finished Create list of st… nodes for the current kernel. [ OK ] Finished Load Kernel Module drm. [ OK ] Finished Load Kernel Modules. [ OK ] Finished Uncomplicated firewall. Mounting FUSE Control File System... Mounting Kernel Configuration File System... [ OK ] Started File System Check Daemon to report status. Starting Apply Kernel Variables... [ OK ] Finished File System Check on Root Device. [ OK ] Mounted FUSE Control File System. [ OK ] Mounted Kernel Configuration File System. Starting Remount Root and Kernel File Systems... [ OK ] Finished Set the console keyboard layout. [ OK ] Finished Remount Root and Kernel File Systems. Starting Flush Journal to Persistent Storage... Starting Load/Save Random Seed... Starting Create System Users... [ OK ] Finished Apply Kernel Variables. [ OK ] Finished Load/Save Random Seed. [ OK ] Finished udev Coldplug all Devices. Starting udev Wait for Complete Device Initialization... [ OK ] Finished Flush Journal to Persistent Storage. [ OK ] Finished Create System Users. Starting Create Static Device Nodes in /dev... [ OK ] Finished Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Dispatch Password …ts to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Found device /dev/ttyS0. [ 5.143071] ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy. [ OK ] Listening on Load/Save RF …itch Status /dev/rfkill Watch. [ OK ] Finished udev Wait for Complete Device Initialization. Starting Device-Mapper Multipath Device Controller... [ OK ] Finished Monitoring of LVM… dmeventd or progress polling. [ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Reached target Local File Systems (Pre). Mounting Mount unit for am…on-ssm-agent, revision 4046... Mounting Mount unit for core18, revision 2253... Mounting Mount unit for core20, revision 1242... Mounting Mount unit for lxd, revision 21835... Mounting Mount unit for snapd, revision 14066... [ OK ] Mounted Mount unit for amazon-ssm-agent, revision 4046. [ OK ] Mounted Mount unit for core20, revision 1242. [ OK ] Mounted Mount unit for lxd, revision 21835. [ OK ] Mounted Mount unit for core18, revision 2253. [ OK ] Mounted Mount unit for snapd, revision 14066. [ OK ] Reached target Local File Systems. Starting Load AppArmor profiles... Starting Set console font and keymap... Starting Create final runt…dir for shutdown pivot root... Starting Tell Plymouth To Write Out Runtime Data... Starting Commit a transient machine-id on disk... Starting Create Volatile Files and Directories... [ OK ] Finished Set console font and keymap. [ OK ] Finished Create final runt…e dir for shutdown pivot root. [ OK ] Finished Create Volatile Files and Directories. Starting Network Time Synchronization... Starting Update UTMP about System Boot/Shutdown... [ OK ] Finished Tell Plymouth To Write Out Runtime Data. [ OK ] Finished Commit a transient machine-id on disk. [ OK ] Finished Update UTMP about System Boot/Shutdown. [ OK ] Started Network Time Synchronization. [ OK ] Reached target System Time Set. [ OK ] Reached target System Time Synchronized. [ OK ] Finished Load AppArmor profiles. Starting Load AppArmor pro…managed internally by snapd... Starting Initial cloud-init job (pre-networking)... [ OK ] Finished Load AppArmor pro…s managed internally by snapd. [ 7.338567] cloud-init[427]: Cloud-init v. 21.4-0ubuntu1~20.04.1 running 'init-local' at Wed, 04 May 2022 07:25:00 +0000. Up 6.65 seconds. [ OK ] Finished Initial cloud-init job (pre-networking). [ OK ] Reached target Network (Pre). Starting Network Service... [ OK ] Started Network Service. Starting Wait for Network to be Configured... Starting Network Name Resolution... [ OK ] Started Network Name Resolution. [ OK ] Reached target Network. [ OK ] Reached target Host and Network Name Lookups. [ OK ] Finished Wait for Network to be Configured. Starting Initial cloud-ini… (metadata service crawler)... [ 10.522602] cloud-init[470]: Cloud-init v. 21.4-0ubuntu1~20.04.1 running 'init' at Wed, 04 May 2022 07:25:04 +0000. Up 10.24 seconds. [ 10.524522] cloud-init[470]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++ [ 10.526254] cloud-init[470]: ci-info: +--------+------+----------------------------+-----------------+--------+-------------------+ [ 10.528568] cloud-init[470]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 10.530479] cloud-init[470]: ci-info: +--------+------+----------------------------+-----------------+--------+-------------------+ [ 10.533016] cloud-init[470]: ci-info: | ens5 | True | 10.178.104.100 | 255.255.255.192 | global | 06:43:f9:01:0c:41 | [ 10.534921] cloud-init[470]: ci-info: | ens5 | True | fe80::443:f9ff:fe01:c41/64 | . | link | 06:43:f9:01:0c:41 | [ 10.536758] cloud-init[470]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 10.538690] cloud-init[470]: ci-info: | lo | True | ::1/128 | . | host | . | [ 10.540779] cloud-init[470]: ci-info: +--------+------+----------------------------+-----------------+--------+-------------------+ [ 10.542510] cloud-init[470]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++ [ 10.544076] cloud-init[470]: ci-info: +-------+---------------+---------------+-----------------+-----------+-------+ [ 10.545703] cloud-init[470]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 10.547278] cloud-init[470]: ci-info: +-------+---------------+---------------+-----------------+-----------+-------+ [ 10.549266] cloud-init[470]: ci-info: | 0 | 0.0.0.0 | 10.178.104.65 | 0.0.0.0 | ens5 | UG | [ 10.550851] cloud-init[470]: ci-info: | 1 | 10.178.104.64 | 0.0.0.0 | 255.255.255.192 | ens5 | U | [ 10.552433] cloud-init[470]: ci-info: | 2 | 10.178.104.65 | 0.0.0.0 | 255.255.255.255 | ens5 | UH | [ 10.553974] cloud-init[470]: ci-info: +-------+---------------+---------------+-----------------+-----------+-------+ [ 10.555527] cloud-init[470]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 10.556908] cloud-init[470]: ci-info: +-------+-------------+---------+-----------+-------+ [ 10.558398] cloud-init[470]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 10.559821] cloud-init[470]: ci-info: +-------+-------------+---------+-----------+-------+ [ 10.561235] cloud-init[470]: ci-info: | 1 | fe80::/64 | :: | ens5 | U | [ 10.562554] cloud-init[470]: ci-info: | 3 | local | :: | ens5 | U | [ 10.563763] cloud-init[470]: ci-info: | 4 | multicast | :: | ens5 | U | [ 10.565153] cloud-init[470]: ci-info: +-------+-------------+---------+-----------+-------+ [ 11.864388] cloud-init[470]: Generating public/private rsa key pair. [ 11.865549] cloud-init[470]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key [ 11.867674] cloud-init[470]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub [ 11.871135] cloud-init[470]: The key fingerprint is: [ 11.872119] cloud-init[470]: SHA256:kbPb1SFbJR2HP7taqiBe0a9jixqhdql3UWjNkFgz5L8 root@ip-10-178-104-100 [ 11.873573] cloud-init[470]: The key's randomart image is: [ 11.874589] cloud-init[470]: +---[RSA 3072]----+ [ 11.875530] cloud-init[470]: | +=. .++| [ 11.876309] cloud-init[470]: | ..+o .oo| [ 11.877071] cloud-init[470]: | +.= . o. | [ 11.877882] cloud-init[470]: | *o+ = o.| [ 11.878768] cloud-init[470]: | .S..oo . o| [ 11.879539] cloud-init[470]: | . o+..o . | [ 11.880273] cloud-init[470]: | o =.ooE . ..| [ 11.881229] cloud-init[470]: | . +.+.oo. o. | [ 11.882027] cloud-init[470]: | ..oo..++o. | [ 11.882899] cloud-init[470]: +----[SHA256]-----+ [ 11.883954] cloud-init[470]: Generating public/private dsa key pair. [ 11.885108] cloud-init[470]: Your identification has been saved in /etc/ssh/ssh_host_dsa_key [ 11.886533] cloud-init[470]: Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub [ 11.887930] cloud-init[470]: The key fingerprint is: [ 11.888795] cloud-init[470]: SHA256:hyuZ69TAQ06e2Vc+bkh1ExUAYTK2ZidRWcM1ofecErU root@ip-10-178-104-100 [ 11.890327] cloud-init[470]: The key's randomart image is: [ 11.891212] cloud-init[470]: +---[DSA 1024]----+ [ 11.891948] cloud-init[470]: | =.=*++B+| [ 11.892681] cloud-init[470]: | . *. .+.o| [ 11.893467] cloud-init[470]: | o = .oooE | [ 11.896156] cloud-init[470]: | * +o.o+ .ooo| [ 11.897180] cloud-init[470]: | O S + o. .o| [ 11.898116] cloud-init[470]: | * = o .. | [ 11.898928] cloud-init[470]: | = o . o | [ 11.899814] cloud-init[470]: | . o . | [ 11.900650] cloud-init[470]: | .o | [ 11.901406] cloud-init[470]: +----[SHA256]-----+ [ 11.902118] cloud-init[470]: Generating public/private ecdsa key pair. [ 11.903259] cloud-init[470]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key [ 11.904531] cloud-init[470]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub [ 11.905857] cloud-init[470]: The key fingerprint is: [ 11.906877] cloud-init[470]: SHA256:AOFIIYhQt/8Is7pA/PYrZlg0RzVWpJwaugZ9soa9MYY root@ip-10-178-104-100 [ 11.910202] cloud-init[470]: The key's randomart image is: [ 11.911144] cloud-init[470]: +---[ECDSA 256]---+ [ 11.912154] cloud-init[470]: |*.+.+..+oo | [ 11.913145] cloud-init[470]: |oo + +o + | [ 11.914094] cloud-init[470]: | . +..+ | [ 11.914951] cloud-init[470]: |. .o.oo. | [ 11.915757] cloud-init[470]: | +.+=o. S | [ 11.916715] cloud-init[470]: |. *.=+ o | [ 11.920961] cloud-init[470]: |.Eo%. . . | [ 11.924083] cloud-init[470]: | o=== | [ 11.925205] cloud-init[470]: | =o.o. | [ 11.929088] cloud-init[470]: +----[SHA256]-----+ [ 11.929971] cloud-init[470]: Generating public/private ed25519 key pair. [ 11.931907] cloud-init[470]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key [ 11.935763] cloud-init[470]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub [ 11.940837] cloud-init[470]: The key fingerprint is: [ 11.941625] cloud-init[470]: SHA256:PRvVvcuvKiatC6qlqtuY942Kakmz+Rot9A9K1nRwTjU root@ip-10-178-104-100 [ 11.946017] cloud-init[470]: The key's randomart image is: [ 11.949892] cloud-init[470]: +--[ED25519 256]--+ [ 11.951442] cloud-init[470]: | E | [ 11.956350] cloud-init[470]: | . . . . | [ OK ] Finished Initial cloud-ini…ob (metadata service crawler). [ 11.957168] cloud-init[470]: | . o . . .| [ 11.962540] cloud-init[470]: | = . . .| [ 11.963491] cloud-init[470]: | . . o S + . | [ 11.965722] cloud-init[470]: |.o= . + . .| [ OK ] Reached target Cloud-config availability. [ 11.967103] cloud-init[470]: |.===. . .. o | [ 11.970792] cloud-init[470]: |oX=oo+ .. + .| [ 11.974042] cloud-init[470]: |%*B=+.. o= ......| [ 11.976021] cloud-init[470]: +----[SHA256]-----+ [ OK ] Reached target Network is Online. [ OK ] Reached target System Initialization. [ OK ] Started ACPI Events Check. [ OK ] Started Trigger to poll fo…y enabled on GCP LTS non-pro). [ OK ] Started Daily apt download activities. [ OK ] Started Daily apt upgrade and clean activities. [ OK ] Started Periodic ext4 Onli…ata Check for All Filesystems. [ OK ] Started Discard unused blocks once a week. [ OK ] Started Refresh fwupd metadata regularly. [ OK ] Started Daily rotation of log files. [ OK ] Started Daily man-db regeneration. [ OK ] Started Message of the Day. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Ubuntu Advantage Timer for running repeated jobs. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on ACPID Listen Socket. [ OK ] Listening on cloud-init hotplug hook socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Open-iSCSI iscsid Socket. Starting Socket activation for snappy daemon. [ OK ] Listening on UUID daemon activation socket. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Availability of block devices... [ OK ] Listening on Socket activation for snappy daemon. [ OK ] Finished Availability of block devices. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Accounts Service... [ OK ] Started ACPI event daemon. Starting LSB: automatic crash report generation... Starting Deferred execution scheduler... [ OK ] Started Regular background program processing daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Save initial kernel messages after boot. Starting Remove Stale Onli…t4 Metadata Check Snapshots... Starting EC2 Instance Connect Host Key Harvesting... Starting Record successful boot for GRUB... [ OK ] Started irqbalance daemon. Starting Dispatcher daemon for systemd-networkd... Starting Pollinate to seed…udo random number generator... Starting System Logging Service... Starting Snap Daemon... Starting Login Service... Starting Permit User Sessions... Starting Disk Manager... [ OK ] Started Deferred execution scheduler. [ OK ] Finished Permit User Sessions. Starting Hold until boot process finishes up... Starting Terminate Plymouth Boot Screen... [ OK ] Finished Hold until boot process finishes up. [ OK ] Started Serial Getty on ttyS0. Starting Set console scheme... [ OK ] Finished Terminate Plymouth Boot Screen. [ OK ] Finished Set console scheme. [ OK ] Finished Remove Stale Onli…ext4 Metadata Check Snapshots. [ OK ] Created slice system-getty.slice. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started System Logging Service. [ OK ] Started LSB: automatic crash report generation. [ OK ] Finished Record successful boot for GRUB. Starting GRUB failed boot detection... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started Unattended Upgrades Shutdown. [ OK ] Finished GRUB failed boot detection. [ OK ] Started Dispatcher daemon for systemd-networkd. [ OK ] Started Hostname Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Accounts Service. [ OK ] Started Disk Manager. [ OK ] Finished EC2 Instance Connect Host Key Harvesting. [ OK ] Finished Pollinate to seed…seudo random number generator. Starting OpenBSD Secure Shell server... [ OK ] Started OpenBSD Secure Shell server. [ OK ] Started Snap Daemon. Starting Wait until snapd is fully seeded... [ OK ] Started snap.lxd.hook.inst…-45f9-82e9-76ec3439656d.scope. [ OK ] Listening on Socket unix for snap application lxd.daemon. Starting Service for snap application lxd.activate... [ OK ] Finished Service for snap application lxd.activate. [ OK ] Started snap.lxd.hook.conf…-491c-a4a8-0d09fa854bce.scope. [ OK ] Started Service for snap a…on-ssm-agent.amazon-ssm-agent. Ubuntu 20.04.3 LTS ip-10-178-104-100 ttyS0 ip-10-178-104-100 login: 2022/05/04 07:25:12Z: Amazon SSM Agent v3.0.1124.0 is running 2022/05/04 07:25:12Z: OsProductName: Ubuntu 2022/05/04 07:25:12Z: OsVersion: 20.04 [ 18.604307] cloud-init[1296]: Cloud-init v. 21.4-0ubuntu1~20.04.1 running 'modules:config' at Wed, 04 May 2022 07:25:12 +0000. Up 17.84 seconds. [ 19.247143] cloud-init[1321]: Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal InRelease [ 19.247510] cloud-init[1321]: Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] [ 19.247622] cloud-init[1321]: Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] [ 19.377536] cloud-init[1321]: Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] [ 19.404304] cloud-init[1321]: Get:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB] [ 19.541001] cloud-init[1321]: Get:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/universe Translation-en [5124 kB] [ 19.619399] cloud-init[1321]: Get:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 c-n-f Metadata [265 kB] [ 19.622446] cloud-init[1321]: Get:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [144 kB] [ 19.624049] cloud-init[1321]: Get:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/multiverse Translation-en [104 kB] [ 19.625416] cloud-init[1321]: Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/multiverse amd64 c-n-f Metadata [9136 B] [ 19.625967] cloud-init[1321]: Get:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1750 kB] [ 19.640434] cloud-init[1321]: Get:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/main Translation-en [326 kB] [ 19.643569] cloud-init[1321]: Get:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.0 kB] [ 19.644059] cloud-init[1321]: Get:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [947 kB] [ 19.652978] cloud-init[1321]: Get:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [135 kB] [ 19.654549] cloud-init[1321]: Get:16 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B] [ 19.655170] cloud-init[1321]: Get:17 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [921 kB] [ 19.662934] cloud-init[1321]: Get:18 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [206 kB] [ 19.664908] cloud-init[1321]: Get:19 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.7 kB] [ 19.665462] cloud-init[1321]: Get:20 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] [ 19.666032] cloud-init[1321]: Get:21 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7336 B] [ 19.666515] cloud-init[1321]: Get:22 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] [ 19.666754] cloud-init[1321]: Get:23 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] [ 19.667278] cloud-init[1321]: Get:24 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] [ 19.667652] cloud-init[1321]: Get:25 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [864 B] [ 19.667920] cloud-init[1321]: Get:26 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/restricted amd64 c-n-f Metadata [116 B] [ 19.668177] cloud-init[1321]: Get:27 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] [ 19.668615] cloud-init[1321]: Get:28 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.5 kB] [ 19.668989] cloud-init[1321]: Get:29 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] [ 19.669256] cloud-init[1321]: Get:30 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 c-n-f Metadata [116 B] [ 19.807672] cloud-init[1321]: Get:31 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1422 kB] [ 20.194388] cloud-init[1321]: Get:32 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [246 kB] [ 20.200802] cloud-init[1321]: Get:33 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.1 kB] [ 20.201247] cloud-init[1321]: Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [886 kB] [ 20.214842] cloud-init[1321]: Get:35 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [126 kB] [ 20.218049] cloud-init[1321]: Get:36 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B] [ 20.218442] cloud-init[1321]: Get:37 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [700 kB] [ 20.277391] cloud-init[1321]: Get:38 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [124 kB] [ 20.280223] cloud-init[1321]: Get:39 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.4 kB] [ 20.282331] cloud-init[1321]: Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] [ 20.284257] cloud-init[1321]: Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5196 B] [ 20.284540] cloud-init[1321]: Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] [ 28.217184] cloud-init[1321]: Fetched 22.6 MB in 3s (7109 kB/s) [ 29.091601] cloud-init[1321]: Reading package lists... [ 29.172064] cloud-init[1321]: Reading package lists... [ 29.389676] cloud-init[1321]: Building dependency tree... [ 29.390325] cloud-init[1321]: Reading state information... [ 29.574467] cloud-init[1321]: ca-certificates is already the newest version (20210119~20.04.2). [ 29.574781] cloud-init[1321]: ca-certificates set to manually installed. [ 29.574920] cloud-init[1321]: software-properties-common is already the newest version (0.99.9.8). [ 29.575138] cloud-init[1321]: software-properties-common set to manually installed. [ 29.575397] cloud-init[1321]: Suggested packages: [ 29.575643] cloud-init[1321]: python3-aptdaemon.pkcompat [ 29.635866] cloud-init[1321]: The following NEW packages will be installed: [ 29.638769] cloud-init[1321]: apt-transport-https gnupg-agent python3-xkit ubuntu-drivers-common [ 29.640740] cloud-init[1321]: The following packages will be upgraded: [ 29.642819] cloud-init[1321]: curl libcurl4 [ 29.689490] cloud-init[1321]: 2 upgraded, 4 newly installed, 0 to remove and 117 not upgraded. [ 29.735494] cloud-init[1321]: Need to get 478 kB of archives. [ 29.735985] cloud-init[1321]: After this operation, 608 kB of additional disk space will be used. [ 29.736088] cloud-init[1321]: Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/main amd64 python3-xkit all 0.5.0ubuntu4 [18.3 kB] [ 29.738858] cloud-init[1321]: Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-drivers-common amd64 1:0.9.0~0.20.04.7 [53.5 kB] [ 29.741615] cloud-init[1321]: Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4680 B] [ 29.742693] cloud-init[1321]: Get:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.10 [161 kB] [ 29.747140] cloud-init[1321]: Get:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.10 [235 kB] [ 29.751304] cloud-init[1321]: Get:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 gnupg-agent all 2.2.19-3ubuntu2.1 [5232 B] [ 30.025110] cloud-init[1321]: Preconfiguring packages ... [ 30.085864] cloud-init[1321]: Fetched 478 kB in 0s (15.8 MB/s) [ 30.138255] cloud-init[1321]: Selecting previously unselected package python3-xkit. [ 30.675219] cloud-init[1321]: (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63895 files and directories currently installed.) [ 30.684701] cloud-init[1321]: Preparing to unpack .../0-python3-xkit_0.5.0ubuntu4_all.deb ... [ 30.685007] cloud-init[1321]: Unpacking python3-xkit (0.5.0ubuntu4) ... [ 30.719899] cloud-init[1321]: Selecting previously unselected package ubuntu-drivers-common. [ 30.726575] cloud-init[1321]: Preparing to unpack .../1-ubuntu-drivers-common_1%3a0.9.0~0.20.04.7_amd64.deb ... [ 30.726855] cloud-init[1321]: Unpacking ubuntu-drivers-common (1:0.9.0~0.20.04.7) ... [ 30.754205] cloud-init[1321]: Selecting previously unselected package apt-transport-https. [ 30.761858] cloud-init[1321]: Preparing to unpack .../2-apt-transport-https_2.0.6_all.deb ... [ 30.762161] cloud-init[1321]: Unpacking apt-transport-https (2.0.6) ... [ 30.784684] cloud-init[1321]: Preparing to unpack .../3-curl_7.68.0-1ubuntu2.10_amd64.deb ... [ 30.785237] cloud-init[1321]: Unpacking curl (7.68.0-1ubuntu2.10) over (7.68.0-1ubuntu2.7) ... [ 30.839792] cloud-init[1321]: Preparing to unpack .../4-libcurl4_7.68.0-1ubuntu2.10_amd64.deb ... [ 30.851844] cloud-init[1321]: Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.10) over (7.68.0-1ubuntu2.7) ... [ 30.885504] cloud-init[1321]: Selecting previously unselected package gnupg-agent. [ 30.891191] cloud-init[1321]: Preparing to unpack .../5-gnupg-agent_2.2.19-3ubuntu2.1_all.deb ... [ 30.892462] cloud-init[1321]: Unpacking gnupg-agent (2.2.19-3ubuntu2.1) ... [ 30.914778] cloud-init[1321]: Setting up apt-transport-https (2.0.6) ... [ 30.915935] cloud-init[1321]: Setting up gnupg-agent (2.2.19-3ubuntu2.1) ... [ 30.916953] cloud-init[1321]: Setting up libcurl4:amd64 (7.68.0-1ubuntu2.10) ... [ 30.917995] cloud-init[1321]: Setting up curl (7.68.0-1ubuntu2.10) ... [ 30.918971] cloud-init[1321]: Setting up python3-xkit (0.5.0ubuntu4) ... [ 31.062871] cloud-init[1321]: Setting up ubuntu-drivers-common (1:0.9.0~0.20.04.7) ... [ 31.229912] cloud-init[1321]: Created symlink /etc/systemd/system/display-manager.service.wants/gpu-manager.service → /lib/systemd/system/gpu-manager.service. [ 31.230791] cloud-init[1321]: Created symlink /etc/systemd/system/oem-config.service.wants/gpu-manager.service → /lib/systemd/system/gpu-manager.service. [ 31.747952] cloud-init[1321]: Processing triggers for man-db (2.9.1-1) ... [ 32.058206] cloud-init[1321]: Processing triggers for libc-bin (2.31-0ubuntu9.2) ... [ 36.167214] cloud-init[1321]: Warning: apt-key output should not be parsed (stdout is not a terminal) [ 36.302993] cloud-init[1321]: OK [ 36.918148] cloud-init[1321]: Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal InRelease [ 36.919603] cloud-init[1321]: Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [ 36.920964] cloud-init[1321]: Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [ 36.946036] cloud-init[1321]: Get:4 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB] [ 37.187667] cloud-init[1321]: Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease [ 37.274373] cloud-init[1321]: Get:6 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [15.5 kB] [ 41.825848] cloud-init[1321]: Fetched 73.1 kB in 0s (159 kB/s) [ 42.966961] cloud-init[1321]: Reading package lists... [ 43.065140] cloud-init[1321]: Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal InRelease [ 43.065805] cloud-init[1321]: Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [ 43.070488] cloud-init[1321]: Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [ 43.093085] cloud-init[1321]: Hit:4 https://download.docker.com/linux/ubuntu focal InRelease [ 43.316079] cloud-init[1321]: Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease [ 44.587906] cloud-init[1321]: Reading package lists... [ 44.667340] cloud-init[1321]: Reading package lists... [ 44.942350] cloud-init[1321]: Building dependency tree... [ 44.943943] cloud-init[1321]: Reading state information... [ 45.121183] cloud-init[1321]: The following additional packages will be installed: [ 45.122832] cloud-init[1321]: docker-ce-rootless-extras docker-scan-plugin pigz slirp4netns [ 45.124698] cloud-init[1321]: Suggested packages: [ 45.125029] cloud-init[1321]: aufs-tools cgroupfs-mount | cgroup-lite [ 45.194056] cloud-init[1321]: The following NEW packages will be installed: [ 45.194392] cloud-init[1321]: containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras [ 45.195394] cloud-init[1321]: docker-scan-plugin pigz slirp4netns [ 45.234073] cloud-init[1321]: 0 upgraded, 7 newly installed, 0 to remove and 117 not upgraded. [ 45.302508] cloud-init[1321]: Need to get 96.4 MB of archives. [ 45.303584] cloud-init[1321]: After this operation, 405 MB of additional disk space will be used. [ 45.304169] cloud-init[1321]: Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 pigz amd64 2.4-1 [57.4 kB] [ 45.309000] cloud-init[1321]: Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 slirp4netns amd64 0.4.3-1 [74.3 kB] [ 45.329736] cloud-init[1321]: Get:3 https://download.docker.com/linux/ubuntu focal/stable amd64 containerd.io amd64 1.5.11-1 [22.9 MB] [ 45.714033] cloud-init[1321]: Get:4 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-cli amd64 5:20.10.14~3-0~ubuntu-focal [41.0 MB] [ 46.341701] cloud-init[1321]: Get:5 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce amd64 5:20.10.14~3-0~ubuntu-focal [20.9 MB] [ 46.660768] cloud-init[1321]: Get:6 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-rootless-extras amd64 5:20.10.14~3-0~ubuntu-focal [7932 kB] [ 46.784711] cloud-init[1321]: Get:7 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-scan-plugin amd64 0.17.0~ubuntu-focal [3521 kB] [ 47.077735] cloud-init[1321]: dpkg-preconfigure: unable to re-open stdin: No such file or directory [ 47.116059] cloud-init[1321]: Fetched 96.4 MB in 2s (61.8 MB/s) [ 47.149850] cloud-init[1321]: Selecting previously unselected package pigz. [ 47.184153] cloud-init[1321]: (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63957 files and directories currently installed.) [ 47.197964] cloud-init[1321]: Preparing to unpack .../0-pigz_2.4-1_amd64.deb ... [ 47.202773] cloud-init[1321]: Unpacking pigz (2.4-1) ... [ 47.266225] cloud-init[1321]: Selecting previously unselected package containerd.io. [ 47.278222] cloud-init[1321]: Preparing to unpack .../1-containerd.io_1.5.11-1_amd64.deb ... [ 47.293101] cloud-init[1321]: Unpacking containerd.io (1.5.11-1) ... [ 49.851654] cloud-init[1321]: Selecting previously unselected package docker-ce-cli. [ 49.856894] cloud-init[1321]: Preparing to unpack .../2-docker-ce-cli_5%3a20.10.14~3-0~ubuntu-focal_amd64.deb ... [ 49.860487] cloud-init[1321]: Unpacking docker-ce-cli (5:20.10.14~3-0~ubuntu-focal) ... [ 53.919227] cloud-init[1321]: Selecting previously unselected package docker-ce. [ 53.925602] cloud-init[1321]: Preparing to unpack .../3-docker-ce_5%3a20.10.14~3-0~ubuntu-focal_amd64.deb ... [ 53.929686] cloud-init[1321]: Unpacking docker-ce (5:20.10.14~3-0~ubuntu-focal) ... [ 56.164305] cloud-init[1321]: Selecting previously unselected package docker-ce-rootless-extras. [ 56.171802] cloud-init[1321]: Preparing to unpack .../4-docker-ce-rootless-extras_5%3a20.10.14~3-0~ubuntu-focal_amd64.deb ... [ 56.176839] cloud-init[1321]: Unpacking docker-ce-rootless-extras (5:20.10.14~3-0~ubuntu-focal) ... [ 56.825131] cloud-init[1321]: Selecting previously unselected package docker-scan-plugin. [ 56.830858] cloud-init[1321]: Preparing to unpack .../5-docker-scan-plugin_0.17.0~ubuntu-focal_amd64.deb ... [ 56.833852] cloud-init[1321]: Unpacking docker-scan-plugin (0.17.0~ubuntu-focal) ... [ 57.256559] cloud-init[1321]: Selecting previously unselected package slirp4netns. [ 57.264200] cloud-init[1321]: Preparing to unpack .../6-slirp4netns_0.4.3-1_amd64.deb ... [ 57.274046] cloud-init[1321]: Unpacking slirp4netns (0.4.3-1) ... [ 57.358553] cloud-init[1321]: Setting up slirp4netns (0.4.3-1) ... [ 57.367888] cloud-init[1321]: Setting up docker-scan-plugin (0.17.0~ubuntu-focal) ... [ 57.390732] cloud-init[1321]: Setting up containerd.io (1.5.11-1) ... [ 57.551987] cloud-init[1321]: Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service. Starting containerd container runtime... [ OK ] Started containerd container runtime. [ 58.073694] cloud-init[1321]: Setting up docker-ce-cli (5:20.10.14~3-0~ubuntu-focal) ... [ 58.084574] cloud-init[1321]: Setting up pigz (2.4-1) ... [ 58.104539] cloud-init[1321]: Setting up docker-ce-rootless-extras (5:20.10.14~3-0~ubuntu-focal) ... [ 58.114512] cloud-init[1321]: Setting up docker-ce (5:20.10.14~3-0~ubuntu-focal) ... [ 58.273501] cloud-init[1321]: Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service. [ 58.592540] cloud-init[1321]: Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket. Starting Docker Socket for the API. [ OK ] Listening on Docker Socket for the API. Starting Docker Application Container Engine... [ OK ] Started Docker Application Container Engine. [ 60.076540] cloud-init[1321]: Processing triggers for man-db (2.9.1-1) ... [ 62.450325] cloud-init[1321]: Processing triggers for systemd (245.4-4ubuntu3.13) ... [ 66.139317] cloud-init[1321]: Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install. [ 66.141527] cloud-init[1321]: Executing: /lib/systemd/systemd-sysv-install enable docker [ 66.837892] cloud-init[1321]: Unable to find image 'daskdev/dask:latest' locally [ 68.034494] cloud-init[1321]: latest: Pulling from daskdev/dask [ 68.034834] cloud-init[1321]: 4d32b49e2995: Pulling fs layer [ 68.034944] cloud-init[1321]: 267bae6de278: Pulling fs layer [ 68.035114] cloud-init[1321]: cd3ab84d6761: Pulling fs layer [ 68.035360] cloud-init[1321]: 181f9e4f703f: Pulling fs layer [ 68.035575] cloud-init[1321]: 084d92a6edad: Pulling fs layer [ 68.035710] cloud-init[1321]: 181f9e4f703f: Waiting [ 68.035894] cloud-init[1321]: 084d92a6edad: Waiting [ 68.714619] cloud-init[1321]: 4d32b49e2995: Verifying Checksum [ 68.714966] cloud-init[1321]: 4d32b49e2995: Download complete [ 69.047166] cloud-init[1321]: 181f9e4f703f: Download complete [ 69.380957] cloud-init[1321]: 084d92a6edad: Verifying Checksum [ 69.381461] cloud-init[1321]: 084d92a6edad: Download complete [ 69.383566] cloud-init[1321]: 267bae6de278: Verifying Checksum [ 69.383889] cloud-init[1321]: 267bae6de278: Download complete [ 70.041323] cloud-init[1321]: 4d32b49e2995: Pull complete [ 71.561942] cloud-init[1321]: cd3ab84d6761: Verifying Checksum [ 71.562255] cloud-init[1321]: cd3ab84d6761: Download complete [ 72.916334] cloud-init[1321]: 267bae6de278: Pull complete [ 76.818086] cloud-init[1321]: cd3ab84d6761: Pull complete [ 76.892630] cloud-init[1321]: 181f9e4f703f: Pull complete [ 76.951793] cloud-init[1321]: 084d92a6edad: Pull complete [ 76.972657] cloud-init[1321]: Digest: sha256:7aad85ead8a12dc924ce8a5fbfe032aff6627205b6b491c6c0ce1ab57a7ec972 [ 76.978537] cloud-init[1321]: Status: Downloaded newer image for daskdev/dask:latest [ 82.891299] cloud-init[1321]: + '[' '' ']' [ 82.891654] cloud-init[1321]: no environment.yml [ 82.891799] cloud-init[1321]: + '[' '' == true ']' [ 82.891974] cloud-init[1321]: + CONDA_BIN=/opt/conda/bin/conda [ 82.892173] cloud-init[1321]: + '[' -e /opt/app/environment.yml ']' [ 82.892499] cloud-init[1321]: + echo 'no environment.yml' [ 82.892594] cloud-init[1321]: + '[' '' ']' [ 82.893073] cloud-init[1321]: + '[' '' ']' [ 82.893361] cloud-init[1321]: + exec env DASK_INTERNAL_INHERIT_CONFIG=eyJjb2lsZWQiOiB7ImFjY291bnQiOiAiYXJiYXota2hhbjkiLCAiYmFja2VuZC1vcHRpb25zIjogbnVsbCwgIm5hbWUiOiBudWxsLCAibm8tbWluaW11bS12ZXJzaW9uLWNoZWNrIjogZmFsc2UsICJwcm90b2NvbCI6ICJ0bHMiLCAic2NoZWR1bGVyIjogeyJjbGFzcyI6IG51bGwsICJjcHUiOiBudWxsLCAibWVtb3J5IjogbnVsbCwgInZtLXR5cGVzIjogbnVsbH0sICJzY2hlZHVsZXItb3B0aW9ucyI6IHt9LCAic2VydmVyIjogImh0dHBzOi8vY2xvdWQuY29pbGVkLmlvIiwgInNodXRkb3duLW9uLWNsb3NlIjogdHJ1ZSwgInNvZnR3YXJlIjogbnVsbCwgInRva2VuIjogIjFlMDhmYzRiYWUyZjRhZTdhMTYzMmFhZTRmZjQ2YmNiLTZkODM4NzFmYjdkYWIzZTViMzVlMTAxNDg5ZjY4OWY4YjQxMDRlZDEiLCAidXNlciI6ICJhcmJhei1raGFuOSIsICJ3YWl0LWZvci13b3JrZXJzIjogMC4zLCAid29ya2VyIjogeyJjbGFzcyI6IG51bGwsICJjcHUiOiBudWxsLCAiZ3B1IjogbnVsbCwgImdwdS10eXBlcyI6IG51bGwsICJtZW1vcnkiOiBudWxsLCAidm0tdHlwZXMiOiBudWxsfSwgIndvcmtlci1vcHRpb25zIjoge319LCAidGVtcG9yYXJ5LWRpcmVjdG9yeSI6IG51bGwsICJ0b2tlbml6ZSI6IHsiZW5zdXJlLWRldGVybWluaXN0aWMiOiBmYWxzZX0sICJkYXRhZnJhbWUiOiB7InNodWZmbGUtY29tcHJlc3Npb24iOiBudW[2022-05-04T07:26:17.362646]xsLCAicGFycXVldCI6IHsibWV0YWRhdGEtdGFzay1zaXplLWxvY2FsIjogNTEyLCAibWV0YWRhdGEtdGFzay1zaXplLXJlbW90ZSI6IDE2fX0sICJhcnJheSI6IHsic3ZnIjogeyJzaXplIjogMTIwfSwgInNsaWNpbmciOiB7InNwbGl0LWxhcmdlLWNodW5rcyI6IG51bGx9fSwgIm9wdGltaXphdGlvbiI6IHsiZnVzZSI6IHsiYWN0aXZlIjogbnVsbCwgImF2ZS13aWR0aCI6IDEsICJtYXgtd2lkdGgiOiBudWxsLCAibWF4LWhlaWdodCI6IEluZmluaXR5LCAibWF4LWRlcHRoLW5ldy1lZGdlcyI6IG51bGwsICJzdWJncmFwaHMiOiBudWxsLCAicmVuYW1lLWtleXMiOiB0cnVlfX0sICJjbG91ZHByb3ZpZGVyIjogeyJlY3MiOiB7ImZhcmdhdGVfc2NoZWR1bGVyIjogZmFsc2UsICJmYXJnYXRlX3dvcmtlcnMiOiBmYWxzZSwgInNjaGVkdWxlcl9jcHUiOiAxMDI0LCAic2NoZWR1bGVyX21lbSI6IDQwOTYsICJ3b3JrZXJfY3B1IjogNDA5NiwgIndvcmtlcl9tZW0iOiAxNjM4NCwgIndvcmtlcl9ncHUiOiAwLCAibl93b3JrZXJzIjogMCwgInNjaGVkdWxlcl90aW1lb3V0IjogIjUgbWludXRlcyIsICJpbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImdwdV9pbWFnZSI6ICJyYXBpZHNhaS9yYXBpZHNhaTpsYXRlc3QiLCAiY2x1c3Rlcl9uYW1lX3RlbXBsYXRlIjogImRhc2ste3V1aWR9IiwgImNsdXN0ZXJfYXJuIjogIiIsICJleGVjdXRpb25fcm9sZV9hcm4iOiAiIiwgInRhc2tfcm9sZV9hcm4iOiAiIiwgInRhc[2022-05-04T07:26:17.462520]2tfcm9sZV9wb2xpY2llcyI6IFtdLCAiY2xvdWR3YXRjaF9sb2dzX2dyb3VwIjogIiIsICJjbG91ZHdhdGNoX2xvZ3Nfc3RyZWFtX3ByZWZpeCI6ICJ7Y2x1c3Rlcl9uYW1lfSIsICJjbG91ZHdhdGNoX2xvZ3NfZGVmYXVsdF9yZXRlbnRpb24iOiAzMCwgInZwYyI6ICJkZWZhdWx0IiwgInN1Ym5ldHMiOiBbXSwgInNlY3VyaXR5X2dyb3VwcyI6IFtdLCAidGFncyI6IHt9LCAiZW52aXJvbm1lbnQiOiB7fSwgImZpbmRfYWRkcmVzc190aW1lb3V0IjogNjAsICJza2lwX2NsZWFudXAiOiBmYWxzZX0sICJlYzIiOiB7InJlZ2lvbiI6IG51bGwsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJib290c3RyYXAiOiB0cnVlLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJpbnN0YW5jZV90eXBlIjogInQyLm1pY3JvIiwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImZpbGVzeXN0ZW1fc2l6ZSI6IDQwLCAia2V5X25hbWUiOiBudWxsLCAiaWFtX2luc3RhbmNlX3Byb2ZpbGUiOiB7fX0sICJhenVyZSI6IHsibG9jYXRpb24iOiBudWxsLCAicmVzb3VyY2VfZ3JvdXAiOiBudWxsLCAiYXp1cmV2bSI6IHsidm5ldCI6IG51bGwsICJzZWN1cml0eV9ncm91cCI6IG51bGwsICJwdWJsaWNfaW5ncmVzcyI6IHRydWUsICJ2bV9zaXplIjogIlN0YW5kYXJkX0RTMV92MiIsICJkaXNrX3NpemUiOiA1MCwgInNjaGVkdWxlcl92bV9zaXplIjogbnVsbCwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0[2022-05-04T07:26:17.462581]ZXN0IiwgInZtX2ltYWdlIjogeyJwdWJsaXNoZXIiOiAiQ2Fub25pY2FsIiwgIm9mZmVyIjogIlVidW50dVNlcnZlciIsICJza3UiOiAiMTguMDQtTFRTIiwgInZlcnNpb24iOiAibGF0ZXN0In0sICJib290c3RyYXAiOiB0cnVlLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJtYXJrZXRwbGFjZV9wbGFuIjogbnVsbH19LCAiZGlnaXRhbG9jZWFuIjogeyJ0b2tlbiI6IG51bGwsICJyZWdpb24iOiAibnljMyIsICJzaXplIjogInMtMXZjcHUtMWdiIiwgImltYWdlIjogInVidW50dS0yMC0wNC14NjQifSwgImdjcCI6IHsic291cmNlX2ltYWdlIjogInByb2plY3RzL3VidW50dS1vcy1jbG91ZC9nbG9iYWwvaW1hZ2VzL3VidW50dS1taW5pbWFsLTE4MDQtYmlvbmljLXYyMDIwMTAxNCIsICJ6b25lIjogInVzLWVhc3QxLWMiLCAibmV0d29yayI6ICJkZWZhdWx0IiwgIm5ldHdvcmtfcHJvamVjdGlkIjogbnVsbCwgInByb2plY3RpZCI6ICIiLCAib25faG9zdF9tYWludGVuYW5jZSI6ICJURVJNSU5BVEUiLCAibWFjaGluZV90eXBlIjogIm4xLXN0YW5kYXJkLTEiLCAiZmlsZXN5c3RlbV9zaXplIjogNTAsICJuZ3B1cyI6ICIiLCAiZ3B1X3R5cGUiOiAiIiwgImRpc2tfdHlwZSI6ICJwZC1zdGFuZGFyZCIsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJhdXRvX3NodXRkb3duIjogdHJ1ZSwgInB1YmxpY19pbmdyZXNzIjogdHJ1ZX0sICJoZXR6bmVyIjogeyJ0b2tlbiI6IG51bGwsICJsb2NhdGl[2022-05-04T07:26:17.462612]vbiI6ICJmc24xIiwgInNlcnZlcl90eXBlIjogImN4MTEiLCAiaW1hZ2UiOiAidWJ1bnR1LTIwLjA0IiwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImJvb3RzdHJhcCI6IHRydWV9fSwgImRpc3RyaWJ1dGVkIjogeyJ2ZXJzaW9uIjogMiwgInNjaGVkdWxlciI6IHsiYWxsb3dlZC1mYWlsdXJlcyI6IDMsICJiYW5kd2lkdGgiOiAxMDAwMDAwMDAsICJibG9ja2VkLWhhbmRsZXJzIjogW10sICJkZWZhdWx0LWRhdGEtc2l6ZSI6ICIxa2lCIiwgImV2ZW50cy1jbGVhbnVwLWRlbGF5IjogIjFoIiwgImlkbGUtdGltZW91dCI6IG51bGwsICJ0cmFuc2l0aW9uLWxvZy1sZW5ndGgiOiAxMDAwMDAsICJldmVudHMtbG9nLWxlbmd0aCI6IDEwMDAwMCwgIndvcmstc3RlYWxpbmciOiB0cnVlLCAid29yay1zdGVhbGluZy1pbnRlcnZhbCI6ICIxMDBtcyIsICJ3b3JrZXItdHRsIjogbnVsbCwgInBpY2tsZSI6IHRydWUsICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgInVua25vd24tdGFzay1kdXJhdGlvbiI6ICI1MDBtcyIsICJkZWZhdWx0LXRhc2stZHVyYXRpb25zIjogeyJyZWNodW5rLXNwbGl0IjogIjF1cyIsICJzcGxpdC1zaHVmZmxlIjogIjF1cyJ9LCAidmFsaWRhdGUiOiBmYWxzZSwgImRhc2hib2FyZCI6IHsic3RhdHVzIjogeyJ0YXNrLXN0cmVhbS1sZW5ndGgiOiAxMDAwfSwgInRhc2tzIjogeyJ0YXNrLXN0cmVhbS1sZW5ndGgiOiAxMDAwMDB9LCAidGxzIjogeyJjYS1maW[2022-05-04T07:26:17.462638]xlIjogbnVsbCwgImtleSI6IG51bGwsICJjZXJ0IjogbnVsbH0sICJib2tlaC1hcHBsaWNhdGlvbiI6IHsiYWxsb3dfd2Vic29ja2V0X29yaWdpbiI6IFsiKiJdLCAia2VlcF9hbGl2ZV9taWxsaXNlY29uZHMiOiA1MDAsICJjaGVja191bnVzZWRfc2Vzc2lvbnNfbWlsbGlzZWNvbmRzIjogNTAwfX0sICJsb2NrcyI6IHsibGVhc2UtdmFsaWRhdGlvbi1pbnRlcnZhbCI6ICIxMHMiLCAibGVhc2UtdGltZW91dCI6ICIzMHMifSwgImh0dHAiOiB7InJvdXRlcyI6IFsiZGlzdHJpYnV0ZWQuaHR0cC5zY2hlZHVsZXIucHJvbWV0aGV1cyIsICJkaXN0cmlidXRlZC5odHRwLnNjaGVkdWxlci5pbmZvIiwgImRpc3RyaWJ1dGVkLmh0dHAuc2NoZWR1bGVyLmpzb24iLCAiZGlzdHJpYnV0ZWQuaHR0cC5oZWFsdGgiLCAiZGlzdHJpYnV0ZWQuaHR0cC5wcm94eSIsICJkaXN0cmlidXRlZC5odHRwLnN0YXRpY3MiXX0sICJhbGxvd2VkLWltcG9ydHMiOiBbImRhc2siLCAiZGlzdHJpYnV0ZWQiXSwgImFjdGl2ZS1tZW1vcnktbWFuYWdlciI6IHsic3RhcnQiOiBmYWxzZSwgImludGVydmFsIjogIjJzIiwgInBvbGljaWVzIjogW3siY2xhc3MiOiAiZGlzdHJpYnV0ZWQuYWN0aXZlX21lbW9yeV9tYW5hZ2VyLlJlZHVjZVJlcGxpY2FzIn1dfX0sICJ3b3JrZXIiOiB7ImJsb2NrZWQtaGFuZGxlcnMiOiBbXSwgIm11bHRpcHJvY2Vzc2luZy1tZXRob2QiOiAic3Bhd24iLCAidXNlLWZpbGUtbG9ja2luZyI6IHRydWUsICJjb25uZWN0aW9uc[2022-05-04T07:26:17.462663]yI6IHsib3V0Z29pbmciOiA1MCwgImluY29taW5nIjogMTB9LCAicHJlbG9hZCI6IFtdLCAicHJlbG9hZC1hcmd2IjogW10sICJkYWVtb24iOiB0cnVlLCAidmFsaWRhdGUiOiBmYWxzZSwgInJlc291cmNlcyI6IHt9LCAibGlmZXRpbWUiOiB7ImR1cmF0aW9uIjogbnVsbCwgInN0YWdnZXIiOiAiMCBzZWNvbmRzIiwgInJlc3RhcnQiOiBmYWxzZX0sICJwcm9maWxlIjogeyJpbnRlcnZhbCI6ICIxMG1zIiwgImN5Y2xlIjogIjEwMDBtcyIsICJsb3ctbGV2ZWwiOiBmYWxzZX0sICJtZW1vcnkiOiB7InJlY2VudC10by1vbGQtdGltZSI6ICIzMHMiLCAicmViYWxhbmNlIjogeyJtZWFzdXJlIjogIm9wdGltaXN0aWMiLCAic2VuZGVyLW1pbiI6IDAuMywgInJlY2lwaWVudC1tYXgiOiAwLjYsICJzZW5kZXItcmVjaXBpZW50LWdhcCI6IDAuMX0sICJ0YXJnZXQiOiAwLjYsICJzcGlsbCI6IDAuNywgInBhdXNlIjogMC44LCAidGVybWluYXRlIjogMC45NX0sICJodHRwIjogeyJyb3V0ZXMiOiBbImRpc3RyaWJ1dGVkLmh0dHAud29ya2VyLnByb21ldGhldXMiLCAiZGlzdHJpYnV0ZWQuaHR0cC5oZWFsdGgiLCAiZGlzdHJpYnV0ZWQuaHR0cC5zdGF0aWNzIl19fSwgIm5hbm55IjogeyJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImVudmlyb24iOiB7Ik1BTExPQ19UUklNX1RIUkVTSE9MRF8iOiA2NTUzNiwgIk9NUF9OVU1fVEhSRUFEUyI6IDEsICJNS0xfTlVNX1RIUkVBRFMiOiAxfX0sICJjbGllbnQiOiB7[2022-05-04T07:26:17.462694]ImhlYXJ0YmVhdCI6ICI1cyIsICJzY2hlZHVsZXItaW5mby1pbnRlcnZhbCI6ICIycyIsICJzZWN1cml0eS1sb2FkZXIiOiBudWxsfSwgImRlcGxveSI6IHsibG9zdC13b3JrZXItdGltZW91dCI6ICIxNXMiLCAiY2x1c3Rlci1yZXByLWludGVydmFsIjogIjUwMG1zIn0sICJhZGFwdGl2ZSI6IHsiaW50ZXJ2YWwiOiAiMXMiLCAidGFyZ2V0LWR1cmF0aW9uIjogIjVzIiwgIm1pbmltdW0iOiAwLCAibWF4aW11bSI6IEluZmluaXR5LCAid2FpdC1jb3VudCI6IDN9LCAiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMCwgImRlbGF5IjogeyJtaW4iOiAiMXMiLCAibWF4IjogIjIwcyJ9fSwgImNvbXByZXNzaW9uIjogImF1dG8iLCAic2hhcmQiOiAiNjRNaUIiLCAib2ZmbG9hZCI6ICIxME1pQiIsICJkZWZhdWx0LXNjaGVtZSI6ICJ0Y3AiLCAic29ja2V0LWJhY2tsb2ciOiAyMDQ4LCAicmVjZW50LW1lc3NhZ2VzLWxvZy1sZW5ndGgiOiAwLCAidWN4IjogeyJjdWRhLWNvcHkiOiBudWxsLCAidGNwIjogbnVsbCwgIm52bGluayI6IG51bGwsICJpbmZpbmliYW5kIjogbnVsbCwgInJkbWFjbSI6IG51bGwsICJuZXQtZGV2aWNlcyI6IG51bGwsICJyZXVzZS1lbmRwb2ludHMiOiBudWxsLCAiY3JlYXRlLWN1ZGEtY29udGV4dCI6IG51bGx9LCAienN0ZCI6IHsibGV2ZWwiOiAzLCAidGhyZWFkcyI6IDB9LCAidGltZW91dHMiOiB7ImNvbm5lY3QiOiAiMzBzIiwgInRjcCI6ICIzMHMifSwgInJlcXVpcmUtZW5jcnlwdGlvbiI[2022-05-04T07:26:17.462719]6IG51bGwsICJ0bHMiOiB7ImNpcGhlcnMiOiBudWxsLCAibWluLXZlcnNpb24iOiAxLjIsICJtYXgtdmVyc2lvbiI6IG51bGwsICJjYS1maWxlIjogbnVsbCwgInNjaGVkdWxlciI6IHsiY2VydCI6IG51bGwsICJrZXkiOiBudWxsfSwgIndvcmtlciI6IHsia2V5IjogbnVsbCwgImNlcnQiOiBudWxsfSwgImNsaWVudCI6IHsia2V5IjogbnVsbCwgImNlcnQiOiBudWxsfX0sICJ0Y3AiOiB7ImJhY2tlbmQiOiAidG9ybmFkbyJ9LCAid2Vic29ja2V0cyI6IHsic2hhcmQiOiAiOE1pQiJ9fSwgImRpYWdub3N0aWNzIjogeyJudm1sIjogdHJ1ZSwgImNvbXB1dGF0aW9ucyI6IHsibWF4LWhpc3RvcnkiOiAxMDAsICJpZ25vcmUtbW9kdWxlcyI6IFsiZGlzdHJpYnV0ZWQiLCAiZGFzayIsICJ4YXJyYXkiLCAiY3VkZiIsICJjdW1sIiwgInByZWZlY3QiLCAieGdib29zdCJdfX0sICJkYXNoYm9hcmQiOiB7ImxpbmsiOiAie3NjaGVtZX06Ly97aG9zdH06e3BvcnR9L3N0YXR1cyIsICJleHBvcnQtdG9vbCI6IGZhbHNlLCAiZ3JhcGgtbWF4LWl0ZW1zIjogNTAwMCwgInByb21ldGhldXMiOiB7Im5hbWVzcGFjZSI6ICJkYXNrIn19LCAiYWRtaW4iOiB7InRpY2siOiB7ImludGVydmFsIjogIjIwbXMiLCAibGltaXQiOiAiM3MifSwgIm1heC1lcnJvci1sZW5ndGgiOiAxMDAwMCwgImxvZy1sZW5ndGgiOiAxMDAwMCwgImxvZy1mb3JtYXQiOiAiJShuYW1lKXMgLSAlKGxldmVsbmFtZSlzIC0gJShtZXNzYWdlKXMiLCAicGRiLW9uLW[2022-05-04T07:26:17.562512]VyciI6IGZhbHNlLCAic3lzdGVtLW1vbml0b3IiOiB7ImludGVydmFsIjogIjUwMG1zIn0sICJldmVudC1sb29wIjogInRvcm5hZG8ifSwgInJtbSI6IHsicG9vbC1zaXplIjogbnVsbH19fQ== python -m distributed.cli.dask_scheduler [ 84.763086] cloud-init[1321]: distributed.scheduler - INFO - ----------------------------------------------- [ 86.164862] cloud-init[1321]: distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy [ 86.193343] cloud-init[1321]: distributed.scheduler - INFO - State start [ 86.196844] cloud-init[1321]: distributed.scheduler - INFO - ----------------------------------------------- [ 86.200627] cloud-init[1321]: distributed.scheduler - INFO - Clear task state [ 86.202816] cloud-init[1321]: distributed.scheduler - INFO - Scheduler at: tcp://10.178.104.100:8786 [ 86.203075] cloud-init[1321]: distributed.scheduler - INFO - dashboard at: :8787 ```

Is there anything that I am doing wrong or missing here?

jacobtomlinson commented 2 years ago

Looking at those logs it seems that the scheduler starts correctly. When you say "running state with failed instance" what do you mean?

arbazk007 commented 2 years ago

When I click on created ec2 instance from console and go to monitoring tab, it shows status check instance failed. I thought this could be related as I even though my instance is in running state I cannot ssh to log into it. Also, as I mentioned, I am stuck at waiting for scheduler to run at 10.178.104.100:8786 and it doesn't proceed further.

jacobtomlinson commented 2 years ago

Are you able to open the dashboard at 10.178.104.100:8787?

arbazk007 commented 2 years ago

I am using jupyter lab 1.2 (sagemaker studio) so right now I am not able to access dashboard.

jacobtomlinson commented 2 years ago

If port 8787 is open then you should be able to access the regular Dask web dashboard (not the Jupyter Lab one). You could also just open a terminal in jupyter and run curl http://10.178.104.100:8787 to check that it is accessible.

If the cluster manager is hanging like this then it's likely that the EC2 instance is not accessible from the Sagemaker instance. So to troubleshoot it we need to check that connectivity.

arbazk007 commented 2 years ago

It throws connection timeout error

jacobtomlinson commented 2 years ago

That sounds like the security groups aren't set up correctly to allow access to 8786 and 8787 then.

jacobtomlinson commented 2 years ago

Were you able to check the security rules?

arbazk007 commented 2 years ago

Yes, since dask cloudprovider doesn't provide an option to use private IP, we didn't use ec2 cluster. But yes, if we open ports 8786 and 8787 to all in security groups, it worked. Closing this issue.

jacobtomlinson commented 2 years ago

👍. Just FYI private IP support was added in #353.

arbazk007 commented 2 years ago

Thanks for the information @jacobtomlinson . I was able to create EC2 clusters using private ip.