search

alan-turing-institute / data-safe-haven

https://data-safe-haven.readthedocs.io
BSD 3-Clause "New" or "Revised" License
62 stars 15 forks source link

Cloud init failure for conda clean #118

Closed martintoreilly closed 5 years ago

martintoreilly commented 5 years ago

Issue observed on VM "GeneralizedComputeVM-Ubuntu1804Base-201812061312" (IP address at time of test: 137.117.199.166)

Snippet of cloud-init log from just before error (taken from full cloud-init log)

source cache (/anaconda/conda-bld/src_cache)
Size:                                           0 B

git cache (/anaconda/conda-bld/git_cache)
Size:                                           0 B

hg cache (/anaconda/conda-bld/hg_cache)
Size:                                           0 B

svn cache (/anaconda/conda-bld/svn_cache)
Size:                                           0 B

Total:                                          0 B
Proceed ([y]/n)? 
Removing /anaconda/conda-bld/src_cache
Removing /anaconda/conda-bld/git_cache
Removing /anaconda/conda-bld/hg_cache
Removing /anaconda/conda-bld/svn_cache
/var/lib/cloud/instance/scripts/runcmd: 34: /var/lib/cloud/instance/scripts/runcmd: Syntax error: "(" unexpected
Cloud-init v. 18.4-0ubuntu1~18.04.1 running 'modules:final' at Thu, 06 Dec 2018 13:14:58 +0000. Up 63.24 seconds.
2018-12-06 14:10:55,645 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [2]
2018-12-06 14:10:55,662 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2018-12-06 14:10:55,662 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed

The script appears to crash out either during or immediately after the line - conda clean -a This prompts the user with a Proceed ([y]/n)? prompt. Pressing "enter" accepts as if "y" was typed, so one hypothesis might be that the previous version of the cloud init script ran a command prefixing it with a newline, which accepted this prompt and then ran the new command. However, this section of the cloud init script has remained unchanged between 14:16 on 05 Dec (when the script successfully executed) and 15:06 on 06 Dec (when we got this failure). The relevant part of the cloud init script is:

- echo "*** Cleanup... ***"
  - conda clean -a
  - echo "*** Final conda environments... **
  - conda env list

The next command after conda clean -a is to print a section header to screen, which does not occur prior to the failure, so we cannot see why the behaviour of the script should change. The build successfully completed on VM "GeneralizedComputeVM-Ubuntu1804Base-201812051718" (deployed as RG_DSG_201812061003 with IP address at time of test: 10.250.250.13)

To rerun cloud-init http://fosshelp.blogspot.com/2015/10/ubuntu-how-to-rerun-cloud-init-manually.html

martintoreilly commented 5 years ago

Attempt to re-run conda clean in minimal cloud-init run

Our cloud-init script is stored at /var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D/cloud-config.txt

I made a backup of this then replaced it's contents with

#cloud-config
runcmd:
  - echo "*** Cleanup... ***"
  - conda clean -a
  - echo "*** Final conda environments... ***"
  - conda env list

I then ran this using sudo /usr/bin/cloud-init -d single -n runcmd (from this post. The file at /var/lib/cloud/instances/<instance-id>/cloud-config.txt is copied to /var/lib/cloud/instance/cloud-config.txt when cloud-init is run (verified by checking the contents of this file before and after changing the contents in the <instance-id> directory.

martintoreilly commented 5 years ago

Running sudo /usr/bin/cloud-init -d single -n runcmd when the contents of /var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D/cloud-config.txt is our simple script below:

#cloud-config
runcmd:
  - echo "*** Cleanup... ***"
  - conda clean -a
  - echo "*** Final conda environments... ***"
  - conda env list

Output:

atiadmin@GeneralizedComputeVM-Ubuntu1804Base-201812061312:/var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D$ sudo /usr/bin/cloud-init -d single -n runcmd
2018-12-06 16:19:55,397 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2018-12-06 16:19:55,397 - util.py[DEBUG]: Read 18 bytes from /proc/uptime
2018-12-06 16:19:55,397 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2018-12-06 16:19:55,397 - util.py[DEBUG]: Read 18 bytes from /proc/uptime
2018-12-06 16:19:55,397 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg (quiet=False)
2018-12-06 16:19:55,398 - util.py[DEBUG]: Read 3169 bytes from /etc/cloud/cloud.cfg
2018-12-06 16:19:55,398 - util.py[DEBUG]: Attempting to load yaml from string of length 3169 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,410 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90_dpkg.cfg (quiet=False)
2018-12-06 16:19:55,410 - util.py[DEBUG]: Read 82 bytes from /etc/cloud/cloud.cfg.d/90_dpkg.cfg
2018-12-06 16:19:55,411 - util.py[DEBUG]: Attempting to load yaml from string of length 82 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,411 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90-azure.cfg (quiet=False)
2018-12-06 16:19:55,411 - util.py[DEBUG]: Read 536 bytes from /etc/cloud/cloud.cfg.d/90-azure.cfg
2018-12-06 16:19:55,411 - util.py[DEBUG]: Attempting to load yaml from string of length 536 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,414 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/05_logging.cfg (quiet=False)
2018-12-06 16:19:55,414 - util.py[DEBUG]: Read 2057 bytes from /etc/cloud/cloud.cfg.d/05_logging.cfg
2018-12-06 16:19:55,415 - util.py[DEBUG]: Attempting to load yaml from string of length 2057 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,419 - util.py[DEBUG]: Reading from /run/cloud-init/cloud.cfg (quiet=False)
2018-12-06 16:19:55,419 - util.py[DEBUG]: Read 33 bytes from /run/cloud-init/cloud.cfg
2018-12-06 16:19:55,419 - util.py[DEBUG]: Attempting to load yaml from string of length 33 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,420 - util.py[DEBUG]: Running command ['systemd-detect-virt', '--quiet', '--container'] with allowed return codes [0] (shell=False, capture=True)
2018-12-06 16:19:55,424 - util.py[DEBUG]: Running command ['running-in-container'] with allowed return codes [0] (shell=False, capture=True)
2018-12-06 16:19:55,427 - util.py[DEBUG]: Running command ['lxc-is-container'] with allowed return codes [0] (shell=False, capture=True)
2018-12-06 16:19:55,429 - util.py[DEBUG]: Reading from /proc/1/environ (quiet=False)
2018-12-06 16:19:55,429 - util.py[DEBUG]: Read 153 bytes from /proc/1/environ
2018-12-06 16:19:55,430 - util.py[DEBUG]: Reading from /proc/self/status (quiet=False)
2018-12-06 16:19:55,430 - util.py[DEBUG]: Read 1293 bytes from /proc/self/status
2018-12-06 16:19:55,430 - util.py[DEBUG]: Reading from /proc/cmdline (quiet=False)
2018-12-06 16:19:55,430 - util.py[DEBUG]: Read 152 bytes from /proc/cmdline
2018-12-06 16:19:55,430 - util.py[DEBUG]: Attempting to load yaml from string of length 0 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,431 - util.py[DEBUG]: loaded blob returned None, returning default.
2018-12-06 16:19:55,431 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)
2018-12-06 16:19:55,431 - util.py[DEBUG]: Read 138 bytes from /var/lib/cloud/instance/cloud-config.txt
2018-12-06 16:19:55,431 - util.py[DEBUG]: Attempting to load yaml from string of length 138 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,433 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False)
2018-12-06 16:19:55,434 - util.py[DEBUG]: Read 50876 bytes from /var/lib/cloud/instance/obj.pkl
2018-12-06 16:19:55,465 - util.py[DEBUG]: Reading from /etc/os-release (quiet=False)
2018-12-06 16:19:55,465 - util.py[DEBUG]: Read 386 bytes from /etc/os-release
2018-12-06 16:19:55,468 - util.py[DEBUG]: Reading from /run/cloud-init/.instance-id (quiet=False)
2018-12-06 16:19:55,468 - util.py[DEBUG]: Read 37 bytes from /run/cloud-init/.instance-id
2018-12-06 16:19:55,468 - stages.py[DEBUG]: restored from cache with run check: DataSourceAzure [seed=/dev/sr0]
2018-12-06 16:19:55,468 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg (quiet=False)
2018-12-06 16:19:55,468 - util.py[DEBUG]: Read 3169 bytes from /etc/cloud/cloud.cfg
2018-12-06 16:19:55,468 - util.py[DEBUG]: Attempting to load yaml from string of length 3169 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,481 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90_dpkg.cfg (quiet=False)
2018-12-06 16:19:55,482 - util.py[DEBUG]: Read 82 bytes from /etc/cloud/cloud.cfg.d/90_dpkg.cfg
2018-12-06 16:19:55,482 - util.py[DEBUG]: Attempting to load yaml from string of length 82 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,482 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90-azure.cfg (quiet=False)
2018-12-06 16:19:55,482 - util.py[DEBUG]: Read 536 bytes from /etc/cloud/cloud.cfg.d/90-azure.cfg
2018-12-06 16:19:55,483 - util.py[DEBUG]: Attempting to load yaml from string of length 536 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,486 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/05_logging.cfg (quiet=False)
2018-12-06 16:19:55,486 - util.py[DEBUG]: Read 2057 bytes from /etc/cloud/cloud.cfg.d/05_logging.cfg
2018-12-06 16:19:55,486 - util.py[DEBUG]: Attempting to load yaml from string of length 2057 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,490 - util.py[DEBUG]: Reading from /run/cloud-init/cloud.cfg (quiet=False)
2018-12-06 16:19:55,490 - util.py[DEBUG]: Read 33 bytes from /run/cloud-init/cloud.cfg
2018-12-06 16:19:55,490 - util.py[DEBUG]: Attempting to load yaml from string of length 33 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,491 - util.py[DEBUG]: Attempting to load yaml from string of length 0 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,491 - util.py[DEBUG]: loaded blob returned None, returning default.
2018-12-06 16:19:55,491 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)
2018-12-06 16:19:55,492 - util.py[DEBUG]: Read 138 bytes from /var/lib/cloud/instance/cloud-config.txt
2018-12-06 16:19:55,492 - util.py[DEBUG]: Attempting to load yaml from string of length 138 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,493 - main.py[DEBUG]: Closing stdin
2018-12-06 16:19:55,493 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)
2018-12-06 16:19:55,494 - util.py[DEBUG]: Read 138 bytes from /var/lib/cloud/instance/cloud-config.txt
2018-12-06 16:19:55,494 - util.py[DEBUG]: Attempting to load yaml from string of length 138 with allowed root types (<class 'dict'>,)
2018-12-06 16:19:55,495 - util.py[DEBUG]: Redirecting <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> to | tee -a /var/log/cloud-init-output.log
2018-12-06 16:19:55,497 - util.py[DEBUG]: Redirecting <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'> to | tee -a /var/log/cloud-init-output.log
atiadmin@GeneralizedComputeVM-Ubuntu1804Base-201812061312:/var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D$
martintoreilly commented 5 years ago

Running sudo /usr/bin/cloud-init -d single -n runcmd when the contents of /var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D/cloud-config.txt is our original (2 hour long) config script, I get the same output in a second or two, so I don't think anything is being run.

Output:

atiadmin@GeneralizedComputeVM-Ubuntu1804Base-201812061312:/var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D$ sudo mv cloud-config.txt.bak cloud-config.txt 
atiadmin@GeneralizedComputeVM-Ubuntu1804Base-201812061312:/var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D$ sudo /usr/bin/cloud-init -d single -n runcmd
2018-12-06 16:25:44,870 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2018-12-06 16:25:44,870 - util.py[DEBUG]: Read 18 bytes from /proc/uptime
2018-12-06 16:25:44,871 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2018-12-06 16:25:44,871 - util.py[DEBUG]: Read 18 bytes from /proc/uptime
2018-12-06 16:25:44,871 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg (quiet=False)
2018-12-06 16:25:44,871 - util.py[DEBUG]: Read 3169 bytes from /etc/cloud/cloud.cfg
2018-12-06 16:25:44,871 - util.py[DEBUG]: Attempting to load yaml from string of length 3169 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:44,884 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90_dpkg.cfg (quiet=False)
2018-12-06 16:25:44,884 - util.py[DEBUG]: Read 82 bytes from /etc/cloud/cloud.cfg.d/90_dpkg.cfg
2018-12-06 16:25:44,884 - util.py[DEBUG]: Attempting to load yaml from string of length 82 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:44,885 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90-azure.cfg (quiet=False)
2018-12-06 16:25:44,885 - util.py[DEBUG]: Read 536 bytes from /etc/cloud/cloud.cfg.d/90-azure.cfg
2018-12-06 16:25:44,885 - util.py[DEBUG]: Attempting to load yaml from string of length 536 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:44,888 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/05_logging.cfg (quiet=False)
2018-12-06 16:25:44,888 - util.py[DEBUG]: Read 2057 bytes from /etc/cloud/cloud.cfg.d/05_logging.cfg
2018-12-06 16:25:44,888 - util.py[DEBUG]: Attempting to load yaml from string of length 2057 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:44,893 - util.py[DEBUG]: Reading from /run/cloud-init/cloud.cfg (quiet=False)
2018-12-06 16:25:44,893 - util.py[DEBUG]: Read 33 bytes from /run/cloud-init/cloud.cfg
2018-12-06 16:25:44,894 - util.py[DEBUG]: Attempting to load yaml from string of length 33 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:44,894 - util.py[DEBUG]: Running command ['systemd-detect-virt', '--quiet', '--container'] with allowed return codes [0] (shell=False, capture=True)
2018-12-06 16:25:44,898 - util.py[DEBUG]: Running command ['running-in-container'] with allowed return codes [0] (shell=False, capture=True)
2018-12-06 16:25:44,900 - util.py[DEBUG]: Running command ['lxc-is-container'] with allowed return codes [0] (shell=False, capture=True)
2018-12-06 16:25:44,902 - util.py[DEBUG]: Reading from /proc/1/environ (quiet=False)
2018-12-06 16:25:44,902 - util.py[DEBUG]: Read 153 bytes from /proc/1/environ
2018-12-06 16:25:44,902 - util.py[DEBUG]: Reading from /proc/self/status (quiet=False)
2018-12-06 16:25:44,903 - util.py[DEBUG]: Read 1293 bytes from /proc/self/status
2018-12-06 16:25:44,903 - util.py[DEBUG]: Reading from /proc/cmdline (quiet=False)
2018-12-06 16:25:44,903 - util.py[DEBUG]: Read 152 bytes from /proc/cmdline
2018-12-06 16:25:44,903 - util.py[DEBUG]: Attempting to load yaml from string of length 0 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:44,903 - util.py[DEBUG]: loaded blob returned None, returning default.
2018-12-06 16:25:44,904 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)
2018-12-06 16:25:44,904 - util.py[DEBUG]: Read 18672 bytes from /var/lib/cloud/instance/cloud-config.txt
2018-12-06 16:25:44,904 - util.py[DEBUG]: Attempting to load yaml from string of length 18672 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:44,970 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False)
2018-12-06 16:25:44,970 - util.py[DEBUG]: Read 50876 bytes from /var/lib/cloud/instance/obj.pkl
2018-12-06 16:25:44,985 - util.py[DEBUG]: Reading from /etc/os-release (quiet=False)
2018-12-06 16:25:44,986 - util.py[DEBUG]: Read 386 bytes from /etc/os-release
2018-12-06 16:25:44,987 - util.py[DEBUG]: Reading from /run/cloud-init/.instance-id (quiet=False)
2018-12-06 16:25:44,988 - util.py[DEBUG]: Read 37 bytes from /run/cloud-init/.instance-id
2018-12-06 16:25:44,988 - stages.py[DEBUG]: restored from cache with run check: DataSourceAzure [seed=/dev/sr0]
2018-12-06 16:25:44,988 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg (quiet=False)
2018-12-06 16:25:44,988 - util.py[DEBUG]: Read 3169 bytes from /etc/cloud/cloud.cfg
2018-12-06 16:25:44,988 - util.py[DEBUG]: Attempting to load yaml from string of length 3169 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,001 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90_dpkg.cfg (quiet=False)
2018-12-06 16:25:45,001 - util.py[DEBUG]: Read 82 bytes from /etc/cloud/cloud.cfg.d/90_dpkg.cfg
2018-12-06 16:25:45,001 - util.py[DEBUG]: Attempting to load yaml from string of length 82 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,001 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/90-azure.cfg (quiet=False)
2018-12-06 16:25:45,001 - util.py[DEBUG]: Read 536 bytes from /etc/cloud/cloud.cfg.d/90-azure.cfg
2018-12-06 16:25:45,002 - util.py[DEBUG]: Attempting to load yaml from string of length 536 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,004 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/05_logging.cfg (quiet=False)
2018-12-06 16:25:45,004 - util.py[DEBUG]: Read 2057 bytes from /etc/cloud/cloud.cfg.d/05_logging.cfg
2018-12-06 16:25:45,004 - util.py[DEBUG]: Attempting to load yaml from string of length 2057 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,008 - util.py[DEBUG]: Reading from /run/cloud-init/cloud.cfg (quiet=False)
2018-12-06 16:25:45,008 - util.py[DEBUG]: Read 33 bytes from /run/cloud-init/cloud.cfg
2018-12-06 16:25:45,008 - util.py[DEBUG]: Attempting to load yaml from string of length 33 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,009 - util.py[DEBUG]: Attempting to load yaml from string of length 0 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,009 - util.py[DEBUG]: loaded blob returned None, returning default.
2018-12-06 16:25:45,009 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)
2018-12-06 16:25:45,009 - util.py[DEBUG]: Read 18672 bytes from /var/lib/cloud/instance/cloud-config.txt
2018-12-06 16:25:45,009 - util.py[DEBUG]: Attempting to load yaml from string of length 18672 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,057 - main.py[DEBUG]: Closing stdin
2018-12-06 16:25:45,058 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)
2018-12-06 16:25:45,058 - util.py[DEBUG]: Read 18672 bytes from /var/lib/cloud/instance/cloud-config.txt
2018-12-06 16:25:45,058 - util.py[DEBUG]: Attempting to load yaml from string of length 18672 with allowed root types (<class 'dict'>,)
2018-12-06 16:25:45,106 - util.py[DEBUG]: Redirecting <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> to | tee -a /var/log/cloud-init-output.log
2018-12-06 16:25:45,108 - util.py[DEBUG]: Redirecting <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'> to | tee -a /var/log/cloud-init-output.log
2018-12-06 16:25:45,108 - main.py[DEBUG]: Logging being reset, this logger may no longer be active shortly
Cloud-init v. 18.4-0ubuntu1~18.04.1 running 'single' at Thu, 06 Dec 2018 16:25:44 +0000. Up 11509.79 seconds.
atiadmin@GeneralizedComputeVM-Ubuntu1804Base-201812061312:/var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D$
martintoreilly commented 5 years ago

Just running sudo /usr/bin/cloud-init -d modules with the simple file also complete very fast.

martintoreilly commented 5 years ago

Running sudo cloud-init -d single --name runcmd --frequency always (from cloud-init docs) with the simple file also finished fast.

Output from sudo cloud-init analyze show -i /var/log/cloud-init.log is:

atiadmin@GeneralizedComputeVM-Ubuntu1804Base-201812061312:/var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D$ sudo cloud-init analyze show -i /var/log/cloud-init.log
-- Boot Record 01 --
The total time elapsed since completing an event is printed after the "@" character.
The time the event takes is printed after the "+" character.

Starting stage: init-local
|`->no cache found @00.00400s +00.00000s
|`->found local data from DataSourceAzure @00.10100s +01.48600s
Finished stage: (init-local) 01.86200 seconds 

Starting stage: init-network
|`->restored from cache with run check: DataSourceAzure [seed=/dev/sr0] @04.21900s +00.03100s
|`->setting up datasource @04.32300s +08.58600s
|`->reading and applying user-data @12.96900s +00.08300s
|`->reading and applying vendor-data @13.05200s +00.00000s
|`->activating datasource @13.17900s +00.52900s
|`->config-migrator ran successfully @13.77400s +00.00100s
|`->config-seed_random ran successfully @13.77500s +00.00200s
|`->config-bootcmd ran successfully @13.77700s +00.00100s
|`->config-write-files ran successfully @13.77800s +00.00100s
|`->config-growpart ran successfully @13.78000s +00.56800s
|`->config-resizefs ran successfully @14.34900s +01.24200s
|`->config-disk_setup ran successfully @15.59200s +01.33900s
|`->config-mounts ran successfully @16.93200s +00.17300s
|`->config-set_hostname ran successfully @17.10600s +00.00100s
|`->config-update_hostname ran successfully @17.10800s +00.00100s
|`->config-update_etc_hosts ran successfully @17.11000s +00.00000s
|`->config-ca-certs ran successfully @17.11100s +00.00000s
|`->config-rsyslog ran successfully @17.11200s +00.00100s
|`->config-users-groups ran successfully @17.11300s +00.27500s
|`->config-ssh ran successfully @17.38900s +00.15900s
Finished stage: (init-network) 13.34200 seconds 

Starting stage: modules-config
|`->config-emit_upstart ran successfully @22.70300s +00.00000s
|`->config-snap ran successfully @22.70400s +00.00100s
|`->config-snap_config ran successfully @22.70600s +00.00000s
|`->config-ssh-import-id ran successfully @22.70700s +00.00100s
|`->config-locale ran successfully @22.70900s +00.00100s
|`->config-set-passwords ran successfully @22.71000s +00.00100s
|`->config-grub-dpkg ran successfully @22.71200s +00.89400s
|`->config-apt-pipelining ran successfully @23.60700s +00.00200s
|`->config-apt-configure ran successfully @23.60900s +24.28500s
|`->config-ubuntu-advantage ran successfully @47.89500s +00.00100s
|`->config-ntp ran successfully @47.89600s +00.00100s
|`->config-timezone ran successfully @47.89800s +00.00100s
|`->config-disable-ec2-metadata ran successfully @47.89900s +00.00000s
|`->config-runcmd ran successfully @47.90000s +00.00900s
|`->config-byobu ran successfully @47.90900s +00.00100s
Finished stage: (modules-config) 25.24900 seconds 

Starting stage: modules-final
|`->config-snappy ran successfully @48.65200s +00.00200s
|`->config-package-update-upgrade-install ran successfully @48.65500s +1364.11000s
|`->config-fan ran successfully @1412.76600s +00.00400s
|`->config-landscape ran successfully @1412.77000s +00.00100s
|`->config-lxd ran successfully @1412.77200s +00.00000s
|`->config-puppet ran successfully @1412.77300s +00.00100s
|`->config-chef ran successfully @1412.77400s +00.00100s
|`->config-mcollective ran successfully @1412.77500s +00.00100s
|`->config-salt-minion ran successfully @1412.77600s +00.00100s
|`->config-rightscale_userdata ran successfully @1412.77800s +00.00000s
|`->config-scripts-vendor ran successfully @1412.77900s +00.00100s
|`->config-scripts-per-once ran successfully @1412.78100s +00.00100s
|`->config-scripts-per-boot ran successfully @1412.78200s +00.00000s
|`->config-scripts-per-instance ran successfully @1412.78300s +00.00100s
|`->running config-scripts-user with frequency once-per-instance @1412.78400s +1992.93900s
|`->config-ssh-authkey-fingerprints ran successfully @3405.75200s +00.35800s
|`->config-keys-to-console ran successfully @3406.11100s +01.03500s
|`->config-phone-home ran successfully @3407.15000s +00.00100s
|`->config-final-message ran successfully @3407.15200s +00.01200s
|`->config-power-state-change ran successfully @3407.16500s +00.00100s
Finished stage: (modules-final) 3358.57000 seconds 

Starting stage: single
Total Time: 3399.02300 seconds

-- Boot Record 02 --
The total time elapsed since completing an event is printed after the "@" character.
The time the event takes is printed after the "+" character.

Starting stage: modules-config
|`->config-emit_upstart ran successfully @00.01500s +00.00100s
|`->config-snap previously ran @00.01600s +00.00000s
|`->config-snap_config previously ran @00.01700s +00.00000s
|`->config-ssh-import-id previously ran @00.01700s +00.00000s
|`->config-locale previously ran @00.01800s +00.00000s
|`->config-set-passwords previously ran @00.01800s +00.00000s
|`->config-grub-dpkg previously ran @00.01800s +00.00100s
|`->config-apt-pipelining previously ran @00.01900s +00.00000s
|`->config-apt-configure previously ran @00.01900s +00.00000s
|`->config-ubuntu-advantage previously ran @00.02000s +00.00000s
|`->config-ntp previously ran @00.02000s +00.00000s
|`->config-timezone previously ran @00.02100s +00.00000s
|`->config-disable-ec2-metadata ran successfully @00.02100s +00.00000s
|`->config-runcmd previously ran @00.02200s +00.00000s
|`->config-byobu previously ran @00.02200s +00.00000s
Finished stage: (modules-config) 00.02300 seconds 

Starting stage: init-network
|`->restored from cache with run check: DataSourceAzure [seed=/dev/sr0] @32.75600s +00.02800s
|`->setting up datasource @32.84000s +00.00000s
|`->reading and applying user-data @32.84600s +00.07500s
|`->reading and applying vendor-data @32.92100s +00.00000s
|`->activating datasource @33.04600s +00.63500s
|`->config-migrator ran successfully @33.69800s +00.00300s
|`->config-seed_random previously ran @33.70200s +00.00000s
|`->config-bootcmd ran successfully @33.70200s +00.00100s
|`->config-write-files previously ran @33.70300s +00.00000s
|`->config-growpart ran successfully @33.70400s +00.05000s
|`->config-resizefs ran successfully @33.75400s +00.03200s
|`->config-disk_setup previously ran @33.78600s +00.00100s
|`->config-mounts previously ran @33.78700s +00.00100s
|`->config-set_hostname previously ran @33.78800s +00.00000s
|`->config-update_hostname ran successfully @33.78900s +00.00100s
|`->config-update_etc_hosts ran successfully @33.79100s +00.00000s
|`->config-ca-certs previously ran @33.79200s +00.00000s
|`->config-rsyslog previously ran @33.79300s +00.00000s
|`->config-users-groups previously ran @33.79300s +00.00100s
|`->config-ssh previously ran @33.79400s +00.00000s
Finished stage: (init-network) 01.05500 seconds 

Total Time: 1.07800 seconds

-- Boot Record 03 --
The total time elapsed since completing an event is printed after the "@" character.
The time the event takes is printed after the "+" character.

Starting stage: modules-config
|`->config-emit_upstart ran successfully @00.01100s +00.00100s
|`->config-snap previously ran @00.01200s +00.00000s
|`->config-snap_config previously ran @00.01300s +00.00000s
|`->config-ssh-import-id previously ran @00.01400s +00.00000s
|`->config-locale previously ran @00.01400s +00.00000s
|`->config-set-passwords previously ran @00.01500s +00.00000s
|`->config-grub-dpkg previously ran @00.01600s +00.00000s
|`->config-apt-pipelining previously ran @00.01700s +00.00000s
|`->config-apt-configure previously ran @00.01700s +00.00000s
|`->config-ubuntu-advantage previously ran @00.01800s +00.00000s
|`->config-ntp previously ran @00.01900s +00.00000s
|`->config-timezone previously ran @00.01900s +00.00000s
|`->config-disable-ec2-metadata ran successfully @00.02000s +00.00000s
|`->config-runcmd previously ran @00.02100s +00.00000s
|`->config-byobu previously ran @00.02100s +00.00000s
Finished stage: (modules-config) 00.02200 seconds 

Starting stage: single
Total Time: 0.02200 seconds

3 boot records analyzed
atiadmin@GeneralizedComputeVM-Ubuntu1804Base-201812061312:/var/lib/cloud/instances/6A21DCF3-B09A-0943-A457-36F5DB69884D$
andeElliott commented 5 years ago

Another option would be pipe the 'yes' command although this doesnt answer what changed

On Thu, 6 Dec 2018 at 16:39 martintoreilly notifications@github.com wrote:

Running sudo cloud-init -d single --name runcmd --frequency always (from cloud-init docs https://cloudinit.readthedocs.io/en/latest/topics/debugging.html with the simple file also finished fast.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/alan-turing-institute/data-safe-haven/issues/118#issuecomment-444938786, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJcDBblcRg1XnNIWxC3maSTQybHrZJ9ks5u2UgrgaJpZM4ZGu3D .

-- Andrew Elliott

andeElliott commented 5 years ago

you could also just pipe yes to it if that is helpful?

martintoreilly commented 5 years ago

I'm out of ideas on how to debug the cloud-init file on the VM

fkiraly commented 5 years ago

Can we try to crowdsource the issue to members of Turing? I'm thinking of Alex, Kirstie, et al

fkiraly commented 5 years ago

Given that there's one working day left and this issue is marked "critical"... (plus has a "don't know how to solve it" comment)

darenasc commented 5 years ago

maybe installing it in a docker image and leave it there available for people to log in? https://www.alibabacloud.com/help/doc-detail/57803.htm

cathiest commented 5 years ago

Can we try to crowdsource the issue to members of Turing? I'm thinking of Alex, Kirstie, et al

I don't think Alex is going to have the time. We could put a request out on slack...?

martintoreilly commented 5 years ago

Yes, asking for help would be a good step if it's not working first thing tomorrow. I'll try a couple more things tonight.

martintoreilly commented 5 years ago

Removing the conda clean step altogether does not fix this. I'm going to systematically look at all the edits since yesterday wherever they are in the file.

martintoreilly commented 5 years ago

Error is on line 34 of /var/lib/cloud/instance/scripts/runcmd (see relevant lines from near end of cloud-init log at /var/log/cloud-init-output.log)

done
#
# To activate this environment, use:
# > source activate py36
#
# To deactivate an active environment, use:
# > source deactivate
#

*** Skipping conda cleanup to permit new virtual environments to be created by users... ***
/var/lib/cloud/instance/scripts/runcmd: 34: /var/lib/cloud/instance/scripts/runcmd: Syntax error: "(" unexpected
Cloud-init v. 18.4-0ubuntu1~18.04.1 running 'modules:final' at Thu, 06 Dec 2018 18:55:40 +0000. Up 59.30 seconds.
2018-12-06 20:36:21,476 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [2]
2018-12-06 20:36:21,492 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2018-12-06 20:36:21,492 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
The system is setup and deprovisioned, after 6100.23 seconds

The contents of file at /var/lib/cloud/instance/scripts/runcmd are below (created by cloud-init from the runcmd section of our cloud-init yaml file.

#!/bin/sh
echo "*** Cleaning up apt-get packages... ***"
apt-get clean
export ANACONDA_INSTALL_SCRIPT="Anaconda3-5.3.1-Linux-x86_64.sh"
echo "*** Installing Anaconda... ***"
cd /tmp
curl -O https://repo.anaconda.com/archive/${ANACONDA_INSTALL_SCRIPT}
bash $ANACONDA_INSTALL_SCRIPT -b -p /anaconda
rm $ANACONDA_INSTALL_SCRIPT
sed -i 's|\(.*\)"|\1:/anaconda/bin"|' /etc/environment
export PATH=$PATH:/anaconda/bin
echo "export PATH=$PATH:/anaconda/bin" >> /etc/bash.bashrc
conda config --add channels conda-forge
conda config --add channels anaconda
conda config --add channels pytorch
conda config --add channels r
echo "*** Initial conda environments... ***"
conda env list
export PYTHONPACKAGES="ipykernel jupyter_client jupyterlab keras notebook pystan pytorch r-irkernel tensorflow torchvision"
echo "*** Working on py27 environment... ***"
if [ "$(conda env list | grep py27)" = "" ]; then conda create -y --verbose --name py27 python=2.7 $PYTHONPACKAGES; else conda install -y --verbose --name py27 $PYTHONPACKAGES; fi
echo "*** Working on py35 environment... ***"
if [ "$(conda env list | grep py35)" = "" ]; then conda create -y --verbose --name py35 python=3.5 $PYTHONPACKAGES jupyterhub jupyterhub-ldapauthenticator; else conda install -y --verbose --name py35 jupyterhub jupyterhub-ldapauthenticator $PYTHONPACKAGES; fi
echo "*** Working on py36 environment... ***"
if [ "$(conda env list | grep py36)" = "" ]; then conda create -y --verbose --name py36 python=3.6 $PYTHONPACKAGES jupyterhub jupyterhub-ldapauthenticator; else conda install -y --verbose --name py36 jupyterhub jupyterhub-ldapauthenticator $PYTHONPACKAGES; fi
echo "*** Skipping conda cleanup to permit new virtual environments to be created by users... ***"
echo "*** Final conda environments... **
conda env list
if [ "$(conda env list | grep py27)" = "" ]; then echo "Could not build python 2.7 environment"; exit 1; fi
if [ "$(conda env list | grep py35)" = "" ]; then echo "Could not build python 3.5 environment"; exit 1; fi
if [ "$(conda env list | grep py36)" = "" ]; then echo "Could not build python 3.6 environment"; exit 1; fi
echo "*** Installing additional R packages... ***"
echo "Previously installed CRAN packages:"
Rscript -e "installed.packages()[,'Package']"
export CRANPACKAGES="'abind', 'ada', 'akima', 'ape', 'assertthat', 'AUC', 'backports', 'BBmisc', 'bedr', 'BiocManager', 'bitops', 'boot', 'brms', 'car', 'care', 'caret', 'checkmate', 'chron', 'class', 'cluster', 'coda', 'codetools', 'colorRamps', 'colorspace', 'COMBAT', 'compiler', 'corrgram', 'corrplot', 'cowplot', 'CoxBoost', 'crayon', 'CVST', 'cvTools', 'data.table', 'datasets', 'dbarts', 'DBI', 'deepnet', 'devtools', 'DiagrammeR', 'dichromat', 'digest', 'directlabels', 'dirichletprocess', 'dlm', 'doBy', 'doParallel', 'dplyr', 'DPpackage', 'DT', 'dtw', 'dummies', 'dygraphs', 'e1071', 'emulator', 'evaluate', 'factoextra', 'FactoMineR', 'fda', 'fields', 'fmsb', 'foreach', 'forecast', 'foreign', 'Formula', 'gamlss.dist', 'gamlss.mx', 'gamlss.nl', 'gamlss.spatial', 'gamlss', 'gbm', 'gdata', 'GGally', 'ggforce', 'ggmap', 'ggplot2', 'ggridges', 'ggvis', 'glmnet', 'googleVis', 'gplots', 'graphics', 'grDevices', 'grid', 'gridExtra', 'gtable', 'h2o', 'highr', 'Hmisc', 'htmltools', 'httpuv', 'httr', 'igraph', 'irace', 'IRdisplay', 'iterators', 'jsonlite', 'kernlab', 'KernSmooth', 'kknn', 'kml', 'kmlShape', 'knitr', 'labeling', 'lattice', 'lazyeval', 'LDAvis', 'leaflet', 'lme4', 'loo', 'lubridate', 'magrittr', 'maps', 'maptools', 'markdown', 'MASS', 'Matrix', 'matrixStats', 'mboost', 'mclust', 'MCMCpack', 'McSpatial', 'methods', 'mgcv', 'mime', 'mlbench', 'mlr', 'multcomp', 'munsell', 'ndtv', 'network', 'networkD3', 'neuralnet', 'nlme', 'nnet', 'parallel', 'parallelMap', 'ParamHelpers', 'party', 'pbdZMQ', 'pls', 'plyr', 'polycor', 'pomp', 'PReMiuM', 'pscl', 'purrr', 'pvclust', 'quanteda', 'quantmod', 'R6', 'randomForest', 'RColorBrewer', 'RCurl', 'readr', 'readtext', 'readxl', 'repr', 'reshape', 'reshape2', 'revealjs', 'rgdal', 'rgeos', 'rgl', 'rJava', 'rmarkdown', 'RMySQL', 'ROCR', 'roxygen2', 'rpart', 'RPostgreSQL', 'rPython', 'RSQLite', 'rstan', 'runjags', 'RWeka', 'Scale', 'scales', 'shiny', 'slam', 'sna', 'SnowballC', 'sourcetools', 'sp', 'spacyr', 'spatial', 'splines', 'sqldf', 'stargazer', 'stats', 'stats4', 'stm', 'stringi', 'stringr', 'surveillance', 'survival', 'synthpop', 'tcltk2', 'testthat', 'text2vec', 'tgp', 'threejs', 'tibble', 'tidyr', 'tidyr', 'tidytext', 'tidyverse', 'tmap', 'tools', 'topicmodels', 'traj', 'tsne', 'urca', 'utils', 'uuid', 'varbvs', 'vars', 'vcd', 'vioplot', 'viridis', 'visNetwork', 'wordcloud', 'xgboost', 'XLConnect', 'xlsx', 'XML', 'xtable', 'xts', 'yaml', 'zoo'"
echo "CRAN packages remaining to be installed:"
Rscript -e "list.of.packages <- c($CRANPACKAGES); new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,'Package'])]; print(new.packages); if(length(new.packages)) install.packages(new.packages, repos='https://cran.rstudio.com/')"
echo "Previously installed BioConductor packages:"
export BIOCPACKAGES="'apeglm', 'ballgown', 'Biobase', 'ChemmineR', 'clusterProfiler', 'ComplexHeatmap', 'ConsensusClusterPlus', 'cummeRbund', 'dada2', 'DECIPHER', 'DESeq2', 'destiny', 'DirichletMultinomial', 'DMRcate', 'EBSeq', 'edgeR', 'fastseg', 'FlowSOM', 'flowUtils', 'ggtree', 'GOSemSim', 'GOstats', 'graph', 'graphite', 'GSEABase', 'Gviz', 'interactiveDisplayBase', 'KEGGgraph', 'limma', 'made4', 'maftools', 'metagenomeSeq', 'minet', 'MLInterfaces', 'monocle', 'pathview', 'pcaMethods', 'phyloseq', 'RankProd', 'RBGL', 'RDAVIDWebService', 'Rgraphviz', 'safe', 'SC3', 'scater', 'scde', 'scran', 'SNPRelate', 'SPIA', 'supraHex', 'sva', 'TCGAbiolinks', 'TimeSeriesExperiment', 'topGO', 'treeio'"
Rscript -e "list.of.packages <- c($BIOCPACKAGES); old.packages <- list.of.packages[(list.of.packages %in% installed.packages()[,'Package'])]; print(old.packages);"
echo "BioConductor packages remaining to be installed:"
Rscript -e "list.of.packages <- c($BIOCPACKAGES); new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,'Package'])]; print(new.packages); if(length(new.packages)) BiocManager::install(new.packages)"
echo "*** Installing RStudio... ***"
cd /tmp
wget https://download1.rstudio.org/rstudio-xenial-1.1.463-amd64.deb
gdebi --non-interactive rstudio-xenial-1.1.463-amd64.deb
rm -f rstudio-xenial-1.1.463-amd64.deb
if [ ! -e "/usr/bin/rstudio" ]; then echo "Could not install RStudio"; exit 1; fi
echo "*** Installing Julia... ***"
echo "Preparing directory..."
mkdir -p /opt/julia
cd /opt/julia
echo "Downloading package..."
curl -O https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.1-linux-x86_64.tar.gz
echo "Installing and cleaning up..."
tar -zxf julia-1.0.1-linux-x86_64.tar.gz
rm -rf julia-1.0.1-linux-x86_64.tar.gz
sed -i 's|\(.*\)"|\1:/opt/julia/julia-1.0.1/bin"|' /etc/environment
export PATH=$PATH:/opt/julia/julia-1.0.1/bin
echo "export PATH=$PATH:/opt/julia/julia-1.0.1/bin" >> /etc/bash.bashrc
if [ "$(which julia)" = "" ]; then echo "Could not install Julia"; exit 1; fi
echo "*** Setting up R and Julia kernels for jupyter... ***"
Rscript -e "devtools::install_github('IRkernel/IRkernel')"
Rscript -e "IRkernel::installspec(user = FALSE)"
julia -e 'using Pkg; Pkg.add("IJulia"); Pkg.build("IJulia")'
echo "*** Installing spark... ***"
mkdir -p /opt/spark
cd /opt/spark
curl -O http://apache.mirror.anlx.net/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
tar -zxf spark-2.4.0-bin-hadoop2.7.tgz
rm -rf spark-2.4.0-bin-hadoop2.7.tgz
sed -i 's|\(.*\)"|\1:/opt/spark/spark-2.4.0-bin-hadoop2.7/bin"|' /etc/environment
export PATH=$PATH:/opt/spark/spark-2.4.0-bin-hadoop2.7/bin
echo "export PATH=$PATH:/opt/spark/spark-2.4.0-bin-hadoop2.7/bin" >> /etc/bash.bashrc
if [ "$(which spark-shell)" = "" ]; then echo "Could not install spark"; exit 1; fi
echo "*** Installing azcopy... ***"
cd /tmp
wget -O azcopy.tar.gz https://aka.ms/downloadazcopylinux64
tar -xf azcopy.tar.gz
bash install.sh
rm -rf azcopy.tar.gz azcopy install.sh
if [ "$(which azcopy)" = "" ]; then echo "Could not install azcopy"; exit 1; fi
mkdir -p /opt/azurestorage
cd /opt/azurestorage
wget http://download.microsoft.com/download/A/E/3/AE32C485-B62B-4437-92F7-8B6B2C48CB40/previous-StorageExplorer-linux-x64.tar.gz
tar -zxf previous-StorageExplorer-linux-x64.tar.gz
rm -f previous-StorageExplorer-linux-x64.tar.gz
sed -i 's|\(.*\)"|\1:/opt/azurestorage/"|' /etc/environment
export PATH=$PATH:/opt/azurestorage/
echo "export PATH=$PATH:/opt/azurestorage/" >> /etc/bash.bashrc
if [ "$(which StorageExplorer)" = "" ]; then echo "Could not install StorageExplorer"; exit 1; fi
echo "*** Checking for successful installation... ***"
if [ "$(which python)" != "" ]; then echo "python $(which python)"; echo "$(python --version)"; else echo "python not found!"; exit 1; fi
if [ "$(which R)" != "" ]; then echo "R $(which R)"; echo "$(R --version)"; else echo "R not found!"; exit 1; fi
if [ "$(which psql)" != "" ]; then echo "psql $(which psql)\n $(psql --version)"; else echo "psql not found!"; exit 1; fi
if [ "$(which julia)" != "" ]; then echo "julia $(which julia)"; echo "$(julia --version)"; else echo "julia not found!"; exit 1; fi
if [ "$(which java)" != "" ]; then echo "java $(which java)"; echo "$(java -version)"; else echo "java not found!"; exit 1; fi
if [ "$(which dotnet)" != "" ]; then echo "dotnet $(which dotnet)"; echo "$(dotnet --info)"; else echo "dotnet not found!"; exit 1; fi
if [ "$(which gcc)" != "" ]; then echo "gcc $(which gcc)"; echo "$(gcc --version)"; else echo "gcc not found!"; exit 1; fi
if [ "$(which g++)" != "" ]; then echo "g++ $(which g++)"; echo "$(g++ --version)"; else echo "g++ not found!"; exit 1; fi
if [ "$(which gfortran)" != "" ]; then echo "gfortran $(which gfortran)"; echo "$(gfortran --version)"; else echo "gfortran not found!"; exit 1; fi
if [ "$(which bash)" != "" ]; then echo "bash $(which bash)"; echo "$(bash --version)"; else echo "bash not found!"; exit 1; fi
if [ "$(which git)" != "" ]; then echo "git $(which git)"; echo "$(git --version)"; else echo "git not found!"; exit 1; fi
if [ "$(which htop)" != "" ]; then echo "htop $(which htop)"; echo "$(htop --version)"; else echo "htop not found!"; exit 1; fi
if [ "$(which firefox)" != "" ]; then echo "firefox $(which firefox)"; echo "$(firefox --version)"; else echo "firefox not found!"; exit 1; fi
if [ "$(which vim)" != "" ]; then echo "vim $(which vim)"; echo "$(vim --version)"; else echo "vim not found!"; exit 1; fi
if [ "$(which emacs)" != "" ]; then echo "emacs $(which emacs)"; echo "$(emacs --version)"; else echo "emacs not found!"; exit 1; fi
if [ "$(which nano)" != "" ]; then echo "nano $(which nano)"; echo "$(nano --version)"; else echo "nano not found!"; exit 1; fi
if [ "$(which code)" != "" ]; then echo "code $(which code)"; echo "$(code --version)"; else echo "code not found!"; exit 1; fi
if [ "$(which atom)" != "" ]; then echo "atom $(which atom)"; echo "$(dpkg -s atom)"; else echo "atom not found!"; exit 1; fi
if [ "$(which jupyter)" != "" ]; then echo "jupyter $(which jupyter)"; echo "$(jupyter --version)"; else echo "jupyter not found!"; exit 1; fi
if [ "$(which docker)" != "" ]; then echo "docker $(which docker)"; echo "$(docker --version)"; else echo "docker not found!"; exit 1; fi
if [ "$(which az)" != "" ]; then echo "az $(which az)"; echo "$(az --version)"; else echo "az not found!"; exit 1; fi
if [ "$(which azcopy)" != "" ]; then echo "azcopy $(which azcopy)"; echo "$(azcopy --version)"; else echo "azcopy not found!"; exit 1; fi
if [ "$(which scala)" != "" ]; then echo "scala $(which scala)"; else echo "scala not found!"; exit 1; fi
if [ "$(which nvidia-smi)" != "" ]; then echo "nvidia-smi $(which nvidia-smi)"; echo "$(nvidia-smi --help)"; else echo "nvidia-smi not found!"; exit 1; fi
if [ "$(which rstudio)" != "" ]; then echo "rstudio $(which rstudio)"; echo "$(dpkg -s rstudio)"; else echo "rstudio not found!"; fi
echo "CRAN packages that could not be installed:"
Rscript -e "list.of.packages <- c($CRANPACKAGES); missing.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,'Package'])]; print(missing.packages)"
echo "Bioconductor packages that could not be installed:"
Rscript -e "list.of.packages <- c($BIOCPACKAGES); missing.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,'Package'])]; print(missing.packages)"
waagent -deprovision+user -force
ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
moreilly@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~$

Depending on whether the initial #!/bin/sh comment is counted as a line, the offending line 34 is either: conda env list or if [ "$(conda env list | grep py27)" = "" ]; then echo "Could not build python 2.7 environment"; exit 1; fi

Note that both are after the immediately preceding line, which is not printed to the cloud-init output log. echo "*** Final conda environments... **

martintoreilly commented 5 years ago

Running sudo if [ "$(conda env list | grep py27)" = "" ]; then conda create -y --verbose --name py27 python=2.7 $PYTHONPACKAGES; else conda install -y --verbose --name py27 $PYTHONPACKAGES; fi gives a similar error (syntax error near unexpected tokenthen'`), so this is the likely culprit.

martintoreilly commented 5 years ago

It doesn't look like it's the comparison:

moreilly@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~$ if [ "$(conda env list | grep py27)" = "" ]; then echo HI; else echo BYE; fi
BYE
martintoreilly commented 5 years ago

Actually. I get different behaviour running as root rather than sudo'ing the command

moreilly@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~$ sudo bash
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# if [ "$(conda env list | grep py27)" = "" ]; then conda create -y --verbose --name py27 python=2.7 $PYTHONPACKAGES; else conda install -y --verbose --name py27 $PYTHONPACKAGES; fi 
Traceback (most recent call last):
  File "/anaconda/lib/python3.7/site-packages/conda/exceptions.py", line 819, in __call__
    return func(*args, **kwargs)
  File "/anaconda/lib/python3.7/site-packages/conda/cli/main.py", line 78, in _main
    exit_code = do_call(args, p)
  File "/anaconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/anaconda/lib/python3.7/site-packages/conda/cli/main_install.py", line 11, in execute
    install(args, parser, 'install')
  File "/anaconda/lib/python3.7/site-packages/conda/cli/install.py", line 191, in install
    raise CondaValueError("too few arguments, "
conda.exceptions.CondaValueError: too few arguments, must supply command line package specs or --file
martintoreilly commented 5 years ago

The $PYTHONPACKAGES environmental variable wasn't set

root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# echo $PYTHONPACKAGES

root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# export PYTHONPACKAGES="ipykernel jupyter_client jupyterlab keras notebook pystan pytorch r-irkernel tensorflow torchvision"
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# echo $PYTHONPACKAGES
ipykernel jupyter_client jupyterlab keras notebook pystan pytorch r-irkernel tensorflow torchvision
martintoreilly commented 5 years ago

When it is set, the conda install seems to work.

martintoreilly commented 5 years ago

Ok. Ignore all that. I've got my line counts way off. Line 34 is one of

Rscript -e "installed.packages()[,'Package']"
export CRANPACKAGES="'abind', 'ada', 'akima', 'ape', 'assertthat', 'AUC', 'backports', 'BBmisc', 'bedr', 'BiocManager', 'bitops', 'boot', 'brms', 'car', 'care', 'caret', 'checkmate', 'chron', 'class', 'cluster', 'coda', 'codetools', 'colorRamps', 'colorspace', 'COMBAT', 'compiler', 'corrgram', 'corrplot', 'cowplot', 'CoxBoost', 'crayon', 'CVST', 'cvTools', 'data.table', 'datasets', 'dbarts', 'DBI', 'deepnet', 'devtools', 'DiagrammeR', 'dichromat', 'digest', 'directlabels', 'dirichletprocess', 'dlm', 'doBy', 'doParallel', 'dplyr', 'DPpackage', 'DT', 'dtw', 'dummies', 'dygraphs', 'e1071', 'emulator', 'evaluate', 'factoextra', 'FactoMineR', 'fda', 'fields', 'fmsb', 'foreach', 'forecast', 'foreign', 'Formula', 'gamlss.dist', 'gamlss.mx', 'gamlss.nl', 'gamlss.spatial', 'gamlss', 'gbm', 'gdata', 'GGally', 'ggforce', 'ggmap', 'ggplot2', 'ggridges', 'ggvis', 'glmnet', 'googleVis', 'gplots', 'graphics', 'grDevices', 'grid', 'gridExtra', 'gtable', 'h2o', 'highr', 'Hmisc', 'htmltools', 'httpuv', 'httr', 'igraph', 'irace', 'IRdisplay', 'iterators', 'jsonlite', 'kernlab', 'KernSmooth', 'kknn', 'kml', 'kmlShape', 'knitr', 'labeling', 'lattice', 'lazyeval', 'LDAvis', 'leaflet', 'lme4', 'loo', 'lubridate', 'magrittr', 'maps', 'maptools', 'markdown', 'MASS', 'Matrix', 'matrixStats', 'mboost', 'mclust', 'MCMCpack', 'McSpatial', 'methods', 'mgcv', 'mime', 'mlbench', 'mlr', 'multcomp', 'munsell', 'ndtv', 'network', 'networkD3', 'neuralnet', 'nlme', 'nnet', 'parallel', 'parallelMap', 'ParamHelpers', 'party', 'pbdZMQ', 'pls', 'plyr', 'polycor', 'pomp', 'PReMiuM', 'pscl', 'purrr', 'pvclust', 'quanteda', 'quantmod', 'R6', 'randomForest', 'RColorBrewer', 'RCurl', 'readr', 'readtext', 'readxl', 'repr', 'reshape', 'reshape2', 'revealjs', 'rgdal', 'rgeos', 'rgl', 'rJava', 'rmarkdown', 'RMySQL', 'ROCR', 'roxygen2', 'rpart', 'RPostgreSQL', 'rPython', 'RSQLite', 'rstan', 'runjags', 'RWeka', 'Scale', 'scales', 'shiny', 'slam', 'sna', 'SnowballC', 'sourcetools', 'sp', 'spacyr', 'spatial', 'splines', 'sqldf', 'stargazer', 'stats', 'stats4', 'stm', 'stringi', 'stringr', 'surveillance', 'survival', 'synthpop', 'tcltk2', 'testthat', 'text2vec', 'tgp', 'threejs', 'tibble', 'tidyr', 'tidyr', 'tidytext', 'tidyverse', 'tmap', 'tools', 'topicmodels', 'traj', 'tsne', 'urca', 'utils', 'uuid', 'varbvs', 'vars', 'vcd', 'vioplot', 'viridis', 'visNetwork', 'wordcloud', 'xgboost', 'XLConnect', 'xlsx', 'XML', 'xtable', 'xts', 'yaml', 'zoo'"

in the wider context

echo "*** Skipping conda cleanup to permit new virtual environments to be created by users... ***"
echo "*** Final conda environments... **
conda env list
if [ "$(conda env list | grep py27)" = "" ]; then echo "Could not build python 2.7 environment"; exit 1; fi
if [ "$(conda env list | grep py35)" = "" ]; then echo "Could not build python 3.5 environment"; exit 1; fi
if [ "$(conda env list | grep py36)" = "" ]; then echo "Could not build python 3.6 environment"; exit 1; fi
echo "*** Installing additional R packages... ***"
echo "Previously installed CRAN packages:"
Rscript -e "installed.packages()[,'Package']"
export CRANPACKAGES="'abind', 'ada', 'akima', 'ape', 'assertthat', 'AUC', 'backports', 'BBmisc', 'bedr', 'BiocManager', 'bitops', 'boot', 'brms', 'car', 'care', 'caret', 'checkmate', 'chron', 'class', 'cluster', 'coda', 'codetools', 'colorRamps', 'colorspace', 'COMBAT', 'compiler', 'corrgram', 'corrplot', 'cowplot', 'CoxBoost', 'crayon', 'CVST', 'cvTools', 'data.table', 'datasets', 'dbarts', 'DBI', 'deepnet', 'devtools', 'DiagrammeR', 'dichromat', 'digest', 'directlabels', 'dirichletprocess', 'dlm', 'doBy', 'doParallel', 'dplyr', 'DPpackage', 'DT', 'dtw', 'dummies', 'dygraphs', 'e1071', 'emulator', 'evaluate', 'factoextra', 'FactoMineR', 'fda', 'fields', 'fmsb', 'foreach', 'forecast', 'foreign', 'Formula', 'gamlss.dist', 'gamlss.mx', 'gamlss.nl', 'gamlss.spatial', 'gamlss', 'gbm', 'gdata', 'GGally', 'ggforce', 'ggmap', 'ggplot2', 'ggridges', 'ggvis', 'glmnet', 'googleVis', 'gplots', 'graphics', 'grDevices', 'grid', 'gridExtra', 'gtable', 'h2o', 'highr', 'Hmisc', 'htmltools', 'httpuv', 'httr', 'igraph', 'irace', 'IRdisplay', 'iterators', 'jsonlite', 'kernlab', 'KernSmooth', 'kknn', 'kml', 'kmlShape', 'knitr', 'labeling', 'lattice', 'lazyeval', 'LDAvis', 'leaflet', 'lme4', 'loo', 'lubridate', 'magrittr', 'maps', 'maptools', 'markdown', 'MASS', 'Matrix', 'matrixStats', 'mboost', 'mclust', 'MCMCpack', 'McSpatial', 'methods', 'mgcv', 'mime', 'mlbench', 'mlr', 'multcomp', 'munsell', 'ndtv', 'network', 'networkD3', 'neuralnet', 'nlme', 'nnet', 'parallel', 'parallelMap', 'ParamHelpers', 'party', 'pbdZMQ', 'pls', 'plyr', 'polycor', 'pomp', 'PReMiuM', 'pscl', 'purrr', 'pvclust', 'quanteda', 'quantmod', 'R6', 'randomForest', 'RColorBrewer', 'RCurl', 'readr', 'readtext', 'readxl', 'repr', 'reshape', 'reshape2', 'revealjs', 'rgdal', 'rgeos', 'rgl', 'rJava', 'rmarkdown', 'RMySQL', 'ROCR', 'roxygen2', 'rpart', 'RPostgreSQL', 'rPython', 'RSQLite', 'rstan', 'runjags', 'RWeka', 'Scale', 'scales', 'shiny', 'slam', 'sna', 'SnowballC', 'sourcetools', 'sp', 'spacyr', 'spatial', 'splines', 'sqldf', 'stargazer', 'stats', 'stats4', 'stm', 'stringi', 'stringr', 'surveillance', 'survival', 'synthpop', 'tcltk2', 'testthat', 'text2vec', 'tgp', 'threejs', 'tibble', 'tidyr', 'tidyr', 'tidytext', 'tidyverse', 'tmap', 'tools', 'topicmodels', 'traj', 'tsne', 'urca', 'utils', 'uuid', 'varbvs', 'vars', 'vcd', 'vioplot', 'viridis', 'visNetwork', 'wordcloud', 'xgboost', 'XLConnect', 'xlsx', 'XML', 'xtable', 'xts', 'yaml', 'zoo'"
echo "CRAN packages remaining to be installed:"
Rscript -e "list.of.packages <- c($CRANPACKAGES); new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,'Package'])]; print(new.packages); if(length(new.packages)) install.packages(new.packages, repos='https://cran.rstudio.com/')"
echo "Previously installed BioConductor packages:"
export BIOCPACKAGES="'apeglm', 'ballgown', 'Biobase', 'ChemmineR', 'clusterProfiler', 'ComplexHeatmap', 'ConsensusClusterPlus', 'cummeRbund', 'dada2', 'DECIPHER', 'DESeq2', 'destiny', 'DirichletMultinomial', 'DMRcate', 'EBSeq', 'edgeR', 'fastseg', 'FlowSOM', 'flowUtils', 'ggtree', 'GOSemSim', 'GOstats', 'graph', 'graphite', 'GSEABase', 'Gviz', 'interactiveDisplayBase', 'KEGGgraph', 'limma', 'made4', 'maftools', 'metagenomeSeq', 'minet', 'MLInterfaces', 'monocle', 'pathview', 'pcaMethods', 'phyloseq', 'RankProd', 'RBGL', 'RDAVIDWebService', 'Rgraphviz', 'safe', 'SC3', 'scater', 'scde', 'scran', 'SNPRelate', 'SPIA', 'supraHex', 'sva', 'TCGAbiolinks', 'TimeSeriesExperiment', 'topGO', 'treeio'"
Rscript -e "list.of.packages <- c($BIOCPACKAGES); old.packages <- list.of.packages[(list.of.packages %in% installed.packages()[,'Package'])]; print(old.packages);"
echo "BioConductor packages remaining to be installed:"
Rscript -e "list.of.packages <- c($BIOCPACKAGES); new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,'Package'])]; print(new.packages); if(length(new.packages)) BiocManager::install(new.packages)"
echo "*** Installing RStudio... ***"
martintoreilly commented 5 years ago

All the lines in the wider context above run successfully when run individually from a bash shell as root.

martintoreilly commented 5 years ago

Running /var/lib/cloud/instance/scripts/runcmd directly gives the same line 34 error. Interestingly, running it again results in the python 3.5 and 3.6 anaconda installs being skipped, but the Python 2.7 install being re-run (for the third time including the initial cloud-init run and me running it separately above). Could the issue somehow be related to this?

root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# /var/lib/cloud/instance/scripts/runcmd
*** Cleaning up apt-get packages... ***
*** Installing Anaconda... ***
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  637M  100  637M    0     0   119M      0  0:00:05  0:00:05 --:--:--  117M
ERROR: File or directory already exists: '/anaconda'
If you want to update an existing installation, use the -u option.
*** Initial conda environments... ***
# conda environments:
#
base                  *  /anaconda
py27                     /anaconda/envs/py27
py35                     /anaconda/envs/py35
py36                     /anaconda/envs/py36

*** Working on py27 environment... ***
Solving environment: ...working... done

## Package Plan ##

  environment location: /anaconda/envs/py27

  added / updated specs: 
    - ipykernel
    - jupyter_client
    - jupyterlab
    - keras
    - notebook
    - pystan
    - pytorch
    - r-irkernel
    - tensorflow
    - torchvision

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ipykernel-4.10.0           |           py27_0         145 KB  anaconda
    certifi-2018.10.15         |           py27_0         139 KB  anaconda
    pystan-2.18.0.0            |   py27h962f231_0        18.3 MB  anaconda
    jupyter_client-5.2.3       |           py27_0         122 KB  anaconda
    notebook-5.7.2             |           py27_0         7.2 MB  anaconda
    ca-certificates-2018.03.07 |                0         124 KB  anaconda
    tensorflow-1.12.0          |mkl_py27hc55d17a_0           4 KB  anaconda
    jupyterlab-0.33.11         |           py27_0        10.0 MB  anaconda
    keras-2.2.4                |                0           5 KB  anaconda
    ------------------------------------------------------------
                                           Total:        36.0 MB

The following packages will be UPDATED:

    ca-certificates: 2018.03.07-0              --> 2018.03.07-0                anaconda
    certifi:         2018.10.15-py27_0         --> 2018.10.15-py27_0           anaconda
    ipykernel:       4.10.0-py27_0             --> 4.10.0-py27_0               anaconda
    jupyter_client:  5.2.3-py27_0              --> 5.2.3-py27_0                anaconda
    jupyterlab:      0.33.11-py27_0            --> 0.33.11-py27_0              anaconda
    keras:           2.2.4-0                   --> 2.2.4-0                     anaconda
    notebook:        5.7.2-py27_0              --> 5.7.2-py27_0                anaconda
    openssl:         1.1.1a-h7b6447c_0         --> 1.1.1-h7b6447c_0            anaconda
    pystan:          2.18.0.0-py27h962f231_0   --> 2.18.0.0-py27h962f231_0     anaconda
    pytorch:         0.4.1-py27ha74772b_0      --> 0.4.1-py27__9.0.176_7.1.2_2 pytorch 
    tensorflow:      1.12.0-mkl_py27hc55d17a_0 --> 1.12.0-mkl_py27hc55d17a_0   anaconda
    torchvision:     0.2.1-py27_0              --> 0.2.1-py27_1                pytorch 

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... ===> UNLINKING PACKAGE: defaults::keras-2.2.4-0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::tensorflow-1.12.0-mkl_py27hc55d17a_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::torchvision-0.2.1-py27_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::pytorch-0.4.1-py27ha74772b_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::pystan-2.18.0.0-py27h962f231_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::jupyterlab-0.33.11-py27_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::notebook-5.7.2-py27_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::ipykernel-4.10.0-py27_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::jupyter_client-5.2.3-py27_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::certifi-2018.10.15-py27_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::openssl-1.1.1a-h7b6447c_0 <===
  prefix=/anaconda/envs/py27

===> UNLINKING PACKAGE: defaults::ca-certificates-2018.03.07-0 <===
  prefix=/anaconda/envs/py27

===> LINKING PACKAGE: anaconda::ca-certificates-2018.03.07-0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/ca-certificates-2018.03.07-0

===> LINKING PACKAGE: anaconda::openssl-1.1.1-h7b6447c_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/openssl-1.1.1-h7b6447c_0

===> LINKING PACKAGE: anaconda::certifi-2018.10.15-py27_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/certifi-2018.10.15-py27_0

===> LINKING PACKAGE: anaconda::jupyter_client-5.2.3-py27_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/jupyter_client-5.2.3-py27_0

===> LINKING PACKAGE: anaconda::ipykernel-4.10.0-py27_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/ipykernel-4.10.0-py27_0

===> LINKING PACKAGE: anaconda::notebook-5.7.2-py27_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/notebook-5.7.2-py27_0

===> LINKING PACKAGE: anaconda::jupyterlab-0.33.11-py27_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/jupyterlab-0.33.11-py27_0

===> LINKING PACKAGE: anaconda::pystan-2.18.0.0-py27h962f231_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/pystan-2.18.0.0-py27h962f231_0

===> LINKING PACKAGE: pytorch::pytorch-0.4.1-py27__9.0.176_7.1.2_2 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/pytorch-0.4.1-py27__9.0.176_7.1.2_2

===> LINKING PACKAGE: pytorch::torchvision-0.2.1-py27_1 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/torchvision-0.2.1-py27_1

===> LINKING PACKAGE: anaconda::tensorflow-1.12.0-mkl_py27hc55d17a_0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/tensorflow-1.12.0-mkl_py27hc55d17a_0

===> LINKING PACKAGE: anaconda::keras-2.2.4-0 <===
  prefix=/anaconda/envs/py27
  source=/anaconda/pkgs/keras-2.2.4-0

done
*** Working on py35 environment... ***
Solving environment: ...working... done

# All requested packages already installed.

*** Working on py36 environment... ***
Solving environment: ...working... done

# All requested packages already installed.

*** Skipping conda cleanup to permit new virtual environments to be created by users... ***
/var/lib/cloud/instance/scripts/runcmd: 34: /var/lib/cloud/instance/scripts/runcmd: Syntax error: "(" unexpected
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~#
martintoreilly commented 5 years ago

Running the "skipping conda" line and the immediately following lines in one block results in what looks like a "hanging bash prompt", so it feels like something isn't properly closed.

root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# echo "*** Skipping conda cleanup to permit new virtual environments to be created by users... ***"
*** Skipping conda cleanup to permit new virtual environments to be created by users... ***
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# echo "*** Final conda environments... **
> conda env list
> if [ "$(conda env list | grep py27)" = "" ]; then echo "Could not build python 2.7 environment"; exit 1; fi
> if [ "$(conda env list | grep py35)" = "" ]; then echo "Could not build python 3.5 environment"; exit 1; fi
> if [ "$(conda env list | grep py36)" = "" ]; then echo "Could not build python 3.6 environment"; exit 1; fi
> 
>
martintoreilly commented 5 years ago

Running all the conda environment checks as one block appears to work. Pasting them into the command prompt.

if [ "$(conda env list | grep py27)" = "" ]; then echo "Could not build python 2.7 environment"; exit 1; fi
if [ "$(conda env list | grep py35)" = "" ]; then echo "Could not build python 3.5 environment"; exit 1; fi
if [ "$(conda env list | grep py36)" = "" ]; then echo "Could not build python 3.6 environment"; exit 1; fi

Copy and paste from the command line.

root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# if [ "$(conda env list | grep py27)" = "" ]; then echo "Could not build python 2.7 environment"; exit 1; fi

root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# if [ "$(conda env list | grep py35)" = "" ]; then echo "Could not build python 3.5 environment"; exit 1; fi
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# if [ "$(conda env list | grep py36)" = "" ]; then echo "Could not build python 3.6 environment"; exit 1; fi
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~#
martintoreilly commented 5 years ago

Ok. I'm pretty sure the issue is a missing " at the end of the line echo "*** Final conda environments... **

Fixing this and pasting the last output and following lines results in all lines being successfully executed.

oot@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# echo "*** Skipping conda cleanup to permit new virtual environments to be created by users... ***"
*** Skipping conda cleanup to permit new virtual environments to be created by users... ***
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# echo "*** Final conda environments... ***"
*** Final conda environments... ***
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# conda env list

# conda environments:
#
base                  *  /anaconda
py27                     /anaconda/envs/py27
py35                     /anaconda/envs/py35
py36                     /anaconda/envs/py36

root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# if [ "$(conda env list | grep py27)" = "" ]; then echo "Could not build python 2.7 environment"; exit 1; fi
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# if [ "$(conda env list | grep py35)" = "" ]; then echo "Could not build python 3.5 environment"; exit 1; fi
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~# if [ "$(conda env list | grep py36)" = "" ]; then echo "Could not build python 3.6 environment"; exit 1; fi
root@GeneralizedComputeVM-Ubuntu1804Base-201812061853:~#
martintoreilly commented 5 years ago

Running a test build from commit a0184aa

(py3) MAC-ATI0132:azure-vms moreilly$ ./build_azure_vm_image.sh -i Ubuntu -s "Safe Haven Management Testing"
Ensuring GalleryPreview feature is registered and galleries/images/versions resource is present in namespace Microsoft.Compute (this may take some time).
Current GalleryPreview feature state is Registered.
Resource galleries/images/versions is present.
Provisioning a new VM image in RG_SH_IMAGEGALLERY as part of Safe Haven Management Testing
  VM name: GeneralizedComputeVM-Ubuntu1804Base-201812062325
  Base image: Canonical:UbuntuServer:18.04-LTS:latest
{
  "fqdns": "",
  "id": "/subscriptions/1e79c270-3126-43de-b035-22c3118dd488/resourceGroups/RG_SH_IMAGEGALLERY/providers/Microsoft.Compute/virtualMachines/GeneralizedComputeVM-Ubuntu1804Base-201812062325",
  "location": "westeurope",
  "macAddress": "00-0D-3A-28-21-D7",
  "powerState": "VM running",
  "privateIpAddress": "10.0.0.24",
  "publicIpAddress": "104.40.249.157",
  "resourceGroup": "RG_SH_IMAGEGALLERY",
  "zones": ""
}
This process will take several hours to complete.
  You can monitor installation progress using... ssh atiadmin@104.40.249.157.
  Once logged in, check the installation progress with: tail -f -n+1 /var/log/cloud-init-output.log
(py3) MAC-ATI0132:azure-vms moreilly$
martintoreilly commented 5 years ago

It looks like the missing " was introduced at commit d7b73f2

cloud-init-buildimage-ubuntu.yaml diff:

   - echo "*** Creating py27 environment... ***"
   - conda create -y --verbose --name py27 python=2.7 $PYTHONPACKAGES
   - echo "*** Creating py35 environment... ***"
-  - conda create -y --verbose --name py35 python=3.5 $PYTHONPACKAGES jupyterhub
+  - conda create -y --verbose --name py35 python=3.5 $PYTHONPACKAGES jupyterhub jupyterhub-ldapauthenticator
   - echo "*** Creating py36 environment... ***"
-  - conda create -y --verbose --name py36 python=3.6 $PYTHONPACKAGES jupyterhub
+  - conda create -y --verbose --name py36 python=3.6 $PYTHONPACKAGES jupyterhub jupyterhub-ldapauthenticator
   - echo "*** Cleanup... ***"
   - conda clean -a
   - echo "*** Final conda environments... ***"

cloud-init-buildimage-datascience.yaml diff:

   # Prepare environments
   - export PYTHONPACKAGES="ipykernel jupyter_client jupyterlab keras notebook pystan pytorch r-irkernel tensorflow torchvision"
   - echo "*** Working on py27 environment... ***"
-  - if [ "$(conda env list | grep py27)" = "" ]; then conda create -y -n py27 python=2.7 $PYTHONPACKAGES; else conda install -y -n py27 $PYTHONPACKAGES; fi
+  - if [ "$(conda env list | grep py27)" = "" ]; then conda create -y --verbose --name py27 python=2.7 $PYTHONPACKAGES; else conda install -y --verbose --name py27 $PYTHONPACKAGES; fi
   - echo "*** Working on py35 environment... ***"
-  - if [ "$(conda env list | grep py35)" = "" ]; then conda create -y -n py35 python=3.5 jupyterhub $PYTHONPACKAGES; else conda install -y -n py35 jupyterhub $PYTHONPACKAGES; fi
+  - if [ "$(conda env list | grep py35)" = "" ]; then conda create -y --verbose --name py35 python=3.5 jupyterhub jupyterhub-ldapauthenticator $PYTHONPACKAGES; else conda install -y --verbose --name py35 jupyterhub jupyterhub-ldapauthenticator $PYTHONPACKAGES; fi
   - echo "*** Working on py36 environment... ***"
-  - if [ "$(conda env list | grep py36)" = "" ]; then conda create -y -n py36 python=3.6 jupyterhub $PYTHONPACKAGES; else conda install -y -n py36 jupyterhub $PYTHONPACKAGES; fi
-  - echo "*** Final conda environments... ***"
+  - if [ "$(conda env list | grep py36)" = "" ]; then conda create -y --verbose --name py36 python=3.6 jupyterhub jupyterhub-ldapauthenticator $PYTHONPACKAGES; else conda install -y --verbose --name py36 jupyterhub jupyterhub-ldapauthenticator $PYTHONPACKAGES; fi
+  - echo "*** Final conda environments... **--verbose --name
   - conda env list
   - conda clean -a
martintoreilly commented 5 years ago

DataScience build record moved to PR #98