mounting bucket to datalab

[eilalan]

Hello,

I am working on genomic data and can not allow copying datasets to a VM before processing (very large datasets) while using datalab on Google Cloud. The GCSFuse is a very attractive solution, while mounting buckets to the VM.

Could you please advice what would be the right way to do mount bucket to datalab container on the cloud. I followed the GCSFuse installation and was able to install the gcsfuse package on the container. However, when the mount command is executed (many versions including the following): !gcsfuse -o allow_other --gid 0 --uid 0 --file-mode 777 --dir-mode 777 --key-file=/content/datalab/key.json archs4 /content/datalab/shared) The following output returns:

Using mount point: /content/datalab/shared
Opening GCS connection...
Opening bucket...
Mounting file system...
daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: running fusermount: exit status 1

stderr:
fusermount: fuse device not found, try 'modprobe fuse' first

I tried editing the startup-script docker run commands

1. added --priviledged,
2. added -v $(docker volume create --driver=gcsfuse --name=arch4):/shared None of the above worked. Please advice if there is any way to make it work.

Many thanks, Eila

[nikhilk]

We were discussing this yesterday -- definitely seems interesting to build in support for.

I am only guessing here, but maybe we need to install libfuse?

[eilalan]

I will look into the option of installing libfuse on the container. Could you please let me know what the error means:

daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: running fusermount: exit status 1

stderr:
fusermount: fuse device not found, try 'modprobe fuse' first

Thanks

[yebrahim]

The error is because the docker container doesn't have access to the fuse device. I managed to get another FUSE filesystem (for Google drive) mounted inside the container using the following flags: --security-opt apparmor:unconfined --cap-add mknod --cap-add sys_admin --device=/dev/fuse -v /mnt/drive:/mnt/gdrive:shared

Note the mount points there, which you might need to change.

[eilalan]

The datalab is mounting volumes to save the code for later use. Will it be possible to add mount with different device or should I use the $(docker volume create....) - like the second option on my first post? Many thanks, Eila

[eilalan]

Hi Yasser,

Could you please explain how it is possible that the container does not have access to gcsfuse after i installed in when i am logged in to the container.

I tried to work with the google drive - this is also a good solution for my issue. Thank you for the reference! After creating the folder for the mounting, chmod for the mounting folder, and updting the docker run command with your advice (with / without device option) I was not able to see the mounted folder.

I have tried installing the google-drive-ocamlfuse on the container and was able to get to the point where the verification code is requested, could you please advice if this can be waived for the mounting command?

!google-drive-ocamlfuse -headless -label googleDrive -id XXXX.apps.googleusercontent.com -secret XXX

The output is a link for the verification code. since I am running through the datalab cell and I dont know of a way to provide the verification code. what would be the way to ignore the verification code for the mounting command? or any other advice? Many thank for any help to finalize that issue Best, Eila

[harmon]

We JUST did this. Here's how we did it:

Create a custom run.sh script that calls the existing /datalab/run.sh. This will be your new Dockerfile entry point:

run-custom.sh

#!/bin/bash -e

# Here's where your bucket filesystem will be mounted:
mkdir -p /content/datalab/bucket-share
# This mounts the bucket using your credentials built into the Dockerfile:
gcsfuse --key-file /datalab/.gcloud-creds.json <YOUR_BUCKET_NAME_HERE> /content/datalab/bucket-share

# Run normal Google's datalab startup script:
/datalab/run.sh

New Dockerfile:

FROM gcr.io/cloud-datalab/datalab:latest

RUN apt-get update
RUN apt-get install -y lsb-release
RUN export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
RUN bash -c 'echo "deb http://packages.cloud.google.com/apt gcsfuse-`lsb_release -c -s` main" | tee /etc/apt/sources.list.d/gcsfuse.list'
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
RUN apt-get update
RUN apt-get install -y gcsfuse

# This is the service account JSON creds you create in IAM that has 
# access to read/write the Google Bucket you want to mount.
# You'll need to generate them and download them to 
# a "gcloud-creds.json" file next to this Dockerfile.
COPY gcloud-creds.json /datalab/.gcloud-creds.json

COPY run-custom.sh /datalab/run-custom.sh
ENTRYPOINT [ "/datalab/run-custom.sh" ]

Then you push this to your private GCR repo. Then:

datalab create --image-name <PATH_TO_YOUR_PRIVATE_REPO> instance-name

LASTLY, we are using a modified version of the "datalab create" python file. We had to add to the docker run command here:

https://github.com/googledatalab/datalab/blob/master/tools/cli/commands/create.py#L265-L279

    ExecStart=/usr/bin/docker run --rm -u 0 \
       --name=datalab \
+      --cap-add SYS_ADMIN --device=/dev/fuse \
       -p 127.0.0.1:8080:8080 \
       -v /mnt/disks/datalab-pd/content:/content \
       -v /mnt/disks/datalab-pd/tmp:/tmp \
       --env=HOME=/content \
       --env=DATALAB_ENV=GCE \
       --env=DATALAB_DEBUG=true \
       --env='DATALAB_SETTINGS_OVERRIDES={{ \
            "enableAutoGCSBackups": {1}, \
            "consoleLogLevel": "{2}" \
       }}' \
       --env='DATALAB_GIT_AUTHOR={3}' \
       --env='DATALAB_INITIAL_USER_SETTINGS={4}' \
       {0}

Then it works flawlessly! Maybe they can add a command-line option to datalab create to add this fuse option?

[eilalan]

Thank you! Looks perfect. I agree that it is not much to add and have it out of the box from google with all future compatibility issue supported. Many thanks again. will save me a lot of time.

[nikhilk]

This is good to see ... thanks for sharing @harmon. We should definitely look into making this more out-of-box.

[harmon]

Sure, no problem. We are enjoying Datalab, so I thought I'd give back a bit :)

[eilalan]

Hi all / harmon,

Could you please let me know where should i look for create.py file to edit with the additional docker run. ExecStart=/usr/bin/docker run --rm -u ....

Thanks, Eila

[chmeyers]

https://github.com/googledatalab/datalab/blob/master/tools/cli/commands/create.py

[eilalan]

thanks. that means that there is something basic that I dont know. How do I take the updated file and run it from my cloud shell?

[chmeyers]

One way, if you are okay with actually editing the file in Cloud Shell with vi: git clone https://github.com/googledatalab/datalab.git cd datalab/tools/cli vi commands/create.py python datalaby.py create --args_and_stuff

And actually, Cloud Shell has a Code Editor GUI now so you could use that instead of vi.

[harmon]

@eilalan Alternatively, you could edit the gcloud CLI's version, but that might get overridden when you update datalab next, and you'd have to remove the .pyc files, too.

$ gcloud info
...
Installation Root: [/opt/google-cloud-sdk]  <=== This is the folder to look for
...

$ vi /opt/google-cloud-sdk/platform/datalab-cli/create.py

But I like @chmeyers suggestion more, since you don't mess with your actual datalab CLI.

[eilalan]

Thank you both. The create stage went with no issues. I have updated the datalab create.py locally based on @chmeyers response and used the default (installation) for the connection. The connect command (as part of the updated create command or using datalab connect (default version)) is hanging. I have gather information that might be able to help you with suggestions:

• The command gcloud compute ssh --zone=us-east1-c datalab@datalab-gcsfuse2 worked fine.

• Running datalab connect datalab-gcsfuse2 --ssh-log-level=debug output the following error: channel 1: open failed: connect failed: Connection refused debug1: channel 1: free: direct-tcpip: listening port 8081 for localhost port 8080, connect from 127.0.0.1 port 42885 to 127.0.0.1 port 8081, nchannels 2 debug1: Connection to port 8081 forwarding to localhost port 8080 requested.

• older datalabs connect command is working fine. Any advice what might be the reason?

[eilalan]

I don't think that the issue is within the gcsfuse addition.

Following is a simple example with minimal dockerfile, using the standard datalab create command. Identical connection issue occurs:

The steps are: Running on MAC machine: Docker file:

FROM gcr.io/cloud-datalab/datalab:latest

RUN apt-get update

ENTRYPOINT [ "/datalab/run.sh" ]

build container using the command (only Dockerfile in folder) gcloud container builds submit --tag gcr.io/orielresearch-188115/datalab_orig . Running on google cloud shell: create a datalab contianer: datalab create --image-name datalab_orig datalab-test

The execution is stuck at: Waiting for Datalab to be reachable at http://localhost:8081/

Please let me know if you have any idea how to debug / what is wrong with the above process.

Many thanks, Eila

[eilalan]

I will open a new thread for the datalab connect issue as it is not a gcsfuse setting issue.

[eilalan]

The mounting is working from a local docker on cloud shell - BUT not from the gcr.io I used harmon code with the datalab-extended option to create the docker and pushed it to gcr.io.

When a new datalab instance is created with the new image, the ssh is failing. I looked at the log of the /var/log/startupscript.log and it seems that the pull passed with no issues. The output of running the create with --verbosity debug switch is the follwing:

DEBUG: Executing command: ['/usr/bin/ssh', '-t', '-i', '/home/eila/.ssh/google_compute_engine', '-o', 'CheckHostIP=no', '-o', 'HostKeyAlias=compute.8506066424242652437', '-o', 'IdentitiesOnly=yes', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/home/eila/.ssh/google_compute_known_hosts', u'-o', u'LogLevel=error', u'-4', u'-N', u'-L', u'localhost:8081:localhost:8080', u'datalab@35.185.255.5']
Permission denied (publickey).
DEBUG: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Traceback (most recent call last):
  File "/google/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 797, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/google/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 757, in Run
    resources = command_instance.Run(args)
  File "/google/google-cloud-sdk/lib/surface/compute/ssh.py", line 195, in Run
    return_code = cmd.Run(ssh_helper.env, force_connect=True)
  File "/google/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/ssh/ssh.py", line 934, in Run
    raise CommandError(args[0], return_code=status)
CommandError: [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Connection broken
Attempting to reconnect...
Connecting to d4 via SSH

Is there other permission that I need to provide to support the the volume mounting addition? I have given -compute@developer.gserviceaccount.com Editor permission so maybe another entity that requires permission or a different issue. any suggestions?

[harmon]

Can you show what commands you were running and what the messages/errors are? Just because it can't connect isn't enough info to see why the original command failed. Any clues in "startupscript.log"?

[eilalan]

Ofcourse. The steps are the following:

1. created a folder with the following files: run-extended.sh

   
   CONTENT=$HOME
   DOCKERIMAGE=datalab
   docker-credential-gcr configure-docker
   #ENTRYPOINT="/datalab/run.sh"
   ENTRYPOINT="/datalab/run-custom.sh"
   if [ "$1" != "" ]; then
   if [ "$1" != "shell" ]; then
   CONTENT=$1
   shift
   fi
   if [ "$1" == "shell" ]; then
   ENTRYPOINT="/bin/bash"
   fi
   fi
   if [ -f custom-packages.txt ] || [ -f Dockerfile-extended.in ];
   then
   if [ -f custom-packages.txt ];
   then
       # First create a new Docker file called Dockerfile-custom-packages. Start with the standard image
       # TODO: at some point the local Datalab container will be tagged 'latest' rather than 'local'
       # and the line below should change.
       #echo 'FROM gcr.io/cloud-datalab/datalab:latest' > Dockerfile-custom-packages
       echo "not implemented"
       # Add the script with a list of custom packages to the Dockerfile
       #echo 'ADD custom-packages.txt /datalab/custom-packages.txt' >> Dockerfile-custom-packages
       #echo 'RUN pip install -r /datalab/custom-packages.txt' >> Dockerfile-custom-packages
   
       #DOCKERFILE=Dockerfile-custom-packages
       #DOCKERIMAGE=datalab-custom-packages
   
   elif [ -f Dockerfile-extended.in ];
   then
       DOCKERFILE=Dockerfile-extended.in
       DOCKERIMAGE=datalab-extended
   fi
   
   # Build the customized docker image derived from the standard datalab image
   docker build ${DOCKER_BUILD_ARGS} -t $DOCKERIMAGE -f $DOCKERFILE .
   fi
   if [ "$OSTYPE" == "linux"* ]; then
   PORTMAP="127.0.0.1:8081:8080"
   else
   PORTMAP="8081:8080"
   fi
   # Use this flag to map in web server content during development
   #  -v $REPO_DIR/sources/web:/sources \
   /usr/bin/docker run -it --cap-add SYS_ADMIN --device=/dev/fuse --entrypoint=$ENTRYPOINT \
   -p $PORTMAP \
   -v "$CONTENT/datalab:/content/datalab" \
   -e "PROJECT_ID=$PROJECT_ID" \
   -e "DATALAB_ENV=GCE" \
   -e "EXPERIMENTAL_KERNEL_GATEWAY_URL=${EXPERIMENTAL_KERNEL_GATEWAY_URL}" \
   $DOCKERIMAGE

2. run-custom.sh  (+ chmod 777 run-custom.sh to allow execution)

!/bin/bash -e

Here's where your bucket filesystem will be mounted:

mkdir -p /content/datalab/bucket-share

This mounts the bucket using your credentials built into the Dockerfile:

gcsfuse --key-file /datalab/.gcloud-creds.json <YOUR BUCKET - only the last name - no full address> /content/datalab/bucket-share

Run normal Google's datalab startup script:

/datalab/run.sh

3. Dockerfile-extended.in

Use this file to create a new docker image derived from the standard datalab image

FROM gcr.io/cloud-datalab/datalab:latest MAINTAINER Your Name RUN apt-get update RUN apt-get install -y lsb-release RUN export GCSFUSE_REPO=gcsfuse-lsb_release -c -s RUN bash -c 'echo "deb http://packages.cloud.google.com/apt gcsfuse-lsb_release -c -s main" | tee /etc/apt/sources.list.d/gcsfuse.list' RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - RUN apt-get update RUN apt-get install -y gcsfuse

This is the service account JSON creds you create in IAM that has

access to read/write the Google Bucket you want to mount.

You'll need to generate them and download them to

a "gcloud-creds.json" file next to this Dockerfile.

COPY .json /datalab/.gcloud-creds.json

COPY run-custom.sh /datalab/run-custom.sh ENTRYPOINT [ "/datalab/run-extended.sh" ]

4. Last step:

$ ./run-extended.sh

This will build and run the image on the cloud shell. 

**In the container, I run:**

$df -h

and was able to see the volume

**For Google gcr.io, on cloud shell:**

$docker tag datalab-extended us.gcr.io/orielresearch-188115/datalab-extended $gcloud -- push us.gcr.io/orielresearch-188115/datalab-extended

Execution from the cloud shell with image from gcr.io:

$ datalab create --verbosity debug --image-name us.gcr.io/orielresearch-188115/datalab-extended --zone us-west1-c d5 Also tried the updated version: $ python datalab.py create --image-name us.gcr.io/orielresearch-188115/datalab-extended d2

The error message is (looks like it is log in from my gmail account - owner of the project)

Connecting to d5. This will create an SSH tunnel and may prompt you to create an rsa key pair. To manage these keys, see https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys Connecting to d5 via SSH Waiting for Datalab to be reachable at http://localhost:8081/ DEBUG: Running [gcloud.compute.ssh] with arguments: [--ssh-flag: "['-o', 'LogLevel=error', '-4', '-N', '-L', 'localhost:8081:localhost:8080']", --verbosity: "debug", --zone: "us-west1-c", [USER@]INSTANCE: "datalab@d5"] DEBUG: Current SSH keys in project: [u'eila:ssh-rsa AAAA6tNiIXCrkO+JXqArCPTxBpNl5yFaCKhiZznyI5R7geaBOaQnpJORkwGzs= google-ssh {"userName":"eila@orielresearch.org","expireOn":"2018-01-31T17:31:53+0000"}', u'eila:ssh-rsa AAAAB3NzaC1yc2EAAAADA

The startup log end is:

Jan 31 19:50:48 d5 startup-script[769]: INFO startup-script: dmesg | tail or so. Jan 31 19:50:48 d5 startup-script[769]: INFO startup-script: Checking if the persistent disk needs to be formatted Jan 31 19:50:48 d5 startup-script[769]: INFO startup-script: Formatting the persistent disk Jan 31 19:50:49 d5 startup-script[769]: INFO startup-script: mke2fs 1.43.6 (29-Aug-2017) Jan 31 19:50:49 d5 startup-script[769]: [119B blob data] Jan 31 19:50:49 d5 startup-script[769]: INFO startup-script: Creating filesystem with 52428800 4k blocks and 13107200 inodes Jan 31 19:50:49 d5 startup-script[769]: INFO startup-script: Filesystem UUID: 1e1c8db9-2385-4385-bf27-4ce94f0e50bc Jan 31 19:50:49 d5 startup-script[769]: INFO startup-script: Superblock backups stored on blocks: Jan 31 19:50:49 d5 startup-script[769]: INFO startup-script: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, Jan 31 19:50:49 d5 startup-script[769]: INFO startup-script: 4096000, 7962624, 11239424, 20480000, 23887872 Jan 31 19:50:49 d5 startup-script[769]: [86B blob data] Jan 31 19:51:13 d5 startup-script[769]: [515B blob data] Jan 31 19:51:22 d5 startup-script[769]: INFO startup-script: Creating journal (262144 blocks): done Jan 31 19:51:27 d5 startup-script[769]: [156B blob data] Jan 31 19:51:27 d5 startup-script[769]: INFO startup-script: Creating the datalab directory Jan 31 19:51:27 d5 startup-script[769]: INFO startup-script: Cloning the repo datalab-notebooks Jan 31 19:51:34 d5 startup-script[769]: INFO startup-script: Cloning into '/content/datalab/notebooks'... Jan 31 19:51:38 d5 startup-script[769]: INFO startup-script: Project [orielresearch-188115] repository [datalab-notebooks] was cloned to [/content/datalab/notebooks]. Jan 31 19:51:39 d5 startup-script[769]: INFO startup-script: Creating /mnt/disks/datalab-pd/content/datalab Jan 31 19:51:39 d5 startup-script[769]: INFO startup-script: Creating a 3794360 kilobyte swapfile at /mnt/disks/datalab-pd/swapfile Jan 31 19:52:06 d5 startup-script[769]: INFO startup-script: 3794360+0 records in Jan 31 19:52:06 d5 startup-script[769]: INFO startup-script: 3794360+0 records out Jan 31 19:52:06 d5 startup-script[769]: INFO startup-script: 3885424640 bytes (3.9 GB, 3.6 GiB) copied, 27.6472 s, 141 MB/s Jan 31 19:52:11 d5 startup-script[769]: INFO startup-script: Setting up swapspace version 1, size = 3.6 GiB (3885420544 bytes) Jan 31 19:52:11 d5 startup-script[769]: INFO startup-script: no label, UUID=e42fb2cd-3e13-4738-943a-9b269200ffbc Jan 31 19:52:11 d5 startup-script[769]: INFO startup-script: sysctl: cannot stat /proc/sys/vm/disk_based_swap: No such file or directory

err at the startup script:
```$cat /var/log/startupscript.log | grep -i err
Jan 31 19:50:48 d5 startup-script[769]: INFO startup-script:        missing codepage or helper program, or other error

Thanks, Eila

[harmon]

I think there might be some confusion. Your run-extended.sh is being run on your development laptop/desktop, right? That will build a new custom Docker image, which you are deploying. The "docker run" at the end of run-extended.sh is only there for you to test out the image locally, so any options to docker run in run-extended.sh will NOT end up on your Datalab instance in Google cloud. You added the --cap-add SYS_ADMIN --device=/dev/fuse options there, and that is the wrong place to do it. That option needs to be added here: https://github.com/googledatalab/datalab/blob/master/tools/cli/commands/create.py#L268

As chmeyers mentioned in the other ticket, you can clone https://github.com/googledatalab/datalab and update this file, and then run the datalab.py out of this git repo.

The main idea here is that create.py will generate the startup-script that is given to the VM when datalab create is run. In order to modify what docker run options are used in Datalab, you need to change that startup-script template to modify the datalab.service ExecStart options.

Also, in Dockerfile-extended.in, you'll want to change the ENTRYPOINT to ENTRYPOINT [ "/datalab/run-custom.sh" ].

Give that a shot, and see what the errors are.

[eilalan]

Thank you for your comments The ./run-extended.ini was running from the Google cloud shell and worked fine. For the cloud shell execution, If I take off the switch, ./run-extended will fail with the following message:

Mounting file system...
daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: running fusermount: exit status 1
stderr:
fusermount: fuse device not found, try 'modprobe fuse' first

For the datalab image (thank you), I have commented out the the run command from run-extended.sh, executed the run-extended.sh to create a new docker build, tagged it for google and pushed it to the gcr.io. when I execute the following command: $python datalab.py create --image-name datalab-extended d11 (i sed the git option - chmeyers) the ssh connection is stack:

Waiting for Datalab to be reachable at http://localhost:8081/
Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Connection broken
Attempting to reconnect...
Waiting for Datalab to be reachable at http://localhost:8081/

and nothing on the startuplog

cat /var/log/startupscript.log | grep -i err
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script:        missing codepage or helper program, or other error

$ tail /var/log/startupscript.log 
Jan 31 22:53:12 d12 startup-script[766]: INFO startup-script: Cloning into '/content/datalab/notebooks'...
Jan 31 22:53:17 d12 startup-script[766]: INFO startup-script: Project [orielresearch-188115] repository [datalab-notebooks] was cloned to [/content/datalab/notebooks].
Jan 31 22:53:18 d12 startup-script[766]: INFO startup-script: Creating /mnt/disks/datalab-pd/content/datalab
Jan 31 22:53:18 d12 startup-script[766]: INFO startup-script: Creating a 3794360 kilobyte swapfile at /mnt/disks/datalab-pd/swapfile
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3794360+0 records in
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3794360+0 records out
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3885424640 bytes (3.9 GB, 3.6 GiB) copied, 27.8909 s, 139 MB/s
Jan 31 22:53:50 d12 startup-script[766]: INFO startup-script: Setting up swapspace version 1, size = 3.6 GiB (3885420544 bytes)
Jan 31 22:53:50 d12 startup-script[766]: INFO startup-script: no label, UUID=bdc96f7f-fd20-4940-829a-718ec6215ede
Jan 31 22:53:50 d12 startup-script[766]: INFO startup-script: sysctl: cannot stat /proc/sys/vm/disk_based_swap: No such file or directory

any idea what went wrong? Thanks, Eila

(the ENTRYPOINT and the create.py were updated before I published the message but after I wrote it).

[eilalan]

Just wanted to add that similar results when using harmon code based on the datalab:latest image and run-custom.sh:

$ python datalab.py create --verbosity debug --image-name us.gcr.io/orielresearch-188115/datalab-harmon --zone us-w
est1-c h3

Connecting to h3 via SSH
Waiting for Datalab to be reachable at http://localhost:8081/
DEBUG: Running [gcloud.compute.ssh] with arguments: [--ssh-flag: "['-o', 'LogLevel=error', '-4', '-N', '-L', 'localhost:8081:localhost:8080']", --verbosity: "debug", --zone: "us-west1-c", [USER@]INSTANCE: "datalab@h3"]
DEBUG: Current SSH keys in project: [u'eila:ssh-rsa KzWvr1PfixTtfMyJQ4gWBAf89tEH4ZbdtSLmDM= google-ssh {"userName":"eila@orielresearch.org","expireOn":"2018-01-31T22:54:30+0000"}', u'eila:ecdsa-sha2-nistp256 AAAAE2VmJtFk= google-ssh {"userName":"eila@orielresearch.org","expireOn":"2018-01-31T22:54:29+0000"}', u'datalab:ssh-rsa AAAAB3NzaC12ax eila@cs-6000-devshell-vm-71f7e8c0-ea8d-4ae5-beff-db59e9c0d1e8', u'eila:ssh-rsa pd3L12ax eila@cs-6000-devshell-vm-71f7e8c0-ea8d-4ae5-beff-db59e9c0d1e8']
DEBUG: Running command [/usr/bin/ssh -t -i /home/eila/.ssh/google_compute_engine -o CheckHostIP=no -o HostKeyAlias=compute.9069969224748704285 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/home/eila/.ssh/google_compute_known_hosts -o LogLevel=error -4 -N -L localhost:8081:localhost:8080 datalab@35.197.116.121].
DEBUG: Executing command: ['/usr/bin/ssh', '-t', '-i', '/home/eila/.ssh/google_compute_engine', '-o', 'CheckHostIP=no', '-o', 'HostKeyAlias=compute.9069969224748704285', '-o', 'IdentitiesOnly=yes', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/home/eila/.ssh/google_compute_known_hosts', u'-o', u'LogLevel=error', u'-4', u'-N', u'-L', u'localhost:8081:localhost:8080', u'datalab@35.197.116.121']

strartupscript.log:

eila@h2 ~ $ tail /var/log/startupscript.log 
Feb 01 04:42:57 h2 startup-script[735]: INFO startup-script: Cloning into '/content/datalab/notebooks'...
Feb 01 04:43:00 h2 startup-script[735]: INFO startup-script: Project [orielresearch-188115] repository [datalab-notebooks] was cloned to [/content/datalab/notebooks].
Feb 01 04:43:01 h2 startup-script[735]: INFO startup-script: Creating /mnt/disks/datalab-pd/content/datalab
Feb 01 04:43:01 h2 startup-script[735]: INFO startup-script: Creating a 3794360 kilobyte swapfile at /mnt/disks/datalab-pd/swapfile
Feb 01 04:43:29 h2 startup-script[735]: INFO startup-script: 3794360+0 records in
Feb 01 04:43:29 h2 startup-script[735]: INFO startup-script: 3794360+0 records out
Feb 01 04:43:29 h2 startup-script[735]: INFO startup-script: 3885424640 bytes (3.9 GB, 3.6 GiB) copied, 27.5925 s, 141 MB/s
Feb 01 04:43:33 h2 startup-script[735]: INFO startup-script: Setting up swapspace version 1, size = 3.6 GiB (3885420544 bytes)
Feb 01 04:43:33 h2 startup-script[735]: INFO startup-script: no label, UUID=01a6ffb2-23fc-4e62-bb9c-8c20faf67c38
Feb 01 04:43:33 h2 startup-script[735]: INFO startup-script: sysctl: cannot stat /proc/sys/vm/disk_based_swap: No such file or directory

Search for error:

$ cat /var/log/startupscript.log  | grep -i err
Feb 01 04:42:11 h2 startup-script[735]: INFO startup-script:        missing codepage or helper program, or other 
error

[harmon]

Things I would do next:

gcloud compute ssh d11 (or whatever your vm name is)

If that works, it's not an SSH key issue.

While inside "d11", run:

$ docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS                      NAMES
4f2ca44e7f0c        gcr.io/google_containers/fluentd-gcp:1.18   "/bin/sh -c '/usr/..."   25 minutes ago      Up 25 minutes                                  logger
4b00ea86303f        gcr.io/my-project/datalab:latest           "/datalab/run-gs-c..."   25 minutes ago      Up 25 minutes       127.0.0.1:8080->8080/tcp   datalab

If you don't see the datalab container running, then it's a docker startup issue.

Can you paste your entire /var/log/startupscript.log here? Just the "tail" of it doesn't show us the whole story. You can redact the parts of it you don't want to share (ie: names of things).

[eilalan]

Hi Harmon, $docker ps is empty

$ docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS      
         NAMES
4419c70880dd        gcr.io/google_containers/fluentd-gcp:1.18   "/bin/sh -c '/usr/..."   3 minutes ago       Up 3 minutes                   
         logger

The full startupscript.log is the following:

Jan 31 22:49:32 d12 systemd[1]: Starting Google Compute Engine Startup Scripts...
Jan 31 22:49:33 d12 startup-script[766]: INFO Starting startup scripts.
Jan 31 22:49:33 d12 startup-script[766]: INFO Found startup-script in metadata.
Jan 31 22:49:33 d12 startup-script[766]: INFO startup-script: Getting Docker credentials
Jan 31 22:49:35 d12 startup-script[766]: INFO startup-script: /home/datalab/.docker/config.json configured to use this credential helper for
 GCR registries
Jan 31 22:49:35 d12 startup-script[766]: INFO startup-script: Pulling latest image: us.gcr.io/orielresearch-188115/datalab-extended
Jan 31 22:49:35 d12 startup-script[766]: INFO startup-script: Using default tag: latest
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: latest: Pulling from orielresearch-188115/datalab-extended
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 50aff78429b1: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: f6d82e297bce: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 275abb2c8a6f: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 9f15a39356d6: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: fc0342a94c89: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 1824f6f29083: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 3f3f1a69ca86: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 1d1f2d49e206: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 21c4e23d6c4c: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 6ae6a8955b73: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 90aa7545463a: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: a782d06a090f: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: d8b79fa1a2bb: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 6905e02c791c: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: d307bdff2ac9: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 511519af0a27: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 506f5871a856: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 425ebfbe2cb1: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 2707aac28f3f: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 9be51f250e02: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 3a82d871d4f8: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: b47b9d248caa: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 47e4beefb359: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 0de422820434: Pulling fs layer
--More--
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: cd5b0928574d: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 10bdbb64f802: Pulling fs layer
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 9f15a39356d6: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: fc0342a94c89: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 1824f6f29083: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 3f3f1a69ca86: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 1d1f2d49e206: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 21c4e23d6c4c: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 6ae6a8955b73: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 90aa7545463a: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: a782d06a090f: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: d8b79fa1a2bb: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 6905e02c791c: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: d307bdff2ac9: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 511519af0a27: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 506f5871a856: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 425ebfbe2cb1: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 2707aac28f3f: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 9be51f250e02: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 3a82d871d4f8: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: b47b9d248caa: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 47e4beefb359: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 0de422820434: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: cd5b0928574d: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 10bdbb64f802: Waiting
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 275abb2c8a6f: Verifying Checksum
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 275abb2c8a6f: Download complete
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: f6d82e297bce: Verifying Checksum
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: f6d82e297bce: Download complete
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 9f15a39356d6: Verifying Checksum
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: 9f15a39356d6: Download complete
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: fc0342a94c89: Verifying Checksum
Jan 31 22:49:37 d12 startup-script[766]: INFO startup-script: fc0342a94c89: Download complete
Jan 31 22:49:38 d12 startup-script[766]: INFO startup-script: 3f3f1a69ca86: Verifying Checksum
--More--
Jan 31 22:49:38 d12 startup-script[766]: INFO startup-script: 3f3f1a69ca86: Download complete
Jan 31 22:49:38 d12 startup-script[766]: INFO startup-script: 50aff78429b1: Verifying Checksum
Jan 31 22:49:38 d12 startup-script[766]: INFO startup-script: 50aff78429b1: Download complete
Jan 31 22:49:38 d12 startup-script[766]: INFO startup-script: 1d1f2d49e206: Verifying Checksum
Jan 31 22:49:38 d12 startup-script[766]: INFO startup-script: 1d1f2d49e206: Download complete
Jan 31 22:49:41 d12 startup-script[766]: INFO startup-script: 21c4e23d6c4c: Verifying Checksum
Jan 31 22:49:41 d12 startup-script[766]: INFO startup-script: 21c4e23d6c4c: Download complete
Jan 31 22:49:41 d12 startup-script[766]: INFO startup-script: 6ae6a8955b73: Verifying Checksum
Jan 31 22:49:41 d12 startup-script[766]: INFO startup-script: 6ae6a8955b73: Download complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: 50aff78429b1: Pull complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: f6d82e297bce: Pull complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: 90aa7545463a: Verifying Checksum
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: 90aa7545463a: Download complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: 275abb2c8a6f: Pull complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: 9f15a39356d6: Pull complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: fc0342a94c89: Pull complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: a782d06a090f: Verifying Checksum
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: a782d06a090f: Download complete
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: d8b79fa1a2bb: Verifying Checksum
Jan 31 22:49:43 d12 startup-script[766]: INFO startup-script: d8b79fa1a2bb: Download complete
Jan 31 22:49:44 d12 startup-script[766]: INFO startup-script: d307bdff2ac9: Verifying Checksum
Jan 31 22:49:44 d12 startup-script[766]: INFO startup-script: d307bdff2ac9: Download complete
Jan 31 22:49:45 d12 startup-script[766]: INFO startup-script: 6905e02c791c: Verifying Checksum
Jan 31 22:49:45 d12 startup-script[766]: INFO startup-script: 6905e02c791c: Download complete
Jan 31 22:49:45 d12 startup-script[766]: INFO startup-script: 511519af0a27: Verifying Checksum
Jan 31 22:49:45 d12 startup-script[766]: INFO startup-script: 511519af0a27: Download complete
Jan 31 22:49:46 d12 startup-script[766]: INFO startup-script: 506f5871a856: Verifying Checksum
Jan 31 22:49:46 d12 startup-script[766]: INFO startup-script: 506f5871a856: Download complete
Jan 31 22:49:47 d12 startup-script[766]: INFO startup-script: 425ebfbe2cb1: Verifying Checksum
Jan 31 22:49:47 d12 startup-script[766]: INFO startup-script: 425ebfbe2cb1: Download complete
Jan 31 22:49:47 d12 startup-script[766]: INFO startup-script: 2707aac28f3f: Verifying Checksum
Jan 31 22:49:47 d12 startup-script[766]: INFO startup-script: 2707aac28f3f: Download complete
Jan 31 22:49:48 d12 startup-script[766]: INFO startup-script: 9be51f250e02: Verifying Checksum
Jan 31 22:49:48 d12 startup-script[766]: INFO startup-script: 9be51f250e02: Download complete
--More--
Jan 31 22:49:48 d12 startup-script[766]: INFO startup-script: 3a82d871d4f8: Verifying Checksum
Jan 31 22:49:48 d12 startup-script[766]: INFO startup-script: 3a82d871d4f8: Download complete
Jan 31 22:49:48 d12 startup-script[766]: INFO startup-script: b47b9d248caa: Verifying Checksum
Jan 31 22:49:48 d12 startup-script[766]: INFO startup-script: b47b9d248caa: Download complete
Jan 31 22:49:49 d12 startup-script[766]: INFO startup-script: 47e4beefb359: Verifying Checksum
Jan 31 22:49:49 d12 startup-script[766]: INFO startup-script: 47e4beefb359: Download complete
Jan 31 22:49:50 d12 startup-script[766]: INFO startup-script: 0de422820434: Verifying Checksum
Jan 31 22:49:50 d12 startup-script[766]: INFO startup-script: 0de422820434: Download complete
Jan 31 22:49:50 d12 startup-script[766]: INFO startup-script: cd5b0928574d: Verifying Checksum
Jan 31 22:49:50 d12 startup-script[766]: INFO startup-script: cd5b0928574d: Download complete
Jan 31 22:49:50 d12 startup-script[766]: INFO startup-script: 10bdbb64f802: Verifying Checksum
Jan 31 22:49:50 d12 startup-script[766]: INFO startup-script: 10bdbb64f802: Download complete
Jan 31 22:49:53 d12 startup-script[766]: INFO startup-script: 1824f6f29083: Verifying Checksum
Jan 31 22:49:53 d12 startup-script[766]: INFO startup-script: 1824f6f29083: Download complete
Jan 31 22:52:02 d12 startup-script[766]: INFO startup-script: 1824f6f29083: Pull complete
Jan 31 22:52:03 d12 startup-script[766]: INFO startup-script: 3f3f1a69ca86: Pull complete
Jan 31 22:52:03 d12 startup-script[766]: INFO startup-script: 1d1f2d49e206: Pull complete
Jan 31 22:52:03 d12 startup-script[766]: INFO startup-script: 21c4e23d6c4c: Pull complete
Jan 31 22:52:03 d12 startup-script[766]: INFO startup-script: 6ae6a8955b73: Pull complete
Jan 31 22:52:03 d12 startup-script[766]: INFO startup-script: 90aa7545463a: Pull complete
Jan 31 22:52:05 d12 startup-script[766]: INFO startup-script: a782d06a090f: Pull complete
Jan 31 22:52:05 d12 startup-script[766]: INFO startup-script: d8b79fa1a2bb: Pull complete
Jan 31 22:52:08 d12 startup-script[766]: INFO startup-script: 6905e02c791c: Pull complete
Jan 31 22:52:08 d12 startup-script[766]: INFO startup-script: d307bdff2ac9: Pull complete
Jan 31 22:52:08 d12 startup-script[766]: INFO startup-script: 511519af0a27: Pull complete
Jan 31 22:52:20 d12 startup-script[766]: INFO startup-script: 506f5871a856: Pull complete
Jan 31 22:52:23 d12 startup-script[766]: INFO startup-script: 425ebfbe2cb1: Pull complete
Jan 31 22:52:23 d12 startup-script[766]: INFO startup-script: 2707aac28f3f: Pull complete
Jan 31 22:52:23 d12 startup-script[766]: INFO startup-script: 9be51f250e02: Pull complete
Jan 31 22:52:24 d12 startup-script[766]: INFO startup-script: 3a82d871d4f8: Pull complete
Jan 31 22:52:24 d12 startup-script[766]: INFO startup-script: b47b9d248caa: Pull complete
Jan 31 22:52:24 d12 startup-script[766]: INFO startup-script: 47e4beefb359: Pull complete
Jan 31 22:52:24 d12 startup-script[766]: INFO startup-script: 0de422820434: Pull complete
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: cd5b0928574d: Pull complete
--More--
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: 10bdbb64f802: Pull complete
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Digest: sha256:2ca380b9
8
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Status: Downloaded newer image for us.gcr.io/orielresearch-188115/datalab-exte
nded:latest
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Trying to mount the persistent disk
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: mount: wrong fs type, bad option, bad superblock on /dev/sdb,
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script:        missing codepage or helper program, or other error
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script:        In some cases useful info is found in syslog - try
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script:        dmesg | tail or so.
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Checking if the persistent disk needs to be formatted
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Formatting the persistent disk
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: mke2fs 1.43.6 (29-Aug-2017)
Jan 31 22:52:25 d12 startup-script[766]: [119B blob data]
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Creating filesystem with 52428800 4k blocks and 13107200 inodes
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Filesystem UUID: e39bb6a5-ba52-407f-a9b9-4599ba96024c
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script: Superblock backups stored on blocks:
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script:         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208
,
Jan 31 22:52:25 d12 startup-script[766]: INFO startup-script:         4096000, 7962624, 11239424, 20480000, 23887872
Jan 31 22:52:25 d12 startup-script[766]: [86B blob data]
Jan 31 22:52:50 d12 startup-script[766]: [533B blob data]
Jan 31 22:52:58 d12 startup-script[766]: INFO startup-script: Creating journal (262144 blocks): done
Jan 31 22:53:03 d12 startup-script[766]: [156B blob data]
Jan 31 22:53:03 d12 startup-script[766]: INFO startup-script: Creating the datalab directory
Jan 31 22:53:03 d12 startup-script[766]: INFO startup-script: Cloning the repo datalab-notebooks
Jan 31 22:53:12 d12 startup-script[766]: INFO startup-script: Cloning into '/content/datalab/notebooks'...
Jan 31 22:53:17 d12 startup-script[766]: INFO startup-script: Project [orielresearch-188115] repository [datalab-notebooks] was cloned to [/
content/datalab/notebooks].
Jan 31 22:53:18 d12 startup-script[766]: INFO startup-script: Creating /mnt/disks/datalab-pd/content/datalab
Jan 31 22:53:18 d12 startup-script[766]: INFO startup-script: Creating a 3794360 kilobyte swapfile at /mnt/disks/datalab-pd/swapfile
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3794360+0 records in
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3794360+0 records out
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3885424640 bytes (3.9 GB, 3.6 GiB) copied, 27.8909 s, 139 MB/s
--More--
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3794360+0 records in
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3794360+0 records out
Jan 31 22:53:46 d12 startup-script[766]: INFO startup-script: 3885424640 bytes (3.9 GB, 3.6 GiB) copied, 27.8909 s, 139 MB/s
Jan 31 22:53:50 d12 startup-script[766]: INFO startup-script: Setting up swapspace version 1, size = 3.6 GiB (3885420544 bytes)
Jan 31 22:53:50 d12 startup-script[766]: INFO startup-script: no label, UUID=bdc96f7f-fd20-4940-829a-718ec6215ede
Jan 31 22:53:50 d12 startup-script[766]: INFO startup-script: sysctl: cannot stat /proc/sys/vm/disk_based_swap: No such file or directory
Jan 31 22:53:51 d12 startup-script[766]: INFO startup-script: Return code 0.
Jan 31 22:53:51 d12 startup-script[766]: INFO Finished running startup scripts.
Jan 31 22:53:52 d12 systemd[1]: Started Google Compute Engine Startup Scripts.
-- Reboot --
Feb 01 15:41:54 d12 systemd[1]: Starting Google Compute Engine Startup Scripts...
Feb 01 15:41:55 d12 startup-script[753]: INFO Starting startup scripts.
Feb 01 15:41:55 d12 startup-script[753]: INFO Found startup-script in metadata.
Feb 01 15:41:55 d12 startup-script[753]: INFO startup-script: Getting Docker credentials
Feb 01 15:41:59 d12 startup-script[753]: INFO startup-script: /home/datalab/.docker/config.json configured to use this credential helper for
 GCR registries
Feb 01 15:41:59 d12 startup-script[753]: INFO startup-script: Pulling latest image: us.gcr.io/orielresearch-188115/datalab-extended
Feb 01 15:41:59 d12 startup-script[753]: INFO startup-script: Using default tag: latest
Feb 01 15:41:59 d12 startup-script[753]: INFO startup-script: latest: Pulling from orielresearch-188115/datalab-extended
Feb 01 15:41:59 d12 startup-script[753]: INFO startup-script: Digest: sha256:2ca380b90b7e5d1228
8
Feb 01 15:41:59 d12 startup-script[753]: INFO startup-script: Status: Image is up to date for us.gcr.io/orielresearch-188115/datalab-extende
d:latest
Feb 01 15:41:59 d12 startup-script[753]: INFO startup-script: Trying to mount the persistent disk
Feb 01 15:42:00 d12 startup-script[753]: INFO startup-script: Creating /mnt/disks/datalab-pd/content/datalab
Feb 01 15:42:00 d12 startup-script[753]: INFO startup-script: mkswap: /mnt/disks/datalab-pd/swapfile: warning: wiping old swap signature.
Feb 01 15:42:00 d12 startup-script[753]: INFO startup-script: Setting up swapspace version 1, size = 3.6 GiB (3885420544 bytes)
Feb 01 15:42:00 d12 startup-script[753]: INFO startup-script: no label, UUID=21a91c32-5646-
Feb 01 15:42:00 d12 startup-script[753]: INFO startup-script: sysctl: cannot stat /proc/sys/vm/disk_based_swap: No such file or directory

Thanks, Eila

[harmon]

Ok, now we are getting somewhere! So it was able to download the image, that's good. So next, check the datalab logs for hints as to why it didn't start correctly:

$ sudo journalctl -u datalab

[eilalan]

I am glad that we are getting somewhere... The log is the following and has errors quite at the beginning

Jan 31 22:53:51 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:53:53 d12 docker-credential-gcr[1550]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:53:53 d12 systemd[1]: Started datalab docker container.
Jan 31 22:53:54 d12 docker[1575]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:53:54 d12 docker[1575]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:53:54 d12 docker[1575]: time="2018-01-31T22:53:54Z" level=error msg="error getting events from daemon: context canceled"
Jan 31 22:53:54 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:53:54 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:53:54 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:53:55 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:53:55 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:53:55 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:53:56 d12 docker-credential-gcr[1691]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:53:56 d12 systemd[1]: Started datalab docker container.
Jan 31 22:53:59 d12 docker[1700]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:53:59 d12 docker[1700]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:53:59 d12 docker[1700]: time="2018-01-31T22:53:59Z" level=error msg="error getting events from daemon: context canceled"
Jan 31 22:53:59 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:53:59 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:53:59 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:00 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:00 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:00 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:01 d12 docker-credential-gcr[1778]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:01 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:04 d12 docker[1786]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:04 d12 docker[1786]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:04 d12 docker[1786]: time="2018-01-31T22:54:04Z" level=error msg="error getting events from daemon: context canceled"
Jan 31 22:54:04 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:04 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:04 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:05 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:05 d12 systemd[1]: Stopped datalab docker container.
jan 31 22:54:05 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:05 d12 docker-credential-gcr[1864]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:05 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:06 d12 docker[1872]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:07 d12 docker[1872]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:07 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:07 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:07 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:08 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:08 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:08 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:08 d12 docker-credential-gcr[1980]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:08 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:11 d12 docker[1988]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:11 d12 docker[1988]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:11 d12 docker[1988]: time="2018-01-31T22:54:11Z" level=error msg="error getting events from daemon: net/http: request canceled"
Jan 31 22:54:11 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:11 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:11 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:12 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:12 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:12 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:12 d12 docker-credential-gcr[2065]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:12 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:14 d12 docker[2074]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:16 d12 docker[2074]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:16 d12 docker[2074]: time="2018-01-31T22:54:16Z" level=error msg="error getting events from daemon: net/http: request canceled"
Jan 31 22:54:16 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:16 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:16 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:17 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:17 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:17 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:17 d12 docker-credential-gcr[2152]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:17 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:19 d12 docker[2161]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:19 d12 docker[2161]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:19 d12 docker[2161]: time="2018-01-31T22:54:19Z" level=error msg="error getting events from daemon: context canceled"
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:20 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:20 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:20 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:20 d12 docker-credential-gcr[2281]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:20 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:21 d12 docker[2294]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:21 d12 docker[2294]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:22 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:22 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:22 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:22 d12 docker-credential-gcr[2388]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:22 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:22 d12 docker[2396]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:23 d12 docker[2396]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:24 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:24 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:24 d12 docker-credential-gcr[2471]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:24 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:24 d12 docker[2479]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:24 d12 docker[2479]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:17 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:19 d12 docker[2161]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:19 d12 docker[2161]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:19 d12 docker[2161]: time="2018-01-31T22:54:19Z" level=error msg="error getting events from daemon: context canceled"
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:20 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:20 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:20 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:20 d12 docker-credential-gcr[2281]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:20 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:21 d12 docker[2294]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:21 d12 docker[2294]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:22 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:22 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:22 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:22 d12 docker-credential-gcr[2388]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:22 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:22 d12 docker[2396]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:23 d12 docker[2396]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:24 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:24 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:24 d12 docker-credential-gcr[2471]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:24 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:24 d12 docker[2479]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:24 d12 docker[2479]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:25 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:25 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:25 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:26 d12 docker-credential-gcr[2557]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:26 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:26 d12 docker[2565]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:26 d12 docker[2565]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:27 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:27 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:27 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:27 d12 docker-credential-gcr[2644]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:27 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:28 d12 docker[2652]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:28 d12 docker[2652]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:29 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:29 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:29 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:29 d12 docker-credential-gcr[2730]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:29 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:29 d12 docker[2739]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:30 d12 docker[2739]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:17 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:19 d12 docker[2161]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:19 d12 docker[2161]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:19 d12 docker[2161]: time="2018-01-31T22:54:19Z" level=error msg="error getting events from daemon: context canceled"
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:20 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:20 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:20 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:20 d12 docker-credential-gcr[2281]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:20 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:21 d12 docker[2294]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:21 d12 docker[2294]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:22 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:22 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:22 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:22 d12 docker-credential-gcr[2388]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:22 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:22 d12 docker[2396]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:23 d12 docker[2396]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:24 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:24 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:24 d12 docker-credential-gcr[2471]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:24 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:24 d12 docker[2479]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:24 d12 docker[2479]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:25 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:25 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:25 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:26 d12 docker-credential-gcr[2557]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:26 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:26 d12 docker[2565]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:26 d12 docker[2565]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:27 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:27 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:27 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:27 d12 docker-credential-gcr[2644]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:27 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:28 d12 docker[2652]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:28 d12 docker[2652]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:29 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:29 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:29 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:29 d12 docker-credential-gcr[2730]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:29 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:29 d12 docker[2739]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:30 d12 docker[2739]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:31 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:31 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:31 d12 docker-credential-gcr[2816]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:31 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:31 d12 docker[2824]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:31 d12 docker[2824]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:32 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:32 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:32 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:33 d12 docker-credential-gcr[2904]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:33 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:33 d12 docker[2913]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:33 d12 docker[2913]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:33 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:33 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:33 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:34 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:34 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:34 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:34 d12 docker-credential-gcr[2990]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:34 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:35 d12 docker[2998]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:35 d12 docker[2998]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:35 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:35 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:35 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:36 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:36 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:36 d12 systemd[1]: datalab.service: Start request repeated too quickly.
Jan 31 22:54:36 d12 systemd[1]: Failed to start datalab docker container.
Jan 31 22:54:36 d12 systemd[1]: datalab.service: Unit entered failed state.
lines 137-170Jan 31 22:54:17 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:19 d12 docker[2161]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:19 d12 docker[2161]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:19 d12 docker[2161]: time="2018-01-31T22:54:19Z" level=error msg="error getting events from daemon: context canceled"
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:19 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:20 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:20 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:20 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:20 d12 docker-credential-gcr[2281]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:20 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:21 d12 docker[2294]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:21 d12 docker[2294]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:21 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:22 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:22 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:22 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:22 d12 docker-credential-gcr[2388]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:22 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:22 d12 docker[2396]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:23 d12 docker[2396]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:23 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:24 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:24 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:24 d12 docker-credential-gcr[2471]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:24 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:24 d12 docker[2479]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:24 d12 docker[2479]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:24 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:25 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:25 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:25 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:26 d12 docker-credential-gcr[2557]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:26 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:26 d12 docker[2565]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:26 d12 docker[2565]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:26 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:27 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:27 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:27 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:27 d12 docker-credential-gcr[2644]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:27 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:28 d12 docker[2652]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:28 d12 docker[2652]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:28 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:29 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:29 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:29 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:29 d12 docker-credential-gcr[2730]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:29 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:29 d12 docker[2739]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:30 d12 docker[2739]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:30 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:31 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:31 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:31 d12 docker-credential-gcr[2816]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:31 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:31 d12 docker[2824]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:31 d12 docker[2824]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:31 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:32 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:32 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:32 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:33 d12 docker-credential-gcr[2904]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:33 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:33 d12 docker[2913]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:33 d12 docker[2913]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:33 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:33 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:33 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:34 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:34 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:34 d12 systemd[1]: Starting datalab docker container...
Jan 31 22:54:34 d12 docker-credential-gcr[2990]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Jan 31 22:54:34 d12 systemd[1]: Started datalab docker container.
Jan 31 22:54:35 d12 docker[2998]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Jan 31 22:54:35 d12 docker[2998]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Jan 31 22:54:35 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Jan 31 22:54:35 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:35 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Jan 31 22:54:36 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Jan 31 22:54:36 d12 systemd[1]: Stopped datalab docker container.
Jan 31 22:54:36 d12 systemd[1]: datalab.service: Start request repeated too quickly.
Jan 31 22:54:36 d12 systemd[1]: Failed to start datalab docker container.
Jan 31 22:54:36 d12 systemd[1]: datalab.service: Unit entered failed state.
Jan 31 22:54:36 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
-- Reboot --
Feb 01 15:42:01 d12 systemd[1]: Starting datalab docker container...
Feb 01 15:42:01 d12 docker-credential-gcr[1039]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Feb 01 15:42:01 d12 systemd[1]: Started datalab docker container.
Feb 01 15:42:02 d12 docker[1047]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Feb 01 15:42:02 d12 docker[1047]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Feb 01 15:42:02 d12 docker[1047]: time="2018-02-01T15:42:02Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 15:42:02 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 15:42:02 d12 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 15:42:02 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 15:42:04 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 15:42:04 d12 systemd[1]: Stopped datalab docker container.
Feb 01 15:42:04 d12 systemd[1]: Starting datalab docker container...
Feb 01 15:42:04 d12 docker-credential-gcr[1205]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Feb 01 15:42:04 d12 systemd[1]: Started datalab docker container.
Feb 01 15:42:04 d12 docker[1213]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Feb 01 15:42:04 d12 docker[1213]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Feb 01 15:42:04 d12 docker[1213]: time="2018-02-01T15:42:04Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 15:42:04 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 15:42:04 d12 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 15:42:04 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 15:42:05 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 15:42:05 d12 systemd[1]: Stopped datalab docker container.
Feb 01 15:42:05 d12 systemd[1]: Starting datalab docker container...
Feb 01 15:42:05 d12 docker-credential-gcr[1288]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Feb 01 15:42:05 d12 systemd[1]: Started datalab docker container.
Feb 01 15:42:06 d12 docker[1297]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Feb 01 15:42:06 d12 docker[1297]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Feb 01 15:42:06 d12 docker[1297]: time="2018-02-01T15:42:06Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 15:42:06 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 15:42:06 d12 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 15:42:06 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 15:42:07 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
lines 171-204
Feb 01 15:42:05 d12 systemd[1]: Started datalab docker container.
Feb 01 15:42:06 d12 docker[1297]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Feb 01 15:42:06 d12 docker[1297]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Feb 01 15:42:06 d12 docker[1297]: time="2018-02-01T15:42:06Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 15:42:06 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 15:42:06 d12 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 15:42:06 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 15:42:07 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 15:42:07 d12 systemd[1]: Stopped datalab docker container.
Feb 01 15:42:07 d12 systemd[1]: Starting datalab docker container...
Feb 01 15:42:07 d12 docker-credential-gcr[1382]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Feb 01 15:42:07 d12 systemd[1]: Started datalab docker container.
Feb 01 15:42:08 d12 docker[1390]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Feb 01 15:42:08 d12 docker[1390]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Feb 01 15:42:08 d12 docker[1390]: time="2018-02-01T15:42:08Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 15:42:08 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 15:42:08 d12 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 15:42:08 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 15:42:09 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 15:42:09 d12 systemd[1]: Stopped datalab docker container.
Feb 01 15:42:09 d12 systemd[1]: Starting datalab docker container...
Feb 01 15:42:09 d12 docker-credential-gcr[1477]: /home/datalab/.docker/config.json configured to use this credential helper for GCR registries
Feb 01 15:42:09 d12 systemd[1]: Started datalab docker container.
Feb 01 15:42:09 d12 docker[1485]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat /datalab/run-extended.sh: no s
Feb 01 15:42:09 d12 docker[1485]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \
Feb 01 15:42:09 d12 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 15:42:09 d12 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 15:42:09 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 15:42:10 d12 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 15:42:10 d12 systemd[1]: Stopped datalab docker container.
Feb 01 15:42:10 d12 systemd[1]: datalab.service: Start request repeated too quickly.
Feb 01 15:42:10 d12 systemd[1]: Failed to start datalab docker container.
Feb 01 15:42:10 d12 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 15:42:10 d12 systemd[1]: datalab.service: Failed with result 'exit-code'.
lines 197-230/230 (END)

[harmon]

I've seen errors like this when the docker image isn't actually an image. Like you pushed a container or something. Can you comment out the "docker run" in your run-extended.sh and try rebuilding it, tag the image, push it, and boot a new VM with it? I have a feeling the image you pushed up wasn't built correctly.

Also, if you do want to use the "docker run" in that run-extended.sh to test out your build, be sure to add "--rm" to the docker run options, so that it doesn't modify the image you just built.

[eilalan]

I have commented out the docker run command when you guided me to take out the --cap-add sys_admin --device=/dev/fuse options. in any case, I will rebuild the docker and push it to google containers again

[eilalan]

I have rerun the run-extended, tagged the image and pushed it to google. Updated run-extended.sh When I started working with the run-extended for the cloud shell version, I have updated the DATALAB_ENV from local to GCE and kept it like that

#!/bin/bash -e
# Copyright 2015 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Runs the docker container locally. Auth will happen in the browser when
# the user first opens Datalab after accepting the EULA.

# Passing in 'shell' flag causes the docker container to break into a
# command prompt, which is useful for tinkering within the container before
# manually starting the server.

CONTENT=$HOME
DOCKERIMAGE=datalab-extended
docker-credential-gcr configure-docker
#ENTRYPOINT="/datalab/run.sh"
ENTRYPOINT="/datalab/run-custom.sh"
if [ "$1" != "" ]; then
  if [ "$1" != "shell" ]; then
    CONTENT=$1
    shift
  fi
  if [ "$1" == "shell" ]; then
    ENTRYPOINT="/bin/bash"
  fi
fi

# Custom packages can be installed in datalab prior to run time. There are 2 options
# available. A simple option is to add the packages to a pip requirements file. Create a file
# called 'custom-packages.txt' and place it in the $REPO_DIR/containers/datalab folder. Use
# 'custom-packages-example.txt' as a starting point. Another option is to create a Dockerfile
# called 'Dockerfile-extended.in' and place it in the $REPO_DIR/containers/datalab folder.
# Use 'Dockerfile-extended-example.in' as a starting point. In both cases, a customized
# docker image will be created which is derived from the standard datalab image.
if [ -f custom-packages.txt ] || [ -f Dockerfile-extended.in ];
then
    if [ -f custom-packages.txt ];
    then
        # First create a new Docker file called Dockerfile-custom-packages. Start with the standard image
        # TODO: at some point the local Datalab container will be tagged 'latest' rather than 'local'
        # and the line below should change.
        #echo 'FROM gcr.io/cloud-datalab/datalab:latest' > Dockerfile-custom-packages
        echo "not implemented"
        # Add the script with a list of custom packages to the Dockerfile
        #echo 'ADD custom-packages.txt /datalab/custom-packages.txt' >> Dockerfile-custom-packages
        #echo 'RUN pip install -r /datalab/custom-packages.txt' >> Dockerfile-custom-packages

        #DOCKERFILE=Dockerfile-custom-packages
        #DOCKERIMAGE=datalab-custom-packages

    elif [ -f Dockerfile-extended.in ];
    then
        # DOCKERFILE=Dockerfile-extended.in
        DOCKERFILE=Dockerfile-extended.in
        DOCKERIMAGE=datalab-extended
    fi

    # Build the customized docker image derived from the standard datalab image
    docker build ${DOCKER_BUILD_ARGS} -t $DOCKERIMAGE -f $DOCKERFILE .
fi

# On linux docker runs directly on host machine, so bind to 127.0.0.1 only
# to avoid it being accessible from network.
# On other platform, it needs to bind to all ip addresses so VirtualBox can
# access it. Users need to make sure in their VirtualBox port forwarding
# settings only 127.0.0.1 is bound.
if [ "$OSTYPE" == "linux"* ]; then
  PORTMAP="127.0.0.1:8081:8080"
else
  PORTMAP="8081:8080"
fi

# Use this flag to map in web server content during development
#  -v $REPO_DIR/sources/web:/sources \
#/usr/bin/docker run -it --cap-add SYS_ADMIN --device=/dev/fuse --entrypoint=$ENTRYPOINT \
#  -p $PORTMAP \
#  -v "$CONTENT/datalab:/content/datalab" \
# -e "PROJECT_ID=$PROJECT_ID" \
#  -e "DATALAB_ENV=GCE" \
#  -e "EXPERIMENTAL_KERNEL_GATEWAY_URL=${EXPERIMENTAL_KERNEL_GATEWAY_URL}" \
#  $DOCKERIMAGE

The head of the datalab log:

eb 01 16:57:04 d112 systemd[1]: Starting datalab docker container...
Feb 01 16:57:05 d112 docker-credential-gcr[1490]: /home/datalab/.docker/config.json configured to use this credential helper for GCR re
Feb 01 16:57:05 d112 systemd[1]: Started datalab docker container.
Feb 01 16:57:06 d112 docker[1517]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat 
Feb 01 16:57:06 d112 docker[1517]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting con
Feb 01 16:57:06 d112 docker[1517]: time="2018-02-01T16:57:06Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 16:57:06 d112 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 16:57:06 d112 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 16:57:06 d112 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 16:57:07 d112 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 16:57:07 d112 systemd[1]: Stopped datalab docker container.
Feb 01 16:57:07 d112 systemd[1]: Starting datalab docker container...
Feb 01 16:57:07 d112 docker-credential-gcr[1637]: /home/datalab/.docker/config.json configured to use this credential helper for GCR re
Feb 01 16:57:07 d112 systemd[1]: Started datalab docker container.
Feb 01 16:57:12 d112 docker[1654]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat 
Feb 01 16:57:12 d112 docker[1654]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting con
Feb 01 16:57:12 d112 docker[1654]: time="2018-02-01T16:57:12Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 16:57:12 d112 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 16:57:12 d112 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 16:57:12 d112 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 16:57:13 d112 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 16:57:13 d112 systemd[1]: Stopped datalab docker container.
Feb 01 16:57:13 d112 systemd[1]: Starting datalab docker container...
Feb 01 16:57:13 d112 docker-credential-gcr[1731]: /home/datalab/.docker/config.json configured to use this credential helper for GCR re
Feb 01 16:57:13 d112 systemd[1]: Started datalab docker container.
Feb 01 16:57:14 d112 docker[1739]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat 
Feb 01 16:57:14 d112 docker[1739]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting con
Feb 01 16:57:14 d112 docker[1739]: time="2018-02-01T16:57:14Z" level=error msg="error getting events from daemon: context canceled"
Feb 01 16:57:14 d112 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a
Feb 01 16:57:14 d112 systemd[1]: datalab.service: Unit entered failed state.
Feb 01 16:57:14 d112 systemd[1]: datalab.service: Failed with result 'exit-code'.
Feb 01 16:57:15 d112 systemd[1]: datalab.service: Service hold-off time over, scheduling restart.
Feb 01 16:57:15 d112 systemd[1]: Stopped datalab docker container.

[harmon]

Oh, you didn't change the entry point to /datalab/run-custom.sh, it's still set to run-extended.sh.

On Thu, Feb 1, 2018, 12:03 PM eilalan notifications@github.com wrote:

I have rerun the run-extended, tagged the image and pushed it to google. Updated run-extended.sh When I started working with the run-extended for the cloud shell version, I have updated the DATALAB_ENV from local to GCE and kept it like that

!/bin/bash -e

Copyright 2015 Google Inc. All rights reserved.

#

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#

http://www.apache.org/licenses/LICENSE-2.0

#

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

Runs the docker container locally. Auth will happen in the browser when

the user first opens Datalab after accepting the EULA.

Passing in 'shell' flag causes the docker container to break into a

command prompt, which is useful for tinkering within the container before

manually starting the server.

CONTENT=$HOME DOCKERIMAGE=datalab-extended docker-credential-gcr configure-docker

ENTRYPOINT="/datalab/run.sh"

ENTRYPOINT="/datalab/run-custom.sh" if [ "$1" != "" ]; then if [ "$1" != "shell" ]; then CONTENT=$1 shift fi if [ "$1" == "shell" ]; then ENTRYPOINT="/bin/bash" fi fi

Custom packages can be installed in datalab prior to run time. There are 2 options

available. A simple option is to add the packages to a pip requirements file. Create a file

called 'custom-packages.txt' and place it in the $REPO_DIR/containers/datalab folder. Use

'custom-packages-example.txt' as a starting point. Another option is to create a Dockerfile

called 'Dockerfile-extended.in' and place it in the $REPO_DIR/containers/datalab folder.

Use 'Dockerfile-extended-example.in' as a starting point. In both cases, a customized

docker image will be created which is derived from the standard datalab image.

if [ -f custom-packages.txt ] || [ -f Dockerfile-extended.in ]; then if [ -f custom-packages.txt ]; then

First create a new Docker file called Dockerfile-custom-packages. Start with the standard image

    # TODO: at some point the local Datalab container will be tagged 'latest' rather than 'local'
    # and the line below should change.
    #echo 'FROM gcr.io/cloud-datalab/datalab:latest' > Dockerfile-custom-packages
  echo "not implemented"
    # Add the script with a list of custom packages to the Dockerfile
    #echo 'ADD custom-packages.txt /datalab/custom-packages.txt' >> Dockerfile-custom-packages
    #echo 'RUN pip install -r /datalab/custom-packages.txt' >> Dockerfile-custom-packages

    #DOCKERFILE=Dockerfile-custom-packages
    #DOCKERIMAGE=datalab-custom-packages

elif [ -f Dockerfile-extended.in ];
then
  # DOCKERFILE=Dockerfile-extended.in
    DOCKERFILE=Dockerfile-extended.in
    DOCKERIMAGE=datalab-extended
fi

# Build the customized docker image derived from the standard datalab image
docker build ${DOCKER_BUILD_ARGS} -t $DOCKERIMAGE -f $DOCKERFILE .

fi

On linux docker runs directly on host machine, so bind to 127.0.0.1 only

to avoid it being accessible from network.

On other platform, it needs to bind to all ip addresses so VirtualBox can

access it. Users need to make sure in their VirtualBox port forwarding

settings only 127.0.0.1 is bound.

if [ "$OSTYPE" == "linux"* ]; then PORTMAP="127.0.0.1:8081:8080" else PORTMAP="8081:8080" fi

Use this flag to map in web server content during development

-v $REPO_DIR/sources/web:/sources \

/usr/bin/docker run -it --cap-add SYS_ADMIN --device=/dev/fuse --entrypoint=$ENTRYPOINT \

-p $PORTMAP \

-v "$CONTENT/datalab:/content/datalab" \

-e "PROJECT_ID=$PROJECT_ID" \

-e "DATALAB_ENV=GCE" \

-e "EXPERIMENTAL_KERNEL_GATEWAY_URL=${EXPERIMENTAL_KERNEL_GATEWAY_URL}" \

$DOCKERIMAGE

The head of the datalab log:

eb 01 16:57:04 d112 systemd[1]: Starting datalab docker container... Feb 01 16:57:05 d112 docker-credential-gcr[1490]: /home/datalab/.docker/config.json configured to use this credential helper for GCR re Feb 01 16:57:05 d112 systemd[1]: Started datalab docker container. Feb 01 16:57:06 d112 docker[1517]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat Feb 01 16:57:06 d112 docker[1517]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting con Feb 01 16:57:06 d112 docker[1517]: time="2018-02-01T16:57:06Z" level=error msg="error getting events from daemon: context canceled" Feb 01 16:57:06 d112 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a Feb 01 16:57:06 d112 systemd[1]: datalab.service: Unit entered failed state. Feb 01 16:57:06 d112 systemd[1]: datalab.service: Failed with result 'exit-code'. Feb 01 16:57:07 d112 systemd[1]: datalab.service: Service hold-off time over, scheduling restart. Feb 01 16:57:07 d112 systemd[1]: Stopped datalab docker container. Feb 01 16:57:07 d112 systemd[1]: Starting datalab docker container... Feb 01 16:57:07 d112 docker-credential-gcr[1637]: /home/datalab/.docker/config.json configured to use this credential helper for GCR re Feb 01 16:57:07 d112 systemd[1]: Started datalab docker container. Feb 01 16:57:12 d112 docker[1654]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat Feb 01 16:57:12 d112 docker[1654]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting con Feb 01 16:57:12 d112 docker[1654]: time="2018-02-01T16:57:12Z" level=error msg="error getting events from daemon: context canceled" Feb 01 16:57:12 d112 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a Feb 01 16:57:12 d112 systemd[1]: datalab.service: Unit entered failed state. Feb 01 16:57:12 d112 systemd[1]: datalab.service: Failed with result 'exit-code'. Feb 01 16:57:13 d112 systemd[1]: datalab.service: Service hold-off time over, scheduling restart. Feb 01 16:57:13 d112 systemd[1]: Stopped datalab docker container. Feb 01 16:57:13 d112 systemd[1]: Starting datalab docker container... Feb 01 16:57:13 d112 docker-credential-gcr[1731]: /home/datalab/.docker/config.json configured to use this credential helper for GCR re Feb 01 16:57:13 d112 systemd[1]: Started datalab docker container. Feb 01 16:57:14 d112 docker[1739]: container_linux.go:247: starting container process caused "exec: \"/datalab/run-extended.sh\": stat Feb 01 16:57:14 d112 docker[1739]: /usr/bin/docker: Error response from daemon: oci runtime error: container_linux.go:247: starting con Feb 01 16:57:14 d112 docker[1739]: time="2018-02-01T16:57:14Z" level=error msg="error getting events from daemon: context canceled" Feb 01 16:57:14 d112 systemd[1]: datalab.service: Main process exited, code=exited, status=127/n/a Feb 01 16:57:14 d112 systemd[1]: datalab.service: Unit entered failed state. Feb 01 16:57:14 d112 systemd[1]: datalab.service: Failed with result 'exit-code'. Feb 01 16:57:15 d112 systemd[1]: datalab.service: Service hold-off time over, scheduling restart. Feb 01 16:57:15 d112 systemd[1]: Stopped datalab docker container.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/googledatalab/datalab/issues/1927#issuecomment-362332703, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAIeH3rPQsiNzHG-U7oljnAvbgG_2FVks5tQe5LgaJpZM4Rr-gK .

[eilalan]

what do you mean? ENTRYPOINT="/datalab/run-custom.sh" what am i missing?

[eilalan]

got it at the Dockerfile

[eilalan]

Thank you! Thank you! Thank you!

[harmon]

Awesome! Congrats :)