Looking for a strangely-named image

[arokem]

Hello! Thanks again for this great project.

Running on a mac laptop, having installed caliban with pip, I tried the following:

$ caliban notebook --nogpu
I0204 10:58:44.096491 4624354752 build.py:731] Running command: docker build --rm -f- /Users/arokem/tmp/caliban_test
/Users/arokem/miniconda3/envs/caliban/lib/python3.8/subprocess.py:838: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdin = io.open(p2cwrite, 'wb', bufsize)
/Users/arokem/miniconda3/envs/caliban/lib/python3.8/subprocess.py:844: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdout = io.open(c2pread, 'rb', bufsize)
#1 [internal] load build definition from Dockerfile
#1 sha256:f28aea6def1a6cd99c3dba9a8588148c15a44dbfe43b481d262a0028d8dda975
#1 transferring dockerfile: 1.08kB 0.0s done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:6dc01f396991620f90d338b881b4f6b1f6f0c9d6c01dfad942b60c6ef69159d7
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for gcr.io/blueshift-playground/blueshift:cpu-ubuntu1804-py37
#3 sha256:25c05f5af803364da95cc3e34f6fd1b1fed810090a16c60858c3d711f73887c1
#3 DONE 0.3s

#4 [ 1/12] FROM gcr.io/blueshift-playground/blueshift:cpu-ubuntu1804-py37@sha256:ba03f280085e923a6be72af7d697cf209a17506cbe1963a927fc8633031c7fb0
#4 sha256:afa8668d82673191feeaa31b8753b5b616f1f9c9dbcb24c3e59516d900a74ef9
#4 DONE 0.0s

#10 [internal] load build context
#10 sha256:a35234ecc058bfe64aab295bb66df4d3bde95af86b97423e1400e05f86b78f8a
#10 transferring context: 248B done
#10 DONE 0.0s

#8 [ 5/12] WORKDIR /usr/app
#8 sha256:2018ebc767a5256774e0043b6c3a5b77d0fd9228fe9d1c25218eed5654b31f47
#8 CACHED

#5 [ 2/12] RUN [ $(getent group 20) ] || groupadd --gid 20 20
#5 sha256:83a3c5bf3f1567625775b104b115a569214db23e58cea0b09f683c85dfd0c12e
#5 CACHED

#9 [ 6/12] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends zsh && apt-get clean && rm -rf /var/lib/apt/lists/*
#9 sha256:152bc68698ec58a809ee62324e32d4a31182d396774266aadc4a704749cd0aaa
#9 CACHED

#7 [ 4/12] RUN mkdir -m 777 /usr/app /.creds /.resources /home/arokem
#7 sha256:ace23be941af73ca9b62052532466d06c8ff2656ebf85fb5107f3680461b7771
#7 CACHED

#13 [ 9/12] RUN pip install jupyter
#13 sha256:80d93a11752a3233671d7bef53df010723558d69cf6f3e03fa2369dce1c11247
#13 CACHED

#6 [ 3/12] RUN useradd --no-log-init --no-create-home -u 501 -g 20 --shell /bin/bash arokem
#6 sha256:8c0e7c3001be7c1c7ad87536bcfc588350fc5ab55a04b4f62370cf55249a3548
#6 CACHED

#14 [10/12] COPY --chown=501:20 cloud_sql_proxy.py /.resources
#14 sha256:10d002a7044af1e1cea195fed2a0ac90c7468733f5433146334f03f3097899d0
#14 CACHED

#11 [ 7/12] COPY --chown=501:20 requirements.txt /usr/app
#11 sha256:3d1c7cfc1fa38374f278e323ba6f19471d578bc0faaf13eb5f09cb6a28c809e6
#11 CACHED

#12 [ 8/12] RUN /bin/bash -c "pip install --no-cache-dir -r requirements.txt"
#12 sha256:ea84f2faf52c42eca47379f73b7718c2aaa561a1c60b21cedf5c97d0bdc5e6cb
#12 CACHED

#15 [11/12] COPY --chown=501:20 caliban_launcher.py /.resources
#15 sha256:13f784a9b4ae0df920d8e9b1a22d410508e4bae77af6693cd79d11e3d041a330
#15 CACHED

#16 [12/12] COPY --chown=501:20 ./caliban_launcher_cfg.json /.resources
#16 sha256:c9c4c3a49dee7f24c33b75dd28282056cfb21bb64885626da62e4bb90c55cf3b
#16 CACHED

#17 exporting to image
#17 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#17 exporting layers done
#17 writing image sha256:a06b3ff98fd03ae55ff165572f5a29559ebc8d16b0e78cf46986801d573cbdc8 done
#17 DONE 0.0s
I0204 10:58:44.979649 4624354752 run.py:335] Running command: docker run --ipc host -w /usr/app -u 501:20 -v /Users/arokem/tmp/caliban_test:/usr/app -it --entrypoint python -v /Users/arokem:/home/arokem -p 8889:8889 0.0s -m jupyter notebook --ip=0.0.0.0 --port=8889 --no-browser
Unable to find image '0.0s:latest' locally
docker: Error response from daemon: pull access denied for 0.0s, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

Looks like maybe the docker command is malformed? Why is that 0.0s in there?

[arokem]

Update: seems like this is not notebook-specific. I get a similar issue when calling caliban shell --nogpu. Updating the title of the issue accordingly.

[sritchie]

Hey @arokem ! I'm sure that the issue lives right at this line; the output of your Docker build looks different from mine, so my simple "get the image ID" parsing has failed us.

What version of Docker are you on?

I'm no longer at Google so I'll ask @ajslone to take a look at this problem. The idea solution would be if caliban could pass some flag to docker build that would generate structured logging we could read, instead of what I'm doing here.

There is a python library for Docker, but it is very limited, and couldn't (at the time) support the custom Dockerfile builds we do in Caliban.

[arokem]

Thanks for the quick reply here. I'm on Docker version 20.10.0, build 7287ab3

[ajslone]

Sorry for the delay in looking at this! I'll have a look and try to get a fix in.

[arokem]

FWIW (probably not much?): this works on my version of docker.

[sritchie]

@ajslone ping!

[ajslone]

Ach, my apologies, been a bit swamped. I'll have a look.

[jake-westfall]

I'm running into the same issue when trying to run caliban shell --nogpu on a Macbook Pro.

jake-mbp:basic_example jakewestfall$ caliban shell --nogpu
I0528 13:53:41.221027 140735501751168 build.py:731] Running command: docker build --rm -f- /Users/jakewestfall/Desktop/projects/training/python/2020-10-08_caliban/basic_example
#1 [internal] load build definition from Dockerfile
#1 sha256:f6fa4fe0bc2c1f19770d26b64a8b5a9035209a7fae8990381471f751b3a7a012
#1 transferring dockerfile: 1.01kB 0.0s done
#1 DONE 0.0s

[...lots of text snipped...]

#16 exporting to image
#16 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#16 exporting layers
#16 exporting layers 0.3s done
#16 writing image sha256:a14cb50a19ccd5dc35a94f92ef33561205c270654d7b9d8ed6c602902caa9ac8 done
#16 DONE 0.3s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
I0528 13:56:26.669191 140735501751168 run.py:335] Running command: docker run --ipc host -w /usr/app -u 501:20 -v /Users/jakewestfall/Desktop/projects/training/python/2020-10-08_caliban/basic_example:/usr/app -it --entrypoint /bin/bash -v /Users/jakewestfall:/home/jakewestfall them
Unable to find image 'them:latest' locally
docker: Error response from daemon: pull access denied for them, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

I'm on Docker version 20.10.6, build 370c289

It looks like in docker 20.10.5 the following line was added to the docker build output:

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

So apparently caliban is grabbing the last word of this last line thinking it is the image ID, a strategy which used to work in older versions of docker.

I imagine a hack similar to what @arokem posted would work for my case. But it seems that more generally the image id could be appearing in different locations depending on the user's docker version. So maybe a more acceptable hack would be to grab the last word in the docker build output that starts with sha256:.

[sritchie]

Hey all,

Apologies that this issue has gone unaddressed for so long. I'm not longer at Google, and @ajslone is the maintainer of the project — I'm not sure what he is up to here.

I still have admin access to the project, and would be thrilled to review and merge any fixes that y'all come up. I think

So maybe a more acceptable hack would be to grab the last word in the docker build output that starts with sha256

is a great fix, and more in the spirit of what the code was trying to do in the first place. (I originally attempted to use the python library that Docker provides for this, but it wasn't possible to do things like stream output back to the user from the build process, so I went this shell-out way.)

Cheers, and apologies for this! ~Sam

[callmekofi]

I'm running into the same issue when trying to run caliban shell --nogpuon a MacOS Monterey version 12.0.1

[sritchie]

I've finally fixed this over at #109. I'll merge this shortly. Thanks all!

[sritchie]

Okay, this fix is released as 0.4.2.

FYI, I'll be slowly ramping up on Caliban and getting some more work done here :)