Weirdness with fromImage

[angerman]

I'm trying to use nix2container to build a google cloudshell container, and I'm running into multiple issues :D

My basic configuration looks something like this:

nix2containerPkgs.nix2container.buildImage {
      name = "gcr.io/myproject/mycloudshell";
      tag = "latest";
      fromImage = nix2containerPkgs.nix2container.pullImage {
        imageName = "gcr.io/cloudshell-images/cloudshell";
        imageDigest = "sha256:68f5f1a01574bd795192098d676ac4150610ab89d4c0c23e72f9a0f7ec2cf1db";
        sha256 = "sha256-i++Camqmzugr3aq56UvGujuFqDo8Cj6gD9IY/a/0HpI=";
      };
      maxLayers = 200;
       config.env = [
         "DEBIAN_FRONTEND=noninteractive"
         "PATH=/opt/gradle/bin:/opt/maven/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/usr/local/nvm/versions/node/v16.4.0/bin:/usr/local/rvm/bin:/google/go_appengine:/google/google_appengine"
         "PORT=8080"
         "CLOUD_SDK=/usr/lib/google-cloud-sdk"
         "GCLOUD_CONTAINER_SERVER=gcr.io"
         "LOG=file"
         "DOCKER_HOST=unix:///var/run/docker.sock"
         "NODEJS_VERSION=16.4.0"
         "DEVSHELL_CLIENTS_DIR=/var/run/google/devshell"
         "CREDENTIALS_SERVICE_PORT=8998"
         "KUBECTX_VERSION=0.9.4"
      ];
      config.entrypoint = [
            "/bin/bash"
            "/google/scripts/onrun.sh"
      ];
      config.volumes."/var/lib/docker" = {};
    };
  };

funnily enough, despite it being public (as per gcr), cloud shell complains it can't find the image.

If I on the other hand construct an empty image (using google base image), and google cloud shell's guide (which uses docker I believe), I get a working image. If I then use google cloud shell's guild but use my image (produced as above) as a base image, google cloud shell can find it, but fails to load it.

What I've found so far is that the date tags are different (of couse, nix is at unix:0), but a bit more unexpected is that the layer hashes are all different in both images. E.g. using skopeo inspect the layers all have different hashes; and the nix2container one also misses all the history items, though I doubt that's much of an issue.

The nix2container generated one also doesn't set the

    "Created": "2022-06-04T08:28:10.227221947Z",
    "DockerVersion": "18.09.0",

values. And gcr also seems to be unable to compute the size of the image.

[angerman]

Just to add one more data point. These are the layers as per skopeo inspect from the google cloud shell base image:

    "Layers": [
        "sha256:cbfe985b5bc1d578be10f1863a87a997158bf3c8cddcbf1f25d049ad210fa20b",
        "sha256:2e1bfe63320ec22cf000ba264ac09c48c11a37a1c175a50f6fd314fc9862bd40",
        "sha256:44b9aba2766a5ce891f7f9d8a076a48a71fda78a724c58dbc70567088b749d2e",
        "sha256:30d7158c02dba942c61186106c2b07adf8177d4c3da51e744c82e7ece10aecac",
        "sha256:a4660d663c916d350500fb560e63349193afb4a0c71981870add3c69db0d9154",
        "sha256:02d0a80b7a6c4da4811e837699ac6d34b2e2df73fd55b174d7862b761a5b1bb5",
        "sha256:4a2a4979208f301ac3aefd6c351283e3e002073ae14357c9c10f0ad125fca5e9",
        "sha256:f0ade090fd3bc2d5595688a7fceba981eaa7f9b8e26b11251f3f3c59cfd5a5bb",
        "sha256:ee3a4015a1c87e4cc149840fc37dbcfad6cb0aeb5738c7ccbb45e55a69510155",
        "sha256:229ba37d5e054db1ba603758da24195c60fa26c4448aa1314bc021556e594f38",
        "sha256:8ce61e8947f8ed95072cfff242e9c6cacb4c12039ee87d460e0b5908056878f3",
        "sha256:711c2c23758805806af0281c3087d644677af6d7218f1d111ebf2eaf8c820de9",
        "sha256:b6a2942accd0adb9390d8ab1ea0a7b3989b37b5c1464cec346621e33ef5f10e7",
        "sha256:cfc807cad85c1a0fa5fa956f24ed43438af64d6944ce012bac6728785f305495",
        "sha256:69d836b6052b333d2048dff7ad7768c81414433daf31ad9862d1fa930c852d1c",
        "sha256:4dd20947788ef751552bf6b7f73870ba93abeff4709ed52c7f7d540184dc79a6",
        "sha256:1dc037f9de2997b14c5d85db4c634f212b37763ee589578546b5c4a9c1e94c37",
        "sha256:30299f0351dc10afda18f4cbc0b50d2e27ccdfb9c9662aa3b03e2a93ff80917e",
        "sha256:d0584d305ec0f87c7561d8fbc37f1a363714db647e12ab70dc78b71f0c2e88a7",
        "sha256:3cae59815d2f3788d73d788281d9c2596fb62cb46d6de625e61fcafc29e31ba8",
        "sha256:6517927bd44d909afbf995499371afedfb9c517a195c9624acf2f849cd21d9a7",
        "sha256:22d2856e84b368d2fa4441ada9b651c71f3ef8fa5b0bed8fcdefbc427aa73e95",
        "sha256:8f4d8937bfcce7ddad35dd61b9c457e4fc06190a704004526041bc033b1b0cdb",
        "sha256:aad151415fce55820e23036edcac6535f6f6610659dea7a00a4ee8efb4c01855",
        "sha256:da71c8d20604ce83c8f6e0532e38c0102c6d8ffaac21838b07800108116a1eff",
        "sha256:1555eef99f5ce3682b81d94b027399a3b1e2e23a27e2be5fce9dea8a88e53ce1",
        "sha256:1aac2ba2c5336021a0e950bcff3c32907c18397429b6e3f5b600ed2931fe33af",
        "sha256:8e70e1080403a98dcd2ab2e0d0951678277224c3ea30aa25c27e51f25c616629",
        "sha256:6911edd37e129bfde0649a68b3d0ea25b655e929b78712bf49fb3fd26439ab5d",
        "sha256:17e37c11c14ce30a5ee9844b22356e4c45a75680d31b28c18d2ed0046499c236",
        "sha256:ea7454e65f75cade485d5e091ab3442f35897807e7badd0f10cce7b5575db8aa",
        "sha256:a5e6d61a09fb3a9cfd57b707f4f5ce5a132cf898f472d757eb28119c531f3f6b",
        "sha256:b5f2647e56dc091f98ed0d45f2d497a1939adab00bf633af12bcf1c8dc006c9d",
        "sha256:5f02ee1bbe107fa3b185b1f5995ce96152acbd03ea44de91bc7221911d0502f7",
        "sha256:7bb076c8a942e5586b3d6c54fb165e32ed5de8e7c9bfaa3184e2f3b89affe254",
        "sha256:d94c2f224d1e0e631b94f4a546c320b5cb9959b98596dc0869d5a5205d52f539",
        "sha256:8627a44593f61f156035617cf550385a0866b4444629aa81a5b3bd0af14225a1",
        "sha256:6b2bc5feff35d23ae621ec1a9be95c796176e293251ff0bc4501f5d39878e03e",
        "sha256:23abaf492dd55c2c5eb3d3d1327fc2305fa09bd98b49799ae569d997ee2cd1eb",
        "sha256:0b37de2b4799d9c448d68e13651f2ef8c159905bb1c2e05ff6d6cd9d132aa136",
        "sha256:fb70e78d873963812c353543e1f8f8f761ba4ace5434f981c8c94d57fee7eb49",
        "sha256:00cbbc267eb550345ed678076bbc74077e2a4aa8abbde79fbee2eab48803d137",
        "sha256:8b787773e095a506177d368f4b8215db36718efe582d074652f6b1cc703cd7eb",
        "sha256:38715226797fb1b163dba46146011cb9b25e285321e0ef6942cf36938e42f8a1",
        "sha256:c50f7beee5bd62f7c205a697b3d64fc4d62db9bc9c5d18ee969928c15d4549c5",
        "sha256:1b9f09264803f578d7f3b536def652fae8312e31a19f91a2763277e16310c28b",
        "sha256:678b73e1b27f2fd3cd576351647a7b15382cbce347b309fd7c79a683f794efa0",
        "sha256:344e77f959ab2d1455e48c7fbc27e90ef57e61b99c1cb412ec05aef2ecb6eb26",
        "sha256:1f4cd7532416ad6194b5e020022c85c5102a5f6316a9315dd15fe80e85b55784",
        "sha256:3d0633dfa42cf6ea38ea6a80ee1177343df959885a6d19c61daf399b75a57aca",
        "sha256:3e161ddddadb4bd3179a10eaca99c3197796c90379952ad533e5eea9ecb190da",
        "sha256:529ef8164f99c69088c4e0a89870a2f6e782da9a1580b6e7de4a682d7b8f95ec",
        "sha256:2bef818babd647857c813094919ca259190a1d2c5b76c073b1ccd4ed14a13ce0",
        "sha256:82010eaa38a47a38da295d8eae223adc404b9013d6ef739494c835907fc93b20",
        "sha256:adab4b4d456fa3a02fbea108ad872350f9cc808935c532c6f73a8194a23771ef",
        "sha256:1e2b3300133ae348852843ef603d27262866bd9459f5dc30d278f52af35df588",
        "sha256:f1717085bf6d3bcb5154b2efde5ef34bf88953808ff811fa4df9829e9915c918",
        "sha256:16b78615f63b0d9c335288d21ed1611e967cc565dc07fe9010427449abfee4eb",
        "sha256:8cf8ea3d9531589f7d7f5a6f7e1e73ceddf2b0e7c4392eed19f40a6ce410a287",
        "sha256:6eb4931daf09d9805ea9b37ce7b0cda33f899c26390de3fc8c69fa6d60a742ba",
        "sha256:44527add0de9459f6e826b8279322562115c05a9d30764bb7a8b42eeb239181f",
        "sha256:e4c281e085f694be75acfbfac2b50c4b6559cb597d1a3d97d143a39d3e6c2619",
        "sha256:b015a3602a5ac72ca5b40df994c39bfe8226be001c33e94fa7c5820a40ee801f",
        "sha256:e6980abefcf4a6c44d6ca04cc6ff629f27d9d084642865be93a59b6ab6920657",
        "sha256:17023050039169d79984a18b205621ddbd4a1a7af491d2e5b6c532d0a5f4021f",
        "sha256:705edd1c71a128d007d80bedfb490dfb1c48fa210e19d5c0cf51519b6f882893",
        "sha256:9498029c89e5c044d30d4f5a4a2bd9937257800d5a8fde8b629c1aec43253ec2",
        "sha256:7d772b43c7a0796c8416f7bfac64b19dd4283e7b49018a0ad925c6d2c339b9dd",
        "sha256:dc49ec09232211768ed37ab1d8ccd14b380ecb7fc03917917e3d928ae7007ae5",
        "sha256:bbe7ab2778a5bc26228958320e5b0f84580de55726e393b975a8b9b5e4a0c62d",
        "sha256:e6a331b0bf52e609a456de0e0e3f9f48f8dc14c17759573aef321dc44eaa8b04",
        "sha256:dc696c45a9998ba3b27436689073060d9dde914959bdc47c3032bd690e1319c0",
        "sha256:7b2a2ce14e5ba3936ef7cfbc841731cfa3549a998eb8207dc928b465dbc2be9d",
        "sha256:caf8901d1ca869999ea093816709955e7eb73a49830052309d08362d0f819d03"
    ],

the nix2container-gcr.io-cloushell-images-cloudshell.json looks like this:

{
        "image-config": {},
        "layers": [
                {
                        "digest": "sha256:cbfe985b5bc1d578be10f1863a87a997158bf3c8cddcbf1f25d049ad210fa20b",
                        "size": 0,
                        "diff_ids": "sha256:3fe0c8c55320679dedec17005c5cbc920ebff509f8cd232752e8a8bdb59fe3a5",
                        "mediatype": "application/vnd.oci.image.layer.v1.tar+gzip",
                        "layer-path": "/nix/store/izrfqs2j0qfswrcg0nl7xb4gn18zz9wg-docker-image-gcr.io-cloudshell-images-cloudshell/cbfe985b5bc1d578be10f1863a87a997158bf3c8cddcbf1f25d049ad210fa20b"
                },
                {
                        "digest": "sha256:2e1bfe63320ec22cf000ba264ac09c48c11a37a1c175a50f6fd314fc9862bd40",
                        "size": 0,
                        "diff_ids": "sha256:fbf6cc502eb6bb2f67f0d3ffefcc0551630ff8a2b8116d22f4eea3e7e8e09d3c",
                        "mediatype": "application/vnd.oci.image.layer.v1.tar+gzip",
                        "layer-path": "/nix/store/izrfqs2j0qfswrcg0nl7xb4gn18zz9wg-docker-image-gcr.io-cloudshell-images-cloudshell/2e1bfe63320ec22cf000ba264ac09c48c11a37a1c175a50f6fd314fc9862bd40"
                },
                {
                        "digest": "sha256:44b9aba2766a5ce891f7f9d8a076a48a71fda78a724c58dbc70567088b749d2e",
                        "size": 0,
                        "diff_ids": "sha256:dc4a79ee54526ccef7f437682237373397c0db126a4789dc40db9a4261f2e7d1",
                        "mediatype": "application/vnd.oci.image.layer.v1.tar+gzip",
                        "layer-path": "/nix/store/izrfqs2j0qfswrcg0nl7xb4gn18zz9wg-docker-image-gcr.io-cloudshell-images-cloudshell/44b9aba2766a5ce891f7f9d8a076a48a71fda78a724c58dbc70567088b749d2e"
                },
                ...

notabley the size apears always as 0.

[nlewo]

When i built your image, the resulting image contains 74 layers:

more /nix/store/b7qc6y2q4fx93qgbv2z82nc3rfkzhgj8-image-mycloudshell.json | grep digest | wc -l
74

which is the same number than the upstream image:

nix run nixpkgs#skopeo -- inspect  docker://gcr.io/cloudshell-images/cloudshell | jq .Layers | grep sha256 | wc -l
74

The size should not be 0 but it's not really important since it is only used to display the progress bar when pushing the image to a registry.

I actually don't really understand your issues. Could you please provide an example which fails at some point?

btw, i'm wondering what this image contains ;)

du -hs /nix/store/izrfqs2j0qfswrcg0nl7xb4gn18zz9wg-docker-image-gcr.io-cloudshell-images-cloudshell
7.5G    /nix/store/izrfqs2j0qfswrcg0nl7xb4gn18zz9wg-docker-image-gcr.io-cloudshell-images-cloudshell

[angerman]

@nlewo alright, sure. So, google provides (as part of their suite of tools) a cloud IDE (similar to e.g. gitpod, github codespaces, ...). This basically launches the cloudhshell image on their infrastructure, and provides you an IDE ontop (they use Theia iirc).

Now of course you'd want

• (a) some nix in that image (because, why not, and maybe you'd like to query some nix store...)
• (b) effecitvely a copy of your nix-shell in there as well, so that you have all the tools needed to work.

and they do allow custom images (but they must start from their official coudshell image). The documentation is here: https://cloud.google.com/shell/docs/customizing-container-image, and if you click the guide me link it takes you into a cloud shell to create a custom image (a bit meta, I know).

Once that image is built you can then launch a shell in googles cloud services with a link like this:

https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/nlewo/nix2container&cloudshell_image=gcr.io/project-id/docker-image-name

To push an image to with nix2container to gcr (google container registry), one needs to get some credentials, which can be obtained from using gcloud auth print-access-token in the cloud shell. And then using the skopeo login method, with username oauth2accesstoken, and the token for the password.

However, creating any such image with nix2container (even a bare one), fails to load when opened via the above link.

I have create both images here: https://console.cloud.google.com/gcr/images/spatial-ship-359809

As such

https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/nlewo/nix2container&cloudshell_image=gcr.io/spatial-ship-359809/mycloudshell-gog

should launch an editor for this repository using the google created image (basically a docker file with only a FROM line)

and

https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/nlewo/nix2container&cloudshell_image=gcr.io/spatial-ship-359809/mycloudshell-nix

should launch an editor for this repository using the nix2container created image.

[angerman]

And while the fist link works (opens a cloud shell for this repository), the second one fails with an ominous:

The image requested is either private or does not exist. Cloud Shell does not support temporary environments with private images.

which makes little sense.

Now, if we then go and use the nix2container generated image as a base for the docker file:

FROM gcr.io/spatial-ship-359809/mycloudshell-nix:latest

and build that image. Push it to the gcr, and try to use it (https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/nlewo/nix2container&cloudshell_image=gcr.io/spatial-ship-359809/mycloudshell-nix-gog), we instead get this lovely error:

Cloud Shell is experiencing some issues provisioning a VM to you. Please try again in a few minutes.

which won't go away, even after hours.

Hence, something with the generated images is quite perplexing:

• gcr.io/spatial-ship-359809/mycloudshell-gog:latest -- works
• gcr.io/spatial-ship-359809/mycloudshell-nix:latest -- fails to be found 😕
• gcr.io/spatial-ship-359809/mycloudshell-nix-gog:latest -- is found, but fails to launch.

hence my (rather unsuccessful) quest so far to figure out what exactly is different among them.

[angerman]

@blaggacao I'm afraid that has no effect :-/

[adrian-gierakowski]

@angerman have you tired using buildImage from nixpkgs?

[nlewo]

As suggested by @adrian-gierakowski it would be nice to try with nixpkgs.dockerTools.buildImage: these functions are much more robust than nix2container ones (which are younger).

[adrian-gierakowski]

To isolate the problem I’d build with dockertools.buildImage first and push with standard docker client. If that works then build another image but push with skopeo. If that works the try dockerTools.streamLayeredImage. Then we could try to compare what’s different between the images which worked and the ones which didn’t and what dockertools does differently to nix2container