Minimal example not working

afiore commented 1 year ago

Hi there,

I am trying to use Arion but I have run into an issue when executing the minimal example presented the project docs.

I am using Nix on an Ubuntu machine, and I have installed Arion via home-manager.

When I run arion up in the minimal example folder, I get the following error:

➜ arion up
trace: warning: The check argument to evalModules is deprecated. Please set config._module.check instead.
/nix/store/mhqwamwimfilhwfsglsgrr0g7k7d6akv-docker-compose.yaml
Starting minimal_webserver_1 ... error

ERROR: for minimal_webserver_1  Cannot start service webserver: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown

ERROR: for webserver  Cannot start service webserver: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown
ERROR: Encountered errors while bringing up the project.

Details:

the attached docker-compose file generated by Airon seems correct, as I can access the python executable in /nix/store/fkcl1wzq3106qqgl84bhgk1lp56q6bzg-python3-3.10.7/bin/python.
Docker and docker-compose haven't been built / installed via nix. Their respective versions are
- Docker version 20.10.12, build 20.10.12-0ubuntu4
- Docker Compose version v2.5.0

Generated docker-compose.yaml:

{
  "services": {
    "webserver": {
      "command": [
        "sh",
        "-c",
        "cd \"$$WEB_ROOT\"\n/nix/store/fkcl1wzq3106qqgl84bhgk1lp56q6bzg-python3-3.10.7/bin/python -m http.server\n"
      ],
      "environment": {
        "NIX_REMOTE": "",
        "WEB_ROOT": "/nix/store/qdx3cy9691y429l0qw3hgbshyh9g1v71-nix-2.11.0-doc/share/doc/nix/manual"
      },
      "image": "webserver:xa0d0b0bq98d6b7ppqsmbacir5cqz5ax",
      "ports": [
        "8000:8000"
      ],
      "sysctls": {},
      "volumes": [
        "/nix/store:/nix/store:ro"
      ]
    }
  },
  "version": "3.4",
  "x-arion": {
    "images": [
      {
        "imageExe": "/nix/store/vjdc5izv8gr48bb5bdkspvk10bqpx1ff-stream-webserver",
        "imageName": "webserver",
        "imageTag": "xa0d0b0bq98d6b7ppqsmbacir5cqz5ax"
      }
    ],
    "project": {
      "name": null
    },
    "serviceInfo": {
      "webserver": {
        "defaultExec": [
          "/bin/sh"
        ]
      }
    }
  }
}

Finally, in case this helps somehow, here is the output of docker inspect webserver:xa0d0b0bq98d6b7ppqsmbacir5cqz5ax:

[
    {
        "Id": "sha256:8a1665b0a28cceba84b7b486fa043508b8f4099522fba29b0d5207cc63e53e1c",
        "RepoTags": [
            "webserver:xa0d0b0bq98d6b7ppqsmbacir5cqz5ax"
        ],
        "RepoDigests": [],
        "Parent": "",
        "Comment": "store paths: ['/nix/store/f4vbg73h0pqmqap1c77absj6zv2kriyr-webserver-customisation-layer']",
        "Created": "1970-01-01T00:00:01Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 8433762,
        "VirtualSize": 8433762,
        "GraphDriver": {
            "Data": {
                "MergedDir": "/var/lib/docker/overlay2/2626a4ea4feee9c9c9edd92323032e12cccc4e37570af7e0e1c20b31ed2887ce/merged",
                "UpperDir": "/var/lib/docker/overlay2/2626a4ea4feee9c9c9edd92323032e12cccc4e37570af7e0e1c20b31ed2887ce/diff",
                "WorkDir": "/var/lib/docker/overlay2/2626a4ea4feee9c9c9edd92323032e12cccc4e37570af7e0e1c20b31ed2887ce/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f6bbbe496f177b097df8026a4a3805ef80c9e5af404b8efdab24f368f17cee66"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

roberth commented 1 year ago

I can think of two potential problems.

One is Config.Env is null in the image. Probably arion should set this, and not rely on the defaults, but that may interfere with non-arion base images and I don't expect the defaults to change. (this used to work)

The other is perhaps the relevant bash has been garbage collected. You're using the host store and arion doesn't manage a garbage collection root for you. This is the line that's responsible for adding bash to the config closure.

https://github.com/hercules-ci/arion/blob/865055787a799512b0c8da0522bcbc8670183a9d/src/nix/modules/service/image-recommended.nix#L15

Could you check whether the sh symlink points to an existing store path?

afiore commented 1 year ago

Thanks for your quick reply @roberth.

I am not sure I fully understand the implications of Config.Env being null.

As for the presence of a bash derivation in the nix store, I am following the trail of links and everything seems to be in its place:

$
➜ cat /nix/store/vjdc5izv8gr48bb5bdkspvk10bqpx1ff-stream-webserver
#! /nix/store/dd3713mm8wql4r2d5jxx0f58g16nfm4h-bash-5.1-p16/bin/bash -e
exec "/nix/store/vaa4s8wrizd0hyhpckilk64f79nbd0vx-stream"  /nix/store/xa0d0b0bq98d6b7ppqsmbacir5cqz5ax-webserver-conf.json "$@"

Here's the content of the bash derivation:

➜ tree /nix/store/dd3713mm8wql4r2d5jxx0f58g16nfm4h-bash-5.1-p16/    
/nix/store/dd3713mm8wql4r2d5jxx0f58g16nfm4h-bash-5.1-p16/
├── bin
│   ├── bash
│   └── sh -> bash
└── lib
    └── bash
        ├── accept
        ├── basename
        ├── csv
        ├── cut
[...]

On the other hand, one thing that caught my eyes is that the sh command used in the docker-compose.yaml file is not an absolute path pointing at the store:

  "services": {
    "webserver": {
      "command": [
        "sh",
        "-c",
        "cd \"$$WEB_ROOT\"\n/nix/store/fkcl1wzq3106qqgl84bhgk1lp56q6bzg-python3-3.10.7/bin/python -m http.server\n"
      ],

Shouldn't that be /nix/store/dd3713mm8wql4r2d5jxx0f58g16nfm4h-bash-5.1-p16/bin/sh? Is it the case that sh it's not being resolved because $PATH is empty? Can I verify that somehow?

More details below:

/nix/store/xa0d0b0bq98d6b7ppqsmbacir5cqz5ax-webserver-conf.json is:

{
  "architecture": "amd64",
  "config": {
    "Cmd": []
  },
  "os": "linux",
  "store_dir": "/nix/store",
  "from_image": null,
  "store_layers": [],
  "customisation_layer": "/nix/store/f4vbg73h0pqmqap1c77absj6zv2kriyr-webserver-customisation-layer",
  "repo_tag": "webserver:xa0d0b0bq98d6b7ppqsmbacir5cqz5ax",
  "created": "1970-01-01T00:00:01+00:00"
}

And /nix/store/f4vbg73h0pqmqap1c77absj6zv2kriyr-webserver-customisation-layer/layer.tar contains:

➜ tar tvf layer.tar                                                                    
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/store/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/store/.links/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/db/
-rw------- 0/0               0 1980-01-01 01:00 ./nix/var/nix/db/big-lock
-rw-r--r-- 0/0           45056 1980-01-01 01:00 ./nix/var/nix/db/db.sqlite
-rw------- 0/0         8388608 1980-01-01 01:00 ./nix/var/nix/db/reserved
-rw-r--r-- 0/0               2 1980-01-01 01:00 ./nix/var/nix/db/schema
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/gcroots/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/gcroots/docker/
lrwxrwxrwx 0/0               0 1980-01-01 01:00 ./nix/var/nix/gcroots/docker/7cw8n7r005dm34d37mcjm7bs4g20kzac-dummy-config.json -> /nix/store/7cw8n7r005dm34d37mcjm7bs4g20kzac-dummy-config.json
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/gcroots/per-user/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/gcroots/per-user/nixbld/
lrwxrwxrwx 0/0               0 1980-01-01 01:00 ./nix/var/nix/gcroots/profiles -> /build/old_out/nix/var/nix/profiles
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/profiles/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/profiles/per-user/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/profiles/per-user/nixbld/
drwxr-xr-x 0/0               0 1980-01-01 01:00 ./nix/var/nix/temproots/

roberth commented 1 year ago

Could you try https://github.com/hercules-ci/arion/pull/176? Or perhaps you could try to set PATH yourself first if that's easier.

roberth commented 1 year ago

Sorry, made a mistake with that. Could you try with image.rawConfig = "PATH=/bin" in services.webserver?

afiore commented 1 year ago

I have tried adding the following in services.webserver, but still no luck:

 webserver = {
      image.rawConfig = {
        Env = [ "PATH=/bin" ];
      };
      service.useHostStore = true;
      [...]

I am referring to these docs for the rawConfig format.

➜ arion up
trace: warning: The check argument to evalModules is deprecated. Please set config._module.check instead.
these 7 derivations will be built:
  /nix/store/34acdsqfman0qvcnvv22xmdl04ng5dky-webserver-base.json.drv
  /nix/store/b39sjr2lcrabjpzrm2q3wnycnb2sdf7s-dummy-config.json.drv
  /nix/store/79z7d5z1hxpj68ddw93fxnvi0pq2kgyl-closure-info.drv
  /nix/store/hcfxwbz1zwvnjg54d2xwrgz88gqbh1xq-webserver-customisation-layer.drv
  /nix/store/skmz95pjs4gqrrmjwksvdzww3zg4ygyf-webserver-conf.json.drv
  /nix/store/hyla0smc7f5vnkxx0d9anl4h366kwrzi-stream-webserver.drv
  /nix/store/fxsp858wjdv7v3irgnwdq0881sr7gwj5-docker-compose.yaml.drv
building '/nix/store/34acdsqfman0qvcnvv22xmdl04ng5dky-webserver-base.json.drv'...
building '/nix/store/b39sjr2lcrabjpzrm2q3wnycnb2sdf7s-dummy-config.json.drv'...
building '/nix/store/79z7d5z1hxpj68ddw93fxnvi0pq2kgyl-closure-info.drv'...
building '/nix/store/hcfxwbz1zwvnjg54d2xwrgz88gqbh1xq-webserver-customisation-layer.drv'...
Generating the nix database...
Warning: only the database of the deepest Nix layer is loaded.
         If you want to use nix commands in the container, it would
         be better to only have one layer that contains a nix store.
building '/nix/store/skmz95pjs4gqrrmjwksvdzww3zg4ygyf-webserver-conf.json.drv'...
{
  "architecture": "amd64",
  "config": {
    "Cmd": [],
    "Env": [
      "PATH=/bin"
    ]
  },
  "os": "linux",
  "store_dir": "/nix/store",
  "from_image": null,
  "store_layers": [],
  "customisation_layer": "/nix/store/jm2l0jrp00bsplnvn2l87k9jg1cam343-webserver-customisation-layer",
  "repo_tag": "webserver:snq6h2y3v58fxdjg9xwhjbpxbcp0sypp",
  "created": "1970-01-01T00:00:01+00:00"
}
building '/nix/store/hyla0smc7f5vnkxx0d9anl4h366kwrzi-stream-webserver.drv'...
building '/nix/store/fxsp858wjdv7v3irgnwdq0881sr7gwj5-docker-compose.yaml.drv'...
/nix/store/7hq88bajv49l9iwr12hksy3xy2jpb27a-docker-compose.yaml
No 'fromImage' provided
Creating layer 1 with customisation...
Adding manifests...
Done.
e9145c04fcca: Loading layer [==================================================>]  8.448MB/8.448MB
Loaded image: webserver:snq6h2y3v58fxdjg9xwhjbpxbcp0sypp
Removing minimal_webserver_1
Recreating 3f4a84be652e_minimal_webserver_1 ... error

ERROR: for 3f4a84be652e_minimal_webserver_1  Cannot start service webserver: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown

ERROR: for webserver  Cannot start service webserver: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown
ERROR: Encountered errors while bringing up the project.

basilgood commented 1 year ago

If it helps, I managed to make the minimal example work by replacing sh with ${pkgs.bash}/bin/sh

    webserver = {
      service.useHostStore = true;
      service.command = [
        "${pkgs.bash}/bin/sh"
        "-c"
        ''
          cd "$$WEB_ROOT"
          ${pkgs.python3}/bin/python -m http.server
        ''
      ];

afiore commented 1 year ago

@basilgood this solved it for me as well! Thanks for sharing your solution.

@roberth do you think is worth to accordingly amend the code snippets in the project documentation website? or is sh supposed to be working without fully qualified path?

roberth commented 1 year ago

Oh no, turns out the docs and the examples directory have diverged. This explains my own confusion at least. The one in examples sets enableRecommendedContents = true, which I think wasn't necessary in a pretty old version of arion. I've synced the examples.

afiore commented 1 year ago

@roberth cool! I can confirm that when I include mage.enableRecommendedContents = true; the arion up command works as expected! Closing the issue now.

hercules-ci / arion

Minimal example not working #175