Running CharlieCloud fails with "ch-run: error: No such file or directory (ch-run.c:199:2)"

[olifre]

Trying with current master, I get:

olifre@lisa1 ~/containers/jessie $ ~/charliecloud/charliecloud/bin/ch-run -vv ubuntu/hello -- /bin/bash
newroot: ubuntu/hello
container uid: 1000
container gid: 1007
private /tmp: 0
setup_namespaces 493: uids=1000,1000,1000, gids=1007,1007,1007 + 10,11,16,18,19,27,35,85,100,250,979,985,993,1007,1010,1011,1016,1019,1022,1024,1028
setup_namespaces 495: uids=65534,65534,65534, gids=65534,65534,65534 + 65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534
setup_namespaces 510: uids=1000,1000,1000, gids=65534,65534,65534 + 65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534
setup_namespaces 518: uids=1000,1000,1000, gids=1007,1007,1007 + 65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,1007,65534,65534,65534,65534,65534,65534,65534
enter_udss 176: uids=1000,1000,1000, gids=1007,1007,1007 + 65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,65534,1007,65534,65534,65534,65534,65534,65534,65534
ch-run: error: No such file or directory (ch-run.c:245:2)

The image was built following the tutorial, i.e.:

~/charliecloud/charliecloud/bin/ch-build -t hello ~/charliecloud
~/charliecloud/charliecloud/bin/ch-docker2tar hello .
~/charliecloud/charliecloud/bin/ch-tar2dir hello.tar.gz ubuntu
~/charliecloud/charliecloud/bin/ch-run -vv ubuntu/hello -- /bin/bash

(it currently lives in my home, but I plan to build a proper package for Gentoo in the near future, maybe also for other distros)

Checking with strace, I see:

$ strace -f ~/charliecloud/charliecloud/bin/ch-run -vv ubuntu/hello -- /bin/bash
[...]
mount("ubuntu/hello", "ubuntu/hello", NULL, MS_BIND|MS_REC|MS_PRIVATE, NULL) = 0
mount("/dev", "ubuntu/hello/dev", NULL, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
mount("/etc/passwd", "ubuntu/hello/etc/passwd", NULL, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
mount("/etc/group", "ubuntu/hello/etc/group", NULL, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
mount("/etc/hosts", "ubuntu/hello/etc/hosts", NULL, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
mount("/etc/resolv.conf", "ubuntu/hello/etc/resolv.conf", NULL, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
mount("/proc", "ubuntu/hello/proc", NULL, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
mount("/sys", "ubuntu/hello/sys", NULL, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0
mount("/tmp", "ubuntu/hello/tmp", NULL, MS_BIND|MS_REC, NULL) = 0
mount(NULL, "ubuntu/hello/home", "tmpfs", 0, "size=4m") = 0
mkdir("ubuntu/hello/home/olifre", 0755) = 0
mount("/home/olifre", "ubuntu/hello/home/olifre", NULL, MS_BIND|MS_REC, NULL) = 0
stat("ubuntu/hello/usr/bin/ch-ssh", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
readlink("/proc/self/exe", "/home/olifre/charliecloud/charli"..., 4096) = 49
mount("/home/olifre/charliecloud/charliecloud/bin/ch-ssh", "ubuntu/hello/usr/bin/ch-ssh", NULL, MS_BIND, NULL) = 0
mount("ubuntu", "ubuntu", NULL, MS_BIND|MS_REC|MS_PRIVATE, NULL) = 0
chdir("ubuntu")                         = 0
mount("ubuntu", "/", NULL, MS_MOVE, NULL) = -1 ENOENT (No such file or directory)
write(2, "ch-run: ", 8ch-run: )                 = 8
write(2, "error: No such file or directory"..., 50error: No such file or directory (ch-run.c:245:2)
) = 50
exit_group(1)                           = ?
+++ exited with 1 +++

It's bit strange to see:

mount("ubuntu", "ubuntu", NULL, MS_BIND|MS_REC|MS_PRIVATE, NULL) = 0

in there, when the container actually lives in ubuntu/hello... Any ideas what could be wrong?

[reidpr]

This may be a duplicate of #6; does it work with absolute paths?

[olifre]

This may be a duplicate of #6; does it work with absolute paths?

You are right, that fixes it!

And indeed, even though that might now be a bit off-topic here: I just now also found my mistake (which produced almost the same error message, so I thought it was the same issue) when using Singularity-built images, or docker-built images, extracted manually. Reading through https://github.com/hpc/charliecloud/blob/master/bin/ch-tar2dir , I learnt that you need /oldroot to exist (and place a sentinel within).

So I will now add /oldroot to all images we are building to have them compatible both with Singularity and Charliecloud.

There's still a different problem when using CVMFS, though - I'll have a deeper look and provide more info in a separate issue once I have collected enough informaton.

Thanks! :-)