elpdt852
commented
11 months ago Hi @docteurklein, welcome and cool project!
Note that nlewo/nix2container is completely different from nix-snapshotter/pkg/nix2container. If you use nlewo/nix2container you are creating a container image with normal layers that nix-snapshotter has backwards compatibility for, i.e. these are not Nix native layers that bind mount from your host's Nix store.
I see that you are applying the nix-snapshotter overlay: https://github.com/docteurklein/nixok/blob/208201b5ca137982abcc388290c7eba9c46172e5/flake.nix#L96
That gives you the "client-side" code to build nix-snapshotter compatible images. But on the "server-side", kubernetes doesn't understand them natively. How are you setting that up?
The kubelet needs to configured to use nix-snapshotter as the image service: https://github.com/pdtpartners/nix-snapshotter/blob/v0.1.0/modules/nixos/kubernetes.nix#L10
And containerd needs to be configured to use nix-snapshotter as the snapshotter: https://github.com/pdtpartners/nix-snapshotter/blob/v0.1.0/modules/common/nix-snapshotter-lib.nix#L77-L89
We provide NixOS modules for containerd+nix-snapshotter, but not for kubernetes as everyone runs that differently. Take a look at how our demo VM is set up end-to-end: https://github.com/pdtpartners/nix-snapshotter/blob/v0.1.0/modules/nixos/vm.nix#L11
docteurklein
Hello! First, thank you so much for this, I love the idea! And the feedback loop is soooooo much faster!
I'm struggling to find the reason behind this error message, which only appears in this particular setup: https://github.com/docteurklein/nixok/commit/208201b5ca137982abcc388290c7eba9c46172e5#diff-206b9ce276ab5971a2489d75eb1b12999d4bf3843b7988cbe8d687cfde61dea0R114
When I'm shipping the same config but using nix2container directly, my pod starts, but when I use nix-snapshotter, I get this error message:
Any idea?
details
```json { "ID": "d40b2284de669d166a8a1fac32971cdb88c5e382b1d4d172cab6f5854278fed3", "Labels": { "app": "s1", "io.cri-containerd.kind": "sandbox", "io.kubernetes.pod.name": "s1-648769894-xs96h", "io.kubernetes.pod.namespace": "default", "io.kubernetes.pod.uid": "0b8c18e0-87ca-41ac-a0ab-e58df9d79a14", "pod-template-hash": "648769894" }, "Image": "docker.io/library/pause:latest", "Runtime": { "Name": "io.containerd.runc.v2", "Options": { "type_url": "containerd.runc.v1.Options", "value": "SAE=" } }, "SnapshotKey": "d40b2284de669d166a8a1fac32971cdb88c5e382b1d4d172cab6f5854278fed3", "Snapshotter": "nix", "CreatedAt": "2023-10-03T13:32:31.850481864Z", "UpdatedAt": "2023-10-03T13:32:31.850481864Z", "Extensions": { "io.cri-containerd.sandbox.metadata": { "type_url": "github.com/containerd/cri/pkg/store/sandbox/Metadata", "value": "eyJWZXJzaW9uIjoidjEiLCJNZXRhZGF0YSI6eyJJRCI6ImQ0MGIyMjg0ZGU2NjlkMTY2YThhMWZhYzMyOTcxY2RiODhjNWUzODJiMWQ0ZDE3MmNhYjZmNTg1NDI3OGZlZDMiLCJOYW1lIjoiczEtNjQ4NzY5ODk0LXhzOTZoX2RlZmF1bHRfMGI4YzE4ZTAtODdjYS00MWFjLWEwYWItZTU4ZGY5ZDc5YTE0XzAiLCJDb25maWciOnsibWV0YWRhdGEiOnsibmFtZSI6InMxLTY0ODc2OTg5NC14czk2aCIsInVpZCI6IjBiOGMxOGUwLTg3Y2EtNDFhYy1hMGFiLWU1OGRmOWQ3OWExNCIsIm5hbWVzcGFjZSI6ImRlZmF1bHQifSwiaG9zdG5hbWUiOiJzMS02NDg3Njk4OTQteHM5NmgiLCJsb2dfZGlyZWN0b3J5IjoiL3Zhci9sb2cvcG9kcy9kZWZhdWx0X3MxLTY0ODc2OTg5NC14czk2aF8wYjhjMThlMC04N2NhLTQxYWMtYTBhYi1lNThkZjlkNzlhMTQiLCJkbnNfY29uZmlnIjp7InNlcnZlcnMiOlsiMTAuMC4wLjI1NCJdLCJzZWFyY2hlcyI6WyJkZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwic3ZjLmNsdXN0ZXIubG9jYWwiLCJjbHVzdGVyLmxvY2FsIl0sIm9wdGlvbnMiOlsibmRvdHM6NSJdfSwibGFiZWxzIjp7ImFwcCI6InMxIiwiaW8ua3ViZXJuZXRlcy5wb2QubmFtZSI6InMxLTY0ODc2OTg5NC14czk2aCIsImlvLmt1YmVybmV0ZXMucG9kLm5hbWVzcGFjZSI6ImRlZmF1bHQiLCJpby5rdWJlcm5ldGVzLnBvZC51aWQiOiIwYjhjMThlMC04N2NhLTQxYWMtYTBhYi1lNThkZjlkNzlhMTQiLCJwb2QtdGVtcGxhdGUtaGFzaCI6IjY0ODc2OTg5NCJ9LCJhbm5vdGF0aW9ucyI6eyJrdWJlcm5ldGVzLmlvL2NvbmZpZy5zZWVuIjoiMjAyMy0xMC0wM1QxNTozMjozMS41MjI0NDEzNjQrMDI6MDAiLCJrdWJlcm5ldGVzLmlvL2NvbmZpZy5zb3VyY2UiOiJhcGkiLCJwcm9maWxlcy5ncmFmYW5hLmNvbS9jcHUucG9ydF9uYW1lIjoiaHR0cC1tZXRyaWNzIiwicHJvZmlsZXMuZ3JhZmFuYS5jb20vY3B1LnNjcmFwZSI6InRydWUiLCJwcm9maWxlcy5ncmFmYW5hLmNvbS9tZW1vcnkucG9ydF9uYW1lIjoiaHR0cC1tZXRyaWNzIiwicHJvZmlsZXMuZ3JhZmFuYS5jb20vbWVtb3J5LnNjcmFwZSI6InRydWUifSwibGludXgiOnsiY2dyb3VwX3BhcmVudCI6Ii9rdWJlcG9kcy5zbGljZS9rdWJlcG9kcy1iZXN0ZWZmb3J0LnNsaWNlL2t1YmVwb2RzLWJlc3RlZmZvcnQtcG9kMGI4YzE4ZTBfODdjYV80MWFjX2EwYWJfZTU4ZGY5ZDc5YTE0LnNsaWNlIiwic2VjdXJpdHlfY29udGV4dCI6eyJuYW1lc3BhY2Vfb3B0aW9ucyI6eyJwaWQiOjF9LCJzdXBwbGVtZW50YWxfZ3JvdXBzIjpbMTAwMF0sInNlY2NvbXAiOnt9fSwib3ZlcmhlYWQiOnt9LCJyZXNvdXJjZXMiOnsiY3B1X3BlcmlvZCI6MTAwMDAwLCJjcHVfc2hhcmVzIjoyfX19LCJOZXROU1BhdGgiOiIvdmFyL3J1bi9uZXRucy9jbmktNTg3Y2I2ZjQtODE1NS1iNWIyLTEwMmQtZjVjOGI5MDRkMWRlIiwiSVAiOiIxMC4xLjAuMjciLCJBZGRpdGlvbmFsSVBzIjpudWxsLCJSdW50aW1lSGFuZGxlciI6IiIsIkNOSVJlc3VsdCI6eyJJbnRlcmZhY2VzIjp7ImV0aDAiOnsiSVBDb25maWdzIjpbeyJJUCI6IjEwLjEuMC4yNyIsIkdhdGV3YXkiOiIxMC4xLjAuMSJ9XSwiTWFjIjoiZmU6NmY6ZDI6ZGI6YjU6MTMiLCJTYW5kYm94IjoiL3Zhci9ydW4vbmV0bnMvY25pLTU4N2NiNmY0LTgxNTUtYjViMi0xMDJkLWY1YzhiOTA0ZDFkZSJ9LCJsbyI6eyJJUENvbmZpZ3MiOlt7IklQIjoiMTI3LjAuMC4xIiwiR2F0ZXdheSI6IiJ9LHsiSVAiOiI6OjEiLCJHYXRld2F5IjoiIn1dLCJNYWMiOiIwMDowMDowMDowMDowMDowMCIsIlNhbmRib3giOiIvdmFyL3J1bi9uZXRucy9jbmktNTg3Y2I2ZjQtODE1NS1iNWIyLTEwMmQtZjVjOGI5MDRkMWRlIn0sIm15bmV0Ijp7IklQQ29uZmlncyI6bnVsbCwiTWFjIjoiYTI6OTE6NDY6NTM6MjU6NzciLCJTYW5kYm94IjoiIn0sInZldGg3NmIwY2VkNyI6eyJJUENvbmZpZ3MiOm51bGwsIk1hYyI6IjQyOjFiOjJhOjgzOjFkOjgyIiwiU2FuZGJveCI6IiJ9fSwiRE5TIjpbe30se31dLCJSb3V0ZXMiOlt7ImRzdCI6IjEwLjEuMC4wLzE2In0seyJkc3QiOiIwLjAuMC4wLzAiLCJndyI6IjEwLjEuMC4xIn1dfSwiUHJvY2Vzc0xhYmVsIjoiIn19" } }, "SandboxID": "", "Spec": { "ociVersion": "1.1.0", "process": { "user": { "uid": 0, "gid": 0, "additionalGids": [ 1000 ] }, "args": [ "/bin/pause" ], "cwd": "/", "capabilities": { "bounding": [ "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER", "CAP_MKNOD", "CAP_NET_RAW", "CAP_SETGID", "CAP_SETUID", "CAP_SETFCAP", "CAP_SETPCAP", "CAP_NET_BIND_SERVICE", "CAP_SYS_CHROOT", "CAP_KILL", "CAP_AUDIT_WRITE" ], "effective": [ "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER", "CAP_MKNOD", "CAP_NET_RAW", "CAP_SETGID", "CAP_SETUID", "CAP_SETFCAP", "CAP_SETPCAP", "CAP_NET_BIND_SERVICE", "CAP_SYS_CHROOT", "CAP_KILL", "CAP_AUDIT_WRITE" ], "permitted": [ "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER", "CAP_MKNOD", "CAP_NET_RAW", "CAP_SETGID", "CAP_SETUID", "CAP_SETFCAP", "CAP_SETPCAP", "CAP_NET_BIND_SERVICE", "CAP_SYS_CHROOT", "CAP_KILL", "CAP_AUDIT_WRITE" ] }, "noNewPrivileges": true, "oomScoreAdj": -998 }, "root": { "path": "rootfs", "readonly": true }, "hostname": "s1-648769894-xs96h", "mounts": [ { "destination": "/proc", "type": "proc", "source": "proc", "options": [ "nosuid", "noexec", "nodev" ] }, { "destination": "/dev", "type": "tmpfs", "source": "tmpfs", "options": [ "nosuid", "strictatime", "mode=755", "size=65536k" ] }, { "destination": "/dev/pts", "type": "devpts", "source": "devpts", "options": [ "nosuid", "noexec", "newinstance", "ptmxmode=0666", "mode=0620", "gid=5" ] }, { "destination": "/dev/mqueue", "type": "mqueue", "source": "mqueue", "options": [ "nosuid", "noexec", "nodev" ] }, { "destination": "/sys", "type": "sysfs", "source": "sysfs", "options": [ "nosuid", "noexec", "nodev", "ro" ] }, { "destination": "/dev/shm", "type": "bind", "source": "/run/containerd/io.containerd.grpc.v1.cri/sandboxes/d40b2284de669d166a8a1fac32971cdb88c5e382b1d4d172cab6f5854278fed3/shm", "options": [ "rbind", "ro", "nosuid", "nodev", "noexec" ] }, { "destination": "/etc/resolv.conf", "type": "bind", "source": "/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/d40b2284de669d166a8a1fac32971cdb88c5e382b1d4d172cab6f5854278fed3/resolv.conf", "options": [ "rbind", "ro", "nosuid", "nodev", "noexec" ] } ], "annotations": { "io.kubernetes.cri.container-type": "sandbox", "io.kubernetes.cri.sandbox-cpu-period": "100000", "io.kubernetes.cri.sandbox-cpu-quota": "0", "io.kubernetes.cri.sandbox-cpu-shares": "2", "io.kubernetes.cri.sandbox-id": "d40b2284de669d166a8a1fac32971cdb88c5e382b1d4d172cab6f5854278fed3", "io.kubernetes.cri.sandbox-log-directory": "/var/log/pods/default_s1-648769894-xs96h_0b8c18e0-87ca-41ac-a0ab-e58df9d79a14", "io.kubernetes.cri.sandbox-memory": "0", "io.kubernetes.cri.sandbox-name": "s1-648769894-xs96h", "io.kubernetes.cri.sandbox-namespace": "default", "io.kubernetes.cri.sandbox-uid": "0b8c18e0-87ca-41ac-a0ab-e58df9d79a14" }, "linux": { "resources": { "devices": [ { "allow": false, "access": "rwm" } ], "cpu": { "shares": 2 } }, "cgroupsPath": "kubepods-besteffort-pod0b8c18e0_87ca_41ac_a0ab_e58df9d79a14.slice:cri-containerd:d40b2284de669d166a8a1fac32971cdb88c5e382b1d4d172cab6f5854278fed3", "namespaces": [ { "type": "pid" }, { "type": "ipc" }, { "type": "uts" }, { "type": "mount" }, { "type": "network", "path": "/var/run/netns/cni-587cb6f4-8155-b5b2-102d-f5c8b904d1de" } ], "seccomp": { "defaultAction": "SCMP_ACT_ERRNO", "architectures": [ "SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32" ], "syscalls": [ { "names": [ "accept", "accept4", "access", "adjtimex", "alarm", "bind", "brk", "capget", "capset", "chdir", "chmod", "chown", "chown32", "clock_adjtime", "clock_adjtime64", "clock_getres", "clock_getres_time64", "clock_gettime", "clock_gettime64", "clock_nanosleep", "clock_nanosleep_time64", "close", "close_range", "connect", "copy_file_range", "creat", "dup", "dup2", "dup3", "epoll_create", "epoll_create1", "epoll_ctl", "epoll_ctl_old", "epoll_pwait", "epoll_pwait2", "epoll_wait", "epoll_wait_old", "eventfd", "eventfd2", "execve", "execveat", "exit", "exit_group", "faccessat", "faccessat2", "fadvise64", "fadvise64_64", "fallocate", "fanotify_mark", "fchdir", "fchmod", "fchmodat", "fchown", "fchown32", "fchownat", "fcntl", "fcntl64", "fdatasync", "fgetxattr", "flistxattr", "flock", "fork", "fremovexattr", "fsetxattr", "fstat", "fstat64", "fstatat64", "fstatfs", "fstatfs64", "fsync", "ftruncate", "ftruncate64", "futex", "futex_time64", "futex_waitv", "futimesat", "getcpu", "getcwd", "getdents", "getdents64", "getegid", "getegid32", "geteuid", "geteuid32", "getgid", "getgid32", "getgroups", "getgroups32", "getitimer", "getpeername", "getpgid", "getpgrp", "getpid", "getppid", "getpriority", "getrandom", "getresgid", "getresgid32", "getresuid", "getresuid32", "getrlimit", "get_robust_list", "getrusage", "getsid", "getsockname", "getsockopt", "get_thread_area", "gettid", "gettimeofday", "getuid", "getuid32", "getxattr", "inotify_add_watch", "inotify_init", "inotify_init1", "inotify_rm_watch", "io_cancel", "ioctl", "io_destroy", "io_getevents", "io_pgetevents", "io_pgetevents_time64", "ioprio_get", "ioprio_set", "io_setup", "io_submit", "io_uring_enter", "io_uring_register", "io_uring_setup", "ipc", "kill", "landlock_add_rule", "landlock_create_ruleset", "landlock_restrict_self", "lchown", "lchown32", "lgetxattr", "link", "linkat", "listen", "listxattr", "llistxattr", "_llseek", "lremovexattr", "lseek", "lsetxattr", "lstat", "lstat64", "madvise", "membarrier", "memfd_create", "memfd_secret", "mincore", "mkdir", "mkdirat", "mknod", "mknodat", "mlock", "mlock2", "mlockall", "mmap", "mmap2", "mprotect", "mq_getsetattr", "mq_notify", "mq_open", "mq_timedreceive", "mq_timedreceive_time64", "mq_timedsend", "mq_timedsend_time64", "mq_unlink", "mremap", "msgctl", "msgget", "msgrcv", "msgsnd", "msync", "munlock", "munlockall", "munmap", "name_to_handle_at", "nanosleep", "newfstatat", "_newselect", "open", "openat", "openat2", "pause", "pidfd_open", "pidfd_send_signal", "pipe", "pipe2", "pkey_alloc", "pkey_free", "pkey_mprotect", "poll", "ppoll", "ppoll_time64", "prctl", "pread64", "preadv", "preadv2", "prlimit64", "process_mrelease", "pselect6", "pselect6_time64", "pwrite64", "pwritev", "pwritev2", "read", "readahead", "readlink", "readlinkat", "readv", "recv", "recvfrom", "recvmmsg", "recvmmsg_time64", "recvmsg", "remap_file_pages", "removexattr", "rename", "renameat", "renameat2", "restart_syscall", "rmdir", "rseq", "rt_sigaction", "rt_sigpending", "rt_sigprocmask", "rt_sigqueueinfo", "rt_sigreturn", "rt_sigsuspend", "rt_sigtimedwait", "rt_sigtimedwait_time64", "rt_tgsigqueueinfo", "sched_getaffinity", "sched_getattr", "sched_getparam", "sched_get_priority_max", "sched_get_priority_min", "sched_getscheduler", "sched_rr_get_interval", "sched_rr_get_interval_time64", "sched_setaffinity", "sched_setattr", "sched_setparam", "sched_setscheduler", "sched_yield", "seccomp", "select", "semctl", "semget", "semop", "semtimedop", "semtimedop_time64", "send", "sendfile", "sendfile64", "sendmmsg", "sendmsg", "sendto", "setfsgid", "setfsgid32", "setfsuid", "setfsuid32", "setgid", "setgid32", "setgroups", "setgroups32", "setitimer", "setpgid", "setpriority", "setregid", "setregid32", "setresgid", "setresgid32", "setresuid", "setresuid32", "setreuid", "setreuid32", "setrlimit", "set_robust_list", "setsid", "setsockopt", "set_thread_area", "set_tid_address", "setuid", "setuid32", "setxattr", "shmat", "shmctl", "shmdt", "shmget", "shutdown", "sigaltstack", "signalfd", "signalfd4", "sigprocmask", "sigreturn", "socketcall", "socketpair", "splice", "stat", "stat64", "statfs", "statfs64", "statx", "symlink", "symlinkat", "sync", "sync_file_range", "syncfs", "sysinfo", "tee", "tgkill", "time", "timer_create", "timer_delete", "timer_getoverrun", "timer_gettime", "timer_gettime64", "timer_settime", "timer_settime64", "timerfd_create", "timerfd_gettime", "timerfd_gettime64", "timerfd_settime", "timerfd_settime64", "times", "tkill", "truncate", "truncate64", "ugetrlimit", "umask", "uname", "unlink", "unlinkat", "utime", "utimensat", "utimensat_time64", "utimes", "vfork", "vmsplice", "wait4", "waitid", "waitpid", "write", "writev" ], "action": "SCMP_ACT_ALLOW" }, { "names": [ "socket" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 40, "op": "SCMP_CMP_NE" } ] }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 0, "op": "SCMP_CMP_EQ" } ] }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 8, "op": "SCMP_CMP_EQ" } ] }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 131072, "op": "SCMP_CMP_EQ" } ] }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 131080, "op": "SCMP_CMP_EQ" } ] }, { "names": [ "personality" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 4294967295, "op": "SCMP_CMP_EQ" } ] }, { "names": [ "process_vm_readv", "process_vm_writev", "ptrace" ], "action": "SCMP_ACT_ALLOW" }, { "names": [ "arch_prctl", "modify_ldt" ], "action": "SCMP_ACT_ALLOW" }, { "names": [ "chroot" ], "action": "SCMP_ACT_ALLOW" }, { "names": [ "clone" ], "action": "SCMP_ACT_ALLOW", "args": [ { "index": 0, "value": 2114060288, "op": "SCMP_CMP_MASKED_EQ" } ] }, { "names": [ "clone3" ], "action": "SCMP_ACT_ERRNO", "errnoRet": 38 } ] }, "maskedPaths": [ "/proc/acpi", "/proc/asound", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/sys/firmware", "/proc/scsi" ], "readonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] } } } ```