Open ahuston-0 opened 2 months ago
clone this repo
Could you please prepare a minimal (or, at least, more reduced) test case, this is a bit too much to work with.
@the-sun-will-rise-tomorrow
sorry for the complexity, that repo has quite a few machines and bells and whistles. Tried to reduce it some, still seeing the same behavior when i run the below with this new repo.
Steps to reproduce the behavior:
sudo nix build .#nixosConfigurations.test-machine.config.system.build.toplevel --verbose
Thanks!
I think this is the same issue as: https://github.com/docker/for-linux/issues/1443
It looks like it's a bug in the Linux kernel which we can at best try to work around, and none of the work arounds fix the problem completely :/
Here is a kernel log excerpt from when it happens (i.e. mount
fails):
[ 7.151007] tar: page allocation failure: order:5, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
[ 7.151014] CPU: 12 PID: 2144 Comm: tar Not tainted 6.6.40 #1-NixOS
[ 7.151015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 7.151016] Call Trace:
[ 7.151018] <TASK>
[ 7.151019] dump_stack_lvl+0x47/0x70
[ 7.151023] warn_alloc+0x178/0x1f0
[ 7.151026] ? __alloc_pages_direct_compact+0x1a1/0x2b0
[ 7.151028] __alloc_pages_slowpath.constprop.0+0xd91/0xdf0
[ 7.151029] ? _raw_spin_unlock+0xe/0x30
[ 7.151031] __alloc_pages+0x33c/0x360
[ 7.151034] ? v9fs_alloc_rdir_buf.isra.0+0x2c/0x40 [9p]
[ 7.151038] __kmalloc_large_node+0x73/0x140
[ 7.151039] __kmalloc+0xd4/0x160
[ 7.151040] v9fs_alloc_rdir_buf.isra.0+0x2c/0x40 [9p]
[ 7.151044] v9fs_dir_readdir_dotl+0x68/0x1d0 [9p]
[ 7.151048] ? selinux_file_permission+0x119/0x160
[ 7.151050] iterate_dir+0xa1/0x190
[ 7.151052] __x64_sys_getdents64+0x88/0x140
[ 7.151053] ? __pfx_filldir64+0x10/0x10
[ 7.151054] do_syscall_64+0x39/0x90
[ 7.151056] entry_SYSCALL_64_after_hwframe+0x78/0xe2
[ 7.151058] RIP: 0033:0x7f898daf21a7
[ 7.151062] Code: 89 e8 5b 5d 31 d2 31 ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 11 31 d2 31 c9 31 f6 31 ff 45 31 db c3 0f 1f
[ 7.151063] RSP: 002b:00007fff48cb4718 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
[ 7.151064] RAX: ffffffffffffffda RBX: 00007f898d9ec010 RCX: 00007f898daf21a7
[ 7.151065] RDX: 0000000000020000 RSI: 00007f898d9ec040 RDI: 0000000000000005
[ 7.151065] RBP: 00007f898d9ec014 R08: 0000000000000000 R09: 0000000000000000
[ 7.151066] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f898d9ec040
[ 7.151066] R13: ffffffffffffff88 R14: 0000000000000002 R15: 000000000165d170
[ 7.151067] </TASK>
[ 7.151068] Mem-Info:
[ 7.151069] active_anon:484 inactive_anon:0 isolated_anon:0
active_file:5270 inactive_file:81554 isolated_file:0
unevictable:0 dirty:4370 writeback:0
slab_reclaimable:9459 slab_unreclaimable:12093
mapped:786 shmem:6 pagetables:22
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:3705 free_pcp:43 free_cma:0
[ 7.151071] Node 0 active_anon:1936kB inactive_anon:0kB active_file:21080kB inactive_file:326092kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3144kB dirty:17480kB writeback:0kB shmem:24kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:3856kB pagetables:88kB sec_pagetables:0kB all_unreclaimable? no
[ 7.151073] Node 0 DMA free:1804kB boost:0kB min:88kB low:108kB high:128kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:13000kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 7.151075] lowmem_reserve[]: 0 430 430 430 430
[ 7.151077] Node 0 DMA32 free:13016kB boost:0kB min:2608kB low:3260kB high:3912kB reserved_highatomic:2048KB active_anon:1908kB inactive_anon:0kB active_file:21180kB inactive_file:313024kB unevictable:0kB writepending:17492kB present:507764kB managed:451696kB mlocked:0kB bounce:0kB free_pcp:188kB local_pcp:0kB free_cma:0kB
[ 7.151079] lowmem_reserve[]: 0 0 0 0 0
[ 7.151081] Node 0 DMA: 1*4kB (U) 7*8kB (UE) 1*16kB (E) 4*32kB (UME) 3*64kB (UME) 1*128kB (M) 1*256kB (U) 0*512kB 1*1024kB (M) 0*2048kB 0*4096kB = 1804kB
[ 7.151086] Node 0 DMA32: 114*4kB (UMEH) 320*8kB (UMEH) 321*16kB (UMEH) 38*32kB (UMEH) 37*64kB (MH) 2*128kB (H) 2*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 13016kB
[ 7.151091] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 7.151092] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 7.151093] 86768 total pagecache pages
[ 7.151093] 0 pages in swap cache
[ 7.151094] Free swap = 0kB
[ 7.151094] Total swap = 0kB
[ 7.151094] 130939 pages RAM
[ 7.151095] 0 pages HighMem/MovableOnly
[ 7.151095] 14175 pages reserved
[ 7.151095] 0 pages cma reserved
[ 7.151095] 0 pages hwpoisoned
One way to completely avoid this problem is to use another implementation of OverlayFS.
I opened https://github.com/NixOS/nixpkgs/pull/329696 which adds a new option, useFUSEOverlayFS
, that switches the implementation of OverlayFS over to the older FUSE version, which does not have this problem.
Alright awesome. I see some workarounds there, I'll see if any of them work and then switch to the new option once this is merged
I see some workarounds there
Note that those workarounds need to be applied to the VM that dockerTools creates and runs, not the host machine running nix-build
.
and then switch to the new option once this is merged
It would help if you could test that the patch fixes the problem you were seeing in the first place, and make a note on the pull request to that effect.
I see some workarounds there
Note that those workarounds need to be applied to the VM that dockerTools creates and runs, not the host machine running
nix-build
.and then switch to the new option once this is merged
It would help if you could test that the patch fixes the problem you were seeing in the first place, and make a note on the pull request to that effect.
Oh yeah no problem. Let me try it after work on both machines.
still failing to build but its getting inside the VM. Does the VM not have internet connectivity?
That said, I'm definitely getting inside the VM so that's awesome
error: builder for '/nix/store/2p7waij0cgdp8s200cnmcdy4mzidk872-docker-layer-nextcloud-custom.drv' failed with exit code 100;
last 25 log lines:
> Ign:3 http://deb.debian.org/debian-security bookworm-security InRelease
> Err:1 http://deb.debian.org/debian bookworm InRelease
> Temporary failure resolving 'deb.debian.org'
> Err:2 http://deb.debian.org/debian bookworm-updates InRelease
> Temporary failure resolving 'deb.debian.org'
> Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
> Temporary failure resolving 'deb.debian.org'
> Reading package lists... Done
> W: Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease Temporary failure resolving 'deb.debian.org'
> W: Failed to fetch http://deb.debian.org/debian/dists/bookworm-updates/InRelease Temporary failure resolving 'deb.debian.org'
> W: Failed to fetch http://deb.debian.org/debian-security/dists/bookworm-security/InRelease Temporary failure resolving 'deb.debian.org'
> W: Some index files failed to download. They have been ignored, or old ones used instead.
> + /usr/bin/apt-get install -y --no-install-recommends ffmpeg ghostscript libmagickcore-6.q16-6-extra procps smbclient supervisor
> Reading package lists... Done
> Building dependency tree... Done
> Reading state information... Done
> Package ghostscript is not available, but is referred to by another package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
>
> E: Unable to locate package ffmpeg
> E: Package 'ghostscript' has no installation candidate
> E: Unable to locate package smbclient
> E: Unable to locate package supervisor
> [ 23.261455] reboot: Power down
For full logs, run 'nix log /nix/store/2p7waij0cgdp8s200cnmcdy4mzidk872-docker-layer-nextcloud-custom.drv'.
Does the VM not have internet connectivity?
No, input-addressed derivations don't have network connectivity, as that would allow them to be impure. Content-addressed derivations can access the Internet, but they are expected to produce the exact same output every time, which might not be feasible in the case of a VM image that installs packages with apt-get
. Anyway, I think that's a bit beyond the scope of the original problem, you may get more/better answers on e.g. https://discourse.nixos.org/.
Yeah that's not a problem. I have the sandbox to work off of and can figure it out from there hopefully. Thank you for all the help
Describe the bug
I'm trying to translate the nextcloud docker apache image link to source to a
dockerTools.buildImage
setup. I'm trying to use the runAsRoot functionality to do some of the apt-get steps and such (as this is an Ubuntu-based image), but I'm not even able to get to the point where the script runs. I believe it's failing at this line of the VM setup script.Steps To Reproduce
Steps to reproduce the behavior:
sudo nix build .#nixosConfigurations.palatine-hill.config.system.build.toplevel --verbose
Expected behavior
sudo nix build .#nixosConfigurations.palatine-hill.config.system.build.toplevel --verbose
Additional context
Link to file creating the image: https://github.com/RAD-Development/nix-dotfiles/blob/feature/docker-palatine-hill-migration/systems/palatine-hill/docker/nextcloud-image/default.nix
Builder Logs:
Notify maintainers
Metadata
Please run
nix-shell -p nix-info --run "nix-info -m"
and paste the result.Note on the nix version, I've tried this on 2.23 and 2.18 and got the same result.
Add a :+1: reaction to issues you find important.