search

nix-community / nixos-generators

Collection of image builders [maintainer=@Lassulus]
MIT License
1.85k stars 146 forks source link

System resources "starvation" during building image #389

Closed GrumpyMeow closed 1 month ago

GrumpyMeow commented 1 month ago

Hi! When i generate an image the machine becomes very unresponsive and other processes on the machine starve. Eventually when the image is generated, the machine becomes responsive again and the processes do start to work again. I report this as an issue as it's not common nowadays to be able to have a single process to starve the other running processes.

The diskio indicates that the machine is busy doing a diskread at it's maximum speed steadily (1.45G per second) for a long time.

image

Is it maybe possible to implement some throttling or set the process more "nice"? Having a more responsive machine would also allow me to check what it's currently doing, as the generate i'm currently running is going on for a very long time at full-speed diskread for halve-hour. It's not that big of a drive.

top - 12:39:11 up 20 days, 23:24,  5 users,  load average: 109.73, 112.61, 106.81
Tasks: 858 total,   9 running, 848 sleeping,   0 stopped,   1 zombie
%Cpu(s):  3.3 us, 66.4 sy,  0.0 ni,  5.9 id, 23.2 wa,  0.0 hi,  0.4 si,  0.0 st 
MiB Mem :  44082.0 total,   1195.3 free,  39945.7 used,   9058.8 buff/cache     
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   4136.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                         
2190501 root      20   0       0      0      0 R  82.4   0.0   9:49.03 kworker/u33:26+loop3                                                                                            
2160751 root      20   0       0      0      0 R  81.4   0.0   8:30.61 kworker/u33:6+loop3                                                                                             
2201840 root      20   0       0      0      0 R  81.4   0.0  17:27.90 kworker/u34:30+loop3                                                                                            
2191306 root      20   0       0      0      0 D  81.0   0.0   9:16.70 kworker/u33:29+loop3                                                                                            
2200782 root      20   0       0      0      0 R  80.1   0.0   7:51.88 kworker/u33:14+loop3                                                                                            
2165426 root      20   0       0      0      0 R  77.8   0.0  33:59.50 kworker/u34:25+loop3                                                                                            
2219745 root      20   0       0      0      0 I  42.2   0.0   2:26.40 kworker/u33:12-btrfs-endio                                                                                      
2205358 root      20   0       0      0      0 I  31.0   0.0   6:30.42 kworker/u34:3-btrfs-endio                                                                                       
2201837 root      20   0       0      0      0 I  24.5   0.0   5:39.22 kworker/u34:23-btrfs-endio                                                                                      
1638695 root      20   0   10.9g   3.3g   4864 S  24.2   7.6 167:00.80 kvm                                                                                                             
2219748 root      20   0       0      0      0 I  23.2   0.0   1:25.15 kworker/u33:23-btrfs-endio                                                                                      
2219746 root      20   0       0      0      0 I  22.2   0.0   1:55.86 kworker/u33:18-btrfs-endio                                                                                      
2211595 root      20   0       0      0      0 I  21.2   0.0   3:15.13 kworker/u34:8-btrfs-endio                                                                                       
2201838 root      20   0       0      0      0 I  19.9   0.0   5:34.81 kworker/u34:26-btrfs-endio                                                                                      
2219757 root      20   0       0      0      0 I  17.6   0.0   0:40.87 kworker/u34:17-btrfs-endio                                                                                      
1688986 root      20   0 1253172  15704    256 D  16.7   0.0  42:04.26 .tailscaled-wra                                                                                                 
2190079 root      20   0       0      0      0 R  16.3   0.0   8:50.44 kworker/u33:4+loop3                                                                                             
2219741 root      20   0       0      0      0 R  16.0   0.0   0:30.38 kworker/u33:1+loop3                                                                                             
2190499 root      20   0       0      0      0 I  15.7   0.0  10:14.68 kworker/u33:19-btrfs-endio                                                                                      
1688901 991       20   0 2581324   7372    256 S  15.0   0.0  12:14.22 zabbix_agentd                                                                                                   
2219742 root      20   0       0      0      0 I  14.4   0.0   2:04.04 kworker/u33:2-btrfs-endio                                                                                       
2213465 root      20   0       0      0      0 I  13.4   0.0   2:18.72 kworker/u34:12-btrfs-endio                                                                                      
2185061 30001     20   0 2260760  22636   1024 D  12.7   0.1   7:32.81 nix-env                                                                                                         
1689177 990       20   0 8278520 314288    812 S  12.4   0.7  14:56.92 python3.11                                                                                                      
1689178 980       20   0  262.6g 211724  73472 S  12.4   0.5   6:05.09 jellyfin                                                                                                        
2190498 root      20   0       0      0      0 I  11.8   0.0   7:14.86 kworker/u33:17-btrfs-endio

Details... The machine i'm running NixOS on is a ProxmoxLXC container. The container has been configured to use 12 of the 16 cores. 8GB of memory. (no swap) Command i was running: nixos-generate -c mynewvm.nix -f proxmox

GrumpyMeow commented 1 month ago

When trying to get more detail information for this issue i noticed that atop gave a error message about the memory usage. As it's a container, i was able to give it more memory. I never suspected this to be an issue as Proxmox didn't indicate that memory was an issue. It's a known issue that Proxmox sometimes is unsable to give correct stats. The Proxmox screenshot indicated it never used more that 4gb. But in actuallity it was using 100% of memory.

image

Found: The generated VM image is temporarily written to the /tmp folder in my NixOS LXC. To reduce disk-wear i had this configured with boot.tmp.useTmpfs = true;. So the VM image was being written to memory.