Deployment fails with large numbers of mirrors.

[nuvious]

When doing multiple architectures and/or repositories such as arm vs x86 or when mirroring alternative repositories in addition to the main repository such as docker, etc, the cronjob can run out of resources easily. Here's an example:

Warning Evicted 30h (x3 over 30h) kubelet Pod ephemeral local storage usage exceeds the total limit of containers 8Gi.

The above was with modifications to the resources to allow 4-8 GB of ephemeral storage. Inspecting the job container while it was live confirmed that the cephfs (used in my example) was mounted properly.

nuvious@rpiterm:~/CKAD$ kubectl exec pod/debmirror-manual-job-72wxk --  /bin/bash -c "df -h"
Filesystem                                                                   Size  Used Avail Use% Mounted on
overlay                                                                       31G   11G   19G  37% /
tmpfs                                                                         64M     0   64M   0% /dev
/dev/sda1                                                                     31G   11G   19G  37% /etc/hosts
shm                                                                           64M     0   64M   0% /dev/shm
192.168.1.70:6789,192.168.1.71:6789,192.168.1.72:6789:/kube/debmirror  2.9T  935G  1.9T  33% /var/spool/apt-mirror
tmpfs                                                                         14G   12K   14G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                        7.9G     0  7.9G   0% /proc/acpi
tmpfs                                                                        7.9G     0  7.9G   0% /sys/firmware
nuvious@rpiterm:~/CKAD$

The base path was also confirmed:

nuvious@rpiterm:~/CKAD$ kubectl exec pod/debmirror-manual-job-72wxk -- /bin/sh -c "cat /etc/apt/mirror.list"
set base_path         /var/spool/apt-mirror
set mirror_path       /data
set skel_path         /skel
set var_path          /var
set postmirror_script /postmirror.sh
set defaultarch       amd64

Current hypothesis is that debmirror is writing somewhere else on the filesystem for temporary storage, though I haven't seen anything about that in the documentation. Exploration/resolution steps blow.

Resolution

1. Test to see if splitting up mirror.list into separate sync jobs based on repository, arch, etc can reduce the resource needs sufficiently to allow the jobs to complete a. If so, document how to do this in the documentation
2. Analyze fs events to see where debmirror is writing outside of /var/spool/apt-mirror and allocate a storage volume for that path.

[nuvious]

Closing due to it just being a simple configuraiton issue. The $base_path variable was omitted in the mirror/skel/var paths.

This:

set base_path         /var/spool/apt-mirror
set mirror_path       /data
set skel_path         /skel
set var_path          /var

Should have been this:

set base_path         /var/spool/apt-mirror
set mirror_path       $base_path/data
set skel_path         $base_path/skel
set var_path          $base_path/var