Extending disk space for docker

[Jean-Baptiste-Lasselle]

This taks is about :

• [ ] fully docuement how to manually add a new disk for docker, on a basic debian VirtualBox VM
• [ ] fully automate provisioning the same VirtualBox VM using Hashicorp Packer
• [ ] fully automate provisioning the same VM, using Hashicorp Packer, but for at least one cloud provider : the one cloud provider that is going to be the cheapest one for a first Pesto public service

Basically:

• I provision a VM with debian (i already used bullseye, there should be no problem for bookworm) : with only one disk one partition for all
• Then I add an additional disk, that I format it into a single partition (which will be named for example /dev/sdb1, if my disk device is named /dev/sdb), and mount on /var/lib/pokus/docker (I before, of course mkdir -p /var/lib/pokus/docker)
• And I add the following docker daemon config (stop daemon and disable the service first, after config I restart and re-enable the docker service) :

jbl@pokus2:~/pesto_api_mongo$ cat /etc/docker/daemon.json
{
  "data-root": "/var/lib/pokus/docker"
}

jbl@pokus2:~/pesto_api_mongo$ sudo ls -alh /var/lib/pokus/docker
total 84K
drwx--x---  13 root root 4.0K Jan 13 10:22 .
drwxr-xr-x   3 root root 4.0K Jul 14  2023 ..
drwx--x--x   4 root root 4.0K Dec  8 23:45 buildkit
drwx--x---  13 root root 4.0K Dec 12 18:31 containers
-rw-------   1 root root   36 Dec  8 23:45 engine-id
drwx------   3 root root 4.0K Dec  8 23:45 image
drwx------   2 root root  16K Dec  8 23:00 lost+found
drwxr-x---   3 root root 4.0K Dec  8 23:45 network
drwx--x--- 146 root root  20K Jan 13 10:22 overlay2
drwx------   4 root root 4.0K Dec  8 23:45 plugins
drwx------   2 root root 4.0K Jan 13 10:22 runtimes
drwx------   2 root root 4.0K Dec  8 23:45 swarm
drwx------   2 root root 4.0K Jan 13 10:22 tmp
drwx-----x  24 root root 4.0K Jan 13 10:22 volumes
jbl@pokus2:~/pesto_api_mongo$ sudo fdisk -l

Disk /dev/sdb: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xbfcbbe45

Device     Boot Start      End  Sectors Size Id Type
/dev/sdb1        2048 41943039 41940992  20G 83 Linux

Disk /dev/sda: 17.45 GiB, 18741657600 bytes, 36604800 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa86200f7

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1  *        2048 34603007 34600960 16.5G 83 Linux
/dev/sda2       34605054 36603903  1998850  976M  5 Extended
/dev/sda5       34605056 36603903  1998848  976M 82 Linux swap / Solaris
jbl@pokus2:~/pesto_api_mongo$ sudo df -h /var/lib/pokus/docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        20G   13G  6.2G  67% /var/lib/pokus/docker
jbl@pokus2:~/pesto_api_mongo$ sudo df -h /var/lib/pokus
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        17G  2.7G   13G  18% /

[Jean-Baptiste-Lasselle]

Switch to new disk recipe

Plan:

• we unmount the /dev/sdb1 disk partition mounted on /var/lib/pokus/docker

• Shell script (TOCOMPLETE):

# --- # --- # --- # --- # --- 
# --- # --- # --- 
# First we stop the docker daemon

sudo systemctl stop docker
sudo systemctl stop docker.socket
sudo systemctl disable docker
sudo systemctl daemon-reload
docker ps -a || true

# --- # --- # --- # --- # --- 
# --- # --- # ---  now shutdown VM, add the new virtual disk, and restart the VM
# --- # --- # --- # --- # --- 

# --- # --- # --- # --- # --- 
# --- # --- # ---  now let's start the disk operations

# unmount the /dev/sdb1 disk partition mounted on  /var/lib/pokus/docker

# sudo umount /var/lib/pokus/docker
sudo umount /dev/sdb1

sudo mkdir -p /var/lib/pokus/docker_backup

sudo mount /dev/sdb1 /var/lib/pokus/docker_backup

sudo df -h /var/lib/pokus/docker

sudo df -h /var/lib/pokus/docker_backup

sudo fdisk -l

# ---
# partition the new disk into ext4: with fdisk its not so easy to run the command silently
# sudo fdisk /dev/sdc1
# sudo mkfs -t ext4 /dev/sdc1

export NEW_DISK_PARTITION_UUID=69a924f3-af50-4ddb-a65d-67454bbc1b79
cat <<EOF >./.etc.fstab.addon
UUID=${NEW_DISK_PARTITION_UUID}   /var/lib/pokus/docker   ext4    defaults        0       0
EOF

# ---
# And I will use the previous disk partition as a data disk space on /opt/engineering/data
sudo mkdir -p /opt/engineering/data

export TWENTY_GB_DISK_PARTITION_UUID=69a924f3-af50-4ddb-a65d-67454bbc1b79
cat <<EOF >./.etc.fstab.addon
UUID=${NEW_DISK_PARTITION_UUID}   /opt/engineering/data   ext4    defaults        0       0
EOF

# --- # --- # --- 
# then you copy all docker data from the old partition, to the new partition

sudo cp -fR /opt/engineering/data/* /var/lib/pokus/docker

# --- # --- # --- 
# then to clean up space 
sudo rm -fr /opt/engineering/data/overlay

# --- # --- # --- # --- # --- 
# --- # --- # --- 
# Finally we re-enable the docker daemon
sudo systemctl enable docker
sudo systemctl sart docker.socket
sudo systemctl start docker
sudo systemctl daemon-reload

Hm, attends, la partition de 20GB je pourrais la monter sur un sous répertoire celui qui s'appelle containers là ci-dessous, ça pourrait peut-être aider :

mouais pas vraiment, c'est réellement le /var/lib/pokus/docker/overlay2 qui utilise 18GB, alors que /var/lib/pokus/docker utilise quasiment le même espace, les autres répertoire tous cumulés utilisent mois d'un GB:

[Jean-Baptiste-Lasselle]

sudo fdisk -l

# ---
# interactive, you type first "n" for new, then 2 or3 times just 
# hit enter to let the default values, which will create one 
# single partition on the whole /dev/sdc disk
# - 
# 

sudo fdisk /dev/sdc

sudo mkfs -t ext4 /dev/sdc1 | tee ./dev.sdc1.ext4.format.logs

export NEW_DISK_PARTITION_UUID=$(cat ./dev.sdc1.ext4.format.logs | grep 'Filesystem UUID:' | awk '{ print $NF }')

sudo mkdir -p /var/lib/pokus2/docker

export NEW_DISK_PARTITION_UUID=200e736d-7065-4b05-9495-6cc3d3daa92b
cat <<EOF >./.etc.fstab.addon
UUID=${NEW_DISK_PARTITION_UUID}   /var/lib/pokus2/docker   ext4    defaults        0       0
EOF

cat ./.etc.fstab.addon | sudo tee -a /etc/fstab

# --- 
# then reboot to automatically mount the new disk on [/var/lib/pokus2/docker]
# - 
# 

# ---
# Now copy all files from the currently 
# docker daemon [data-root] directory, to 
# the new /var/lib/pokus2/docker
# - 
# 

sudo cp -rT /var/lib/pokus/docker /var/lib/pokus2/docker

cat <<EOF >./etc.docker.daemon.json
{
  "data-root": "/var/lib/pokus2/docker"
}
EOF

sudo cp /etc/docker/daemon.json /etc/docker/previous.daemon.json

cat ./etc.docker.daemon.json | sudo tee /etc/docker/daemon.json

# --- # --- # --- # --- # --- 
# --- # --- # --- 
# We re-enable the docker daemon
sudo systemctl enable docker
sudo systemctl start docker.socket
sudo systemctl start docker
sudo systemctl daemon-reload
# --- # --- # --- 
# Finally we reboot the VM