full /mnt/stateful_partition for Docker container system files

[vochicong]

In my Datalab docker container, after installation of some apt, conda and pip packages, the root disk / gets full, then Datalab freezes.

Steps to reproduce

1. On my local Ubuntu

   datalab beta create-gpu --machine-type n1-standard-4 --zone asia-east1-a mylab 

Use mylab in the browser, it works as expected

2. From a different terminal on my local Ubuntu

gcloud compute ssh mylab

3. Inside the VM mylab

Get into docker container mylab, install somethings like

apt-get update -y && apt-get install -y p7zip-full --allow-unauthenticated
conda activate py3env
conda install -c anaconda chainer cupy
pip install python-shogi statistics tqdm
# ................

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.2G  577M  644M  48% /
devtmpfs        7.4G     0  7.4G   0% /dev
tmpfs           7.4G     0  7.4G   0% /dev/shm
tmpfs           7.4G  432K  7.4G   1% /run
tmpfs           7.4G     0  7.4G   0% /sys/fs/cgroup
tmpfs           256K     0  256K   0% /mnt/disks
tmpfs           7.4G     0  7.4G   0% /tmp
/dev/sda8        12M   28K   12M   1% /usr/share/oem
/dev/sda1        16G   12G  3.8G  77% /mnt/stateful_partition
tmpfs           1.0M  136K  888K  14% /var/lib/cloud
overlayfs       1.0M  120K  904K  12% /etc
/dev/sdb        196G   17G  170G   9% /mnt/disks/datalab-pd

Note that /mnt/stateful_partition is getting full. If I install more packages, it will eventually get full and the Datalab container freezes. The Docker service will stop doesn't start even after rebooting the VM mylab.

Temporary solution

1. In VM mylab, remove the largest folder under /mnt/stateful_partition/var/lib/docker/overlay2/.
2. Turn off VM mylab
3. From my Ubuntu, run datalab connect mylab, wait until datalab is available (the container has started)
4. ssh into VM mylab, run docker system prune .

[dcompgriff]

I work with datalab a ton, and this is kind of a gripe for me too. I typically save my data files, models, and other artifacts in my /tmp folder from within the datalab instance. However, sometimes I need to install system packages or extra python packages and notice I routinely run up against the /mnt/stateful_partition (Which I believe is the "/" directory in datalab). Is there any easy way to set the option for how large the /mnt/stateful_partition should be in the datalab instance? I can change the disk size of the datalab instance in the create command, but this does nothing for the /mnt/stateful_partition folder size.