Data creep

[barnabasbusa]

What's your CLI version?

0.83.16

Description & steps to reproduce

Each run increases the Docker.raw file size, which will lead to the virtual disk exhaustion eventually.

Use the following run example:

{
  "participants": [
    {
      "el_client_type": "geth",
      "cl_client_type": "teku",
      "count": 3
    },
    {
      "el_client_type": "nethermind",
      "cl_client_type": "teku",
      "el_client_image": "nethermindeth/nethermind:cancun"
    },
    {
      "el_client_type": "besu",
      "cl_client_type": "teku"
    }
  ],
  "network_params": {
    "deneb_fork_epoch": 2,
    "seconds_per_slot": 3
  },
  "global_client_log_level": "info",
  "launch_additional_services": true,
  "mev_type": "full",
  "mev_params": {
    "mev_relay_image": "flashbots/mev-boost-relay:0.28.0a1",
    "mev_builder_image": "flashbots/builder:1.13.2.dev1"
  }
}

Steps to reproduce on MacOS:

• Install docker desktop
• cd ~/Library/Containers/com.docker.docker/Data/vms/0/data
• Delete Docker.raw (if you have one already), or mv to backup.
• Start docker desktop (it will initiate a new Docker.raw file)
• du -sh * will now display that Docker.raw file is approx 1.6G
• kurtosis engine start
• du -sh * will now display that Docker.raw file is approx 1.9G
• kurtosis run github.com/kurtosis-tech/ethereum-package --args-file ~/example.json
• du -sh * will now display that Docker.raw file is approx 9.4GB.
• kurtosis clean -a
• du -sh * will now display that Docker.raw file is approx 9.3GB.
• kurtosis run github.com/kurtosis-tech/ethereum-package --args-file ~/example.json
• du -sh * will now display that Docker.raw file is approx 10GB.
• kurtosis clean -a
• du -sh * will now display that Docker.raw file is approx 9.7GB.
• This leads to a creep of about 200-500MB each time you run a big test like this. It seems like kurtosis clean -a doesn't clean up all datadir and leaves some stuff just hanging around (not talking about docker images).

This leads to an issue like https://github.com/kurtosis-tech/kurtosis/issues/1457#issuecomment-1744696026 where if your virtual disk size is exhausted, docker images will be evicted, and images will have to be pulled every time you run kurtosis.

I have included a example configuration file, as it ensures that you will pull approx 25 images which is a significant amount of docker images.

Desired behavior

Make sure that kurtosis clean -a will clean everything up, and does not lead to any data creep. Between runs check the size of Docker.raw and before and after run the Docker.raw file should be approx to the same size without any additional data.

What is the severity of this bug?

Painful; this is causing significant friction in my workflow.

What area of the product does this pertain to?

CLI: the Command Line Interface

[parithosh]

fyi my storage limit was higher and my docker.raw was 200G -rw-r--r-- 1 parithosh staff 200G Oct 4 09:17 Docker.raw

[h4ck3rk3y]

Related sub ticket - https://github.com/kurtosis-tech/kurtosis/issues/1394

[mieubrisse]

@tedim52 pretty sure that this gets fixed with symlink stuff + enclave rm and clean to clean up dead logs

[tedim52]

Hey @barnabasbusa ! I merged some fixes that should prevent the data creep - as of v0.84.3, logs are removed when you clean -a and enclave rm. Closing this issue for now but if you continue experiencing the data creep, please ping me in Discord and I'll reopen this.

[barnabasbusa]

Thanks @tedim52 looks a lot better now!