Error (criu/protobuf.c:72): Unexpected EOF on (empty-image), unable to restore the container

[Rohanp07]

Description I'm trying to restore a Docker container using CRIU but encountering an error. Although I'm aware that Docker checkpoint could be used for this purpose, I'm interested in implementing iterative migration which needs pre-dump, which Docker does not support yet.

Steps to reproduce the issue:

1. Create a checkpoint of the source container using CRIU dump command : sudo criu dump -t $CONTAINER_ID --images-dir "/home/cp1" --shell-job --leave-running --external 'mnt[]'

2. Create a destination container with the same image: docker create --name "destination" --security-opt seccomp:unconfined --rm busybox sh -c 'i=0; while true; do echo $i; i=$((i+1)); sleep 1; done'

3. Copy the checkpoint to the destination container directory : sudo cp -r "/home/cp1" /var/lib/docker/containers/$(docker ps -aq --no-trunc --filter name=destination)/checkpoints/

4. Try to restore it with CRIU command: sudo criu restore --images-dir "/var/lib/docker/containers/$(docker ps -aq --no-trunc --filter name=destination)/checkpoints/"

Describe the results you received: Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally): I have used Docker restore command after copying the checkpoint to the destination container directory but when I run the container it freezes.

CRIU logs and information:

CRIU full dump/restore logs: Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)

``` (paste your output here) ```

Output of `criu --version`:

``` Version: 3.19 ```

Output of `criu check --all`:

``` sudo: mon_handle_sigchld: waitpid: No child processes Warn (criu/cr-check.c:1346): Nftables based locking requires libnftables and set concatenations support Looks good but some kernel features are missing which, depending on your process tree, may cause dump or restore failure. ```

Additional environment details: Distributor ID: Ubuntu Description: Ubuntu 22.04.4 LTS Release: 22.04 Codename: jammy

Docker version 27.1.1, build 6312585

[adrianreber]

Have you tried Podman? Podman supports pre-dumping.

You should run CRIU with -v4 and share the complete log. It is difficult to understand what is going on without the complete log. I guess the error happens during the last step, criu restore, right?

[Rohanp07]

(00.000000) CRIU run id = 0xeffffffc0001194f (00.000070) Version: 3.19 (gitid 0) (00.000077) Running on rohan-G5-MF Linux 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_D> (00.000091) Loaded kdat cache from /run/criu.kdat (00.000111) Hugetlb size 2 Mb is supported but cannot get dev's number (00.000124) Hugetlb size 1024 Mb is supported but cannot get dev's number (00.000356) rlimit: RLIMIT_NOFILE unlimited for self (00.000387) No inventory.img image (00.000389) Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)

[adrianreber]

If even inventory.img is missing then you copied the files to the wrong location. But as you mentioned doing this outside of the container engine does not really make sense. Try it with Podman, it gives you pre-dump out of the box.

[Rohanp07]

Okay, I'll try that! Thank you for the quick reply ;-)

[adrianreber]

Please close the issue if this is solved for you.

[Rohanp07]

I tried using podman pre-checkpoint feature. These are the commands that I used:

sudo podman container checkpoint -R -e="/home/checkpoints/cp1.tar.gz" source sleep 1 sudo podman container checkpoint -R --pre-checkpoint -e="/home/checkpoints/predump1.tar.gz" source sleep 1 sudo podman container checkpoint -R -P -e="/home/checkpoints/predump2.tar.gz" source sleep 1 sudo podman container checkpoint -R -P -e="/home/checkpoints/predump3.tar.gz" source sleep 1 sudo podman container checkpoint --with-previous -R -e "/home/checkpoints/wpev3.tar.gz" source sleep 1

However, all the checkpoints are the same size (73.9 MB). I need the new checkpoints to capture only the dirty memory or changes since the last checkpoint or predump. What am I doing wrong?

[rst0git]

What am I doing wrong?

Nothing. It is just that Podman currently does not support iterative checkpointing when --export is used. In particular, it does not provide a mechanism for keeping track of previous checkpoints from an archive (see --prev-images-dir in man 8 criu).

For example, this feature can be used as follows:

podman run -d --name looper busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

podman container checkpoint -l --pre-checkpoint
ls /var/lib/containers/storage/overlay-containers/*/userdata/pre-checkpoint/

podman container checkpoint -l --with-previous
ls -al /var/lib/containers/storage/overlay-containers/*/userdata/checkpoint/