Dumping failed when dump a program.

[lijunqiang123]

Here are parts of error infomation: (00.012677) Dumping mm (pid: 37668) (00.012678) ---------------------------------------- (00.012680) 0x400000-0x421000 (132K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0 reg fp shmid: 0x1 (00.012683) 0x620000-0x621000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x20000 reg fp shmid: 0x1 (00.012684) 0x621000-0x622000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x21000 reg fp shmid: 0x1 (00.012686) 0x622000-0x632000 (64K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap shmid: 0 (00.012687) 0x7f1bfeec3000-0x7f1bff083000 (1792K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0 reg fp shmid: 0x2 (00.012689) 0x7f1bff083000-0x7f1bff283000 (2048K) prot 0 flags 0x2 fdflags 0 st 0x41 off 0x1c0000 reg fp shmid: 0x2 (00.012690) 0x7f1bff283000-0x7f1bff287000 (16K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x1c0000 reg fp shmid: 0x2 (00.012691) 0x7f1bff287000-0x7f1bff289000 (8K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x1c4000 reg fp shmid: 0x2 (00.012693) 0x7f1bff289000-0x7f1bff28d000 (16K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap shmid: 0 (00.012694) 0x7f1bff28d000-0x7f1bff2b3000 (152K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0 reg fp shmid: 0x3 (00.012695) 0x7f1bff497000-0x7f1bff49a000 (12K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap shmid: 0 (00.012697) 0x7f1bff4a2000-0x7f1bff4b2000 (64K) prot 0x3 flags 0x21 fdflags 02 st 0x501 off 0 reg as sysv shmid: 0x83c8012 (00.012698) Error (criu/cr-dump.c:431): Task 37668 with SysVIPC shmem map @7f1bff4a2000 doesn't live in IPC ns (00.012700) Error (criu/cr-dump.c:1411): Dump mappings (pid: 37668) failed with -1 (00.012732) Unlock network (00.012740) Unfreezing tasks into 1 (00.012742) Unseizing 37668 into 1 (00.012754) Unseizing 37669 into 1 (00.012761) Error (criu/cr-dump.c:1753): Dumping FAILED.

From the website: https://criu.org/What_cannot_be_checkpointed#Cork-ed_UDP_sockets CRIU can't dump program which has System VIPC. If I need dump the program, what should I do to make CRIU can do this. Looking forward your reply! Thank you.

[adrianreber]

If you want to checkpoint a process that uses System V IPC all involved processes have to be in the same IPC namespace.

You could start the processes you want to dump using unshare -i. Or a container.

[avagin]

CRIU can't dump program which has System VIPC.

This isn't right. CRIU can dump processes which have SysVIPC mappings, but it requires that they run in a separate IPC namespaces.

* SysVIPC memory segment w/o IPC namespace[edit]

IPC objects are not tied to any tasks. Thus once CRIU meets an IPC memory attached to a task, it requires the whole IPC namespace to be dumped as well.

If I need dump the program, what should I do to make CRIU can do this.

You can run your process in a new IPC namespace or you can run it in a docker containers.

[lijunqiang123]

Thank you for your reply. I will try to test it later.

[lijunqiang123]

I successfully dump the program with System V IPC. I don't know if shared memory IPC can be restored when I use criu restore?

[adrianreber]

Just try it and let us know if it works.

[lijunqiang123]

I try to test the program with System V IPC. I start the program using unshare -i. And I can successfully dump the program with System V IPC. But it didn't work when I used CRIU RESTORE. Here are parts of error infomation: (00.000283) kernel pid_max=131072 (00.000286) Reading image tree (00.000310) Add mnt ns 5 pid 44000 (00.000315) Add net ns 2 pid 44000 (00.000321) pstree pid_max=44000 (00.000327) Migrating process tree (SID 43472->43685) (00.000330) Migrating process tree (GID 43998->44030) (00.000333) Will restore in 8000000 namespaces (00.000336) NS mask to use 8000000 (00.000360) Collecting 37/54 (flags 2) (00.000378) Collected [home/ljq/Desktop/Ctest/shm_example/shmwrite] ID 0x1 (00.000385) Collected [lib/x86_64-linux-gnu/libc-2.23.so] ID 0x2 (00.000389) Collected [lib/x86_64-linux-gnu/ld-2.23.so] ID 0x3 (00.000393) Collected [dev/pts/23] ID 0x5 (00.000422) Collected [home/ljq/Desktop/Ctest/shm_example] ID 0x6 (00.000427) Collected [.] ID 0x7 (00.000432) - ... done (00.000435) Collecting 43/63 (flags 0) (00.000440) No remap-fpath.img image (00.000444)- ... done (00.000462) No cgroup.img image (00.000471) Running pre-restore scripts (00.000502) No mountpoints-5.img image (00.000506) mnt: Reading mountpoint images (id 5 pid 44000) (00.000513) No netns-2.img image (00.000589) Forking task with 44000 pid (flags 0x8000000) (00.000727) PID: real 44001 virt 44000 (00.000795) Wait until namespaces are created (00.000959) Error (criu/cr-restore.c:1650): Pid 44001 do not match expected 44000 (00.001222) Error (criu/cr-restore.c:2294): Restoring FAILED.

[lijunqiang123]

I try to test the program with System V IPC. I start the program using unshare -i. And I can successfully dump the program with System V IPC. But it didn't work when I used CRIU RESTORE. Here are parts of error infomation: (00.000283) kernel pid_max=131072 (00.000286) Reading image tree (00.000310) Add mnt ns 5 pid 44000 (00.000315) Add net ns 2 pid 44000 (00.000321) pstree pid_max=44000 (00.000327) Migrating process tree (SID 43472->43685) (00.000330) Migrating process tree (GID 43998->44030) (00.000333) Will restore in 8000000 namespaces (00.000336) NS mask to use 8000000 (00.000360) Collecting 37/54 (flags 2) (00.000378) Collected [home/ljq/Desktop/Ctest/shm_example/shmwrite] ID 0x1 (00.000385) Collected [lib/x86_64-linux-gnu/libc-2.23.so] ID 0x2 (00.000389) Collected [lib/x86_64-linux-gnu/ld-2.23.so] ID 0x3 (00.000393) Collected [dev/pts/23] ID 0x5 (00.000422) Collected [home/ljq/Desktop/Ctest/shm_example] ID 0x6 (00.000427) Collected [.] ID 0x7 (00.000432) - ... done (00.000435) Collecting 43/63 (flags 0) (00.000440) No remap-fpath.img image (00.000444)- ... done (00.000462) No cgroup.img image (00.000471) Running pre-restore scripts (00.000502) No mountpoints-5.img image (00.000506) mnt: Reading mountpoint images (id 5 pid 44000) (00.000513) No netns-2.img image (00.000589) Forking task with 44000 pid (flags 0x8000000) (00.000727) PID: real 44001 virt 44000 (00.000795) Wait until namespaces are created (00.000959) Error (criu/cr-restore.c:1650): Pid 44001 do not match expected 44000 (00.001222) Error (criu/cr-restore.c:2294): Restoring FAILED.

[adrianreber]

It seems you now have a PID collision. Do you already have a process running with PID 44000 on your system?

[lijunqiang123]

There is a process whose PID is 44000. But when I use criu dump, the log file shows that dumping finished successfully. I checked the process again after dump, and the pid 44000 still existed. And I can't kill this process using kill -9 44000 command.

[adrianreber]

As long as the PID 44000 is in use CRIU cannot restore the process. Is PID 44000 the process you checkpointed with CRIU or another process? What message do you get when you try to kill the process? Which process is it?

[lijunqiang123]

I tried again. There are two programs, one is to read message from shared memory (shmread.c) , the other is to write message to the same shared memory(shmwrite.c). I started the shmread program using unshare -i ./shmread 、dumped the program usingcriu dump --tree `pidof shmwrite` --images-dir img-server/ -v4 -o dump.log --shell-job and restore the program using criu restore --images-dir img-server/ -v4 -o rst.log --shell-job. The final results show that criu dump and restore are successful. But when I input a string from keyboard and writed it the shared memory, it didn't work. The restored program that I used criu restore can no longer send data to shmread programs. Here are the results:

shmread.c:

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/shm.h>
#include "shmdata.h"

int main()
{
    int running = 1;
    void *shm = NULL;
    struct shared_use_st *shared;
    int shmid;

    shmid = shmget((key_t)1234, sizeof(struct shared_use_st), 0666|IPC_CREAT);
    if(shmid == -1)
    {
        fprintf(stderr, "shmget failed\n");
        exit(EXIT_FAILURE);
    }

    shm = shmat(shmid, 0, 0);
    if(shm == (void*)-1)
    {
        fprintf(stderr, "shmat failed\n");
        exit(EXIT_FAILURE);
    }
    printf("\nMemory attached at %X\n", (int)shm);

    shared = (struct shared_use_st*)shm;
    shared->written = 0;

    int fx = fork();
    if(fx == -1) {
        fprintf(stderr, "fork error\n");
        exit(EXIT_FAILURE);
    }
    if (fx == 0) {
        execv("./shmwrite", (char *)NULL);
        exit(1);
    }
    else {}

    int x = 0;
    while(running)
    {
        if(shared->written != 0)
        {
            printf("You wrote: %s\n", shared->text);
            sleep(rand() % 3);
            //set shared memory can be written
            shared->written = 0;
            int status;
                        // Before achieved the waitpid line, using the criu dump command in other terminal
                        // waitpid can release zombie process. 
            if (waitpid(fx, &status, 0) <= 0)  fprintf(stderr, "%s\n", "waitpid() failed");
            if(strncmp(shared->text, "end", 3) == 0)
                running = 0;
        }
        else
            sleep(1);
    }

    if(shmdt(shm) == -1)
    {
        fprintf(stderr, "shmdt failed\n");
        exit(EXIT_FAILURE);
    }

    if(shmctl(shmid, IPC_RMID, 0) == -1)
    {
        fprintf(stderr, "shmctl(IPC_RMID) failed\n");
        exit(EXIT_FAILURE);
    }
    exit(EXIT_SUCCESS);
}

shmwrite.c:

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/shm.h>
#include "shmdata.h"

int main()
{
    int running = 1;
    void *shm = NULL;
    struct shared_use_st *shared = NULL;
    char buffer[BUFSIZ + 1];
    int shmid;

    shmid = shmget((key_t)1234, sizeof(struct shared_use_st), 0666|IPC_CREAT);
    if(shmid == -1)
    {
        fprintf(stderr, "shmget failed\n");
        exit(EXIT_FAILURE);
    }

    shm = shmat(shmid, (void*)0, 0);
    if(shm == (void*)-1)
    {
        fprintf(stderr, "shmat failed\n");
        exit(EXIT_FAILURE);
    }
    printf("Memory attached at %X\n", (int)shm);

    shared = (struct shared_use_st*)shm;
    while(running)
    {

        while(shared->written == 1)
        {
            sleep(1);
            printf("Waiting...\n");
        }

        printf("Enter some text: ");
        fgets(buffer, BUFSIZ, stdin);
        strncpy(shared->text, buffer, TEXT_SZ);

        shared->written = 1;

        if(strncmp(buffer, "end", 3) == 0)
            running = 0;
    }

    if(shmdt(shm) == -1)
    {
        fprintf(stderr, "shmdt failed\n");
        exit(EXIT_FAILURE);
    }
    sleep(2);
    exit(EXIT_SUCCESS);
}

shmdata.h:

#ifndef _SHMDATA_H_HEADER
#define _SHMDATA_H_HEADER

#define TEXT_SZ 2048

struct shared_use_st
{
    int written; //flag:    0:Writable   Not 0: Readable
    char text[TEXT_SZ];   //text
};

#endif

[adrianreber]

Are both processes running in the same IPC namespace. That is necessary. You should start processes in the same IPC namespace and checkpoint them with one dump command. Make one process the child process of the other or something like this.

[lijunqiang123]

Yes, I used fork() to execute shmwrite. That is, the shmwrite program is a child process of the shmread program.

[adrianreber]

Not sure what you are trying to achieve, but if you checkpoint the parent process (shmread) it should work.

If you are checkpointing the child process shmwrite and restoring it, it will run in another IPC namespace. If you want shmwrite to be restored in the original IPC namespace of shmread you have to use the --join-ns option of CRIU.

[lijunqiang123]

Do I need to run criu dump and criu restore commands in the same namespace as shmread program when I use them?

[adrianreber]

Do I need to run criu dump and criu restore commands in the same namespace as shmread program when I use them?

Not sure, but you can try to use --join-ns to tell CRIU which namespace to use as destination.

[lijunqiang123]

Yes, it sounds strange. In my usage scenario, I need to checkpoint the child process shmwrite and restore it instead of the parent process. I try to use --join-ns to test it later. Thank you.

[lijunqiang123]

I used the command criu restore --images-dir img-server/ -v4 -o rst.log --shell-job -J ipc:`pidof shmread` to restore the shmwrite program. But It didn't work. I don't know if I mistakenly used the - J command. Here are some information:

[adrianreber]

At this point I am not sure it is actually possible what you want to do. Not sure if CRIU can correctly restore a shared memory process if not all involved process in the shared memory are part of the checkpoint.

@avagin do you know if it is possible to checkpoint and restore only one process of a ipc namespace?

[Snorch]

Failed to create shm set error means that there is already shared memory with same key and shmget(key, , IPC_CREAT | IPC_EXCL) can't create another one. That is obvious consequence of that you dumped only one process from the pair of processes using the same shared memory, and then try to restore it near non-dumped process.

CRIU should only support dumping all tasks of IPC namespace as a whole, because it is default Container case.

Imagine that you hold ipc semaphore and criu dumps you, what should criu do with all other tasks which has access to this semaphore? Should criu create a dummy process that would still hold the semaphore for us and after restore pass the ownership back to you (so that no other task can access what is protected by this semathor)? Or should it just drop the semaphore and let other tasks use it but risking that your synchronization between your tasks would be broken badly? Programs which do synchronization in shared memory face the same problem. So in general case ipc can't be dumped, only a Container case.

[lijunqiang123]

Thank you for your reply. If I use memory-mapped files (mmap()) instead of shared memory to achieve inter process communication. In this case, can I use CRIU?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[Snorch]

Thank you for your reply. If I use memory-mapped files (mmap()) instead of shared memory to achieve inter process communication. In this case, can I use CRIU?

This is quite a hard question... Imagine two of your processes have mmaped file and they both write to it, and one process is dumped by criu in the middle of write, some data is already there and some would be written only after process restores. What would second process do with this half written data? Sharing memory in this way looks bad to me.

I would suggest using network sockets for communication between dumped and non-dumped parts, like we do for containers (though your programs should be written the way they can survive temporary unavailability of other end of connection).

JFYI: tcp established connections c/r https://criu.org/TCP_connection.