Rewire the identity of a non-main thread upon `execve()`

[dimakuv]

Description of the feature

It is possible that a non-main thread performs execve() to substitute the executing image in the same process. At this point, the non-main thread "assumes" the identity of the (potentially terminated) main thread.

Linux solves this corner case as follows: the main/leader thread is terminated, and the non-main thread assumes its identity (in particular, its PID): https://elixir.bootlin.com/linux/v6.0/source/fs/exec.c#L1078

1277 introduces the parking of the host-OS main thread, so that Linux doesn't lose track of the Gramine process itself. But that PR doesn't change how Gramine behaves internally. Ideally, upon execve(), the non-main thread must "rewire" itself to assume the identity of the main thread.

This may be not as simple as non_main_thread->tid = g_process.pid (and similar rewiring of process-wide IDs):

• Do other Gramine processes (e.g., children) recognize this non-main thread? Does anything change in terms of e.g. how signals are sent and delivered?
• Could there be a data race between this rewiring logic and IPC helper thread? What if some signal comes exactly at the moment of such rewiring?

See #1277 for details.

Why Gramine should implement it?

Because that's what Linux does.

[mkow]

A simple and a bit hacky repro: (the loop and sleeps are there because I was playing with sending signals to it)

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <pthread.h>
#include <time.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/resource.h>

void* f(void* p) {
    char* argv[] = {(char*)p, strdup(""), NULL};
    printf("tid: %d, nice: %d\n", (int)gettid(), (int)getpriority(PRIO_PROCESS, 0));
    setpriority(PRIO_PROCESS, 0, 3);
    printf("tid: %d, nice: %d\n", (int)gettid(), (int)getpriority(PRIO_PROCESS, 0));
    sleep(7);
    execve((char*)p, argv, NULL);
    abort();
}

int main(int argc, char** argv) {
    setbuf(stdout, NULL);
    setbuf(stderr, NULL);
    if (argc == 1) {
        pthread_t t;
        pthread_create(&t, NULL, f, argv[0]);
        pthread_exit(0);
        return 0;
    }

    for (;;) {
        printf("tid: %d, nice: %d\n", (int)gettid(), (int)getpriority(PRIO_PROCESS, 0));
        sleep(1);
    }
}

Sample output on Linux:

tid: 3184510, nice: 0
tid: 3184510, nice: 3
tid: 3184509, nice: 3
tid: 3184509, nice: 3
[...]

Sample output under gramine-direct:

tid: 2, nice: 0
tid: 2, nice: 0
tid: 2, nice: 0
tid: 2, nice: 0

(yup, we have dummy nice in Gramine, but it's there to show that on Linux it's really a thread changing its TID, not execve being executed in the main thread).