Pathnames for named UNIX Domain Sockets

[dimakuv]

Description of the feature

We currently do not implement pathnames for named UDSes (see man 7 unix): https://github.com/gramineproject/gramine/blob/546c598b694d091121babc215bf6b9f9f52bbb4d/libos/src/net/unix.c#L11-L15

Actually, we need two features:

1. Pathnames for named UDSes.
2. Ability to find the UDS created in another, sibling Gramine process.

Pathnames for named UDSes

This shouldn't be too hard. We can re-use the logic already implemented for Named Pipes (FIFOs, see man 7 fifo):

• https://github.com/gramineproject/gramine/blob/546c598b694d091121babc215bf6b9f9f52bbb4d/libos/src/fs/pipe/fs.c#L304-L308
• https://github.com/gramineproject/gramine/blob/546c598b694d091121babc215bf6b9f9f52bbb4d/libos/src/fs/pipe/fs.c#L201
• https://github.com/gramineproject/gramine/blob/546c598b694d091121babc215bf6b9f9f52bbb4d/libos/src/fs/pipe/fs.c#L63
• https://github.com/gramineproject/gramine/blob/546c598b694d091121babc215bf6b9f9f52bbb4d/libos/src/sys/libos_pipe.c#L184

Basically, we create a uds_built_fs that allows to "see" the file in Gramine FS, and then create dentry and inode objects and associate them with the UDS handle.

Finding the UDS created in another, sibling Gramine process

This is harder. Because we have a scenario where one subprocess (wineserver, spawned from the main process bash) creates a UDS, and another subprocess (wine64, spawned from the main process bash) connects to it.

This means that wine64 will not see the created UDS in its Gramine FS view, because the UDS inode is not on the host (recall that what's created on the host is the UDS called smth like /tmp/gramine/0001/12345edcba). And this UDS inode is not in wine64 because the parent process (bash) doesn't have it too; this UDS inode is solely in wineserver.

So we need a mechanism to make the UDS synced to other processes (or some simpler approximation of this). Several suggestions:

1. Use the existing sync engine.
   • Pros: may be used to properly notify all sibling processes about UDS creation/rename/deletion/etc.
   • Cons: very complicated to use; we don't have a proper "checkpoint and restore a dentry+inode" mechanism to reuse.
2. Outsource UDS pathname to the host:
   • For example, during UDS creation, create a host file -- a symlink to the actual /tmp/gramine/0001/12345edcba UDS.
3. Lift UDS ("PAL pipe") name creation from PAL to LibOS. Then LibOS can create names for classic pipes and unnamed UDSes as PAL did before (local to a Gramine instance), but named UDSes will create names as-is.
   • Interestingly, this is also the approach considered for another issue: https://github.com/gramineproject/gramine/discussions/952 (starting with "Expose instance_id...").
4. Any other suggestions?

I strongly prefer approach 3.

Why Gramine should implement it?

This feature is required by Wine.

Wine is composed of two processes:

• wineserver serves as the Windows core (all Windows app's funccalls and syscalls are directed to this process),
• wine64 is the "runtime" for the actual Windows app.

The two processes communicate via a named UDS:

1. wineserver creates the UDS under /tmp/.wine-1002/server-1d82577547758f37-1df4df7ddb596480/socket pathname,
2. wine64 checks if the UDS is there: lstat("socket"):
   1. If wine64 doesn't find this file, then it tries to spawn wineserver again a couple times and then terminates with failure,
   2. If wine64 finds this file, then it proceeds with connecting to this socket.

The two processes are started independently in the same Gramine instance in this way:

#!/bin/bash
wineserver --foreground &
wine64 binaries/helloworld.exe

[boryspoplawski]

If I have entry in fs.mounts in manifest: { path = "/data", "file:/usr/var/data" } and create a socket at path /data/socket, where should the socket be on the host and how would other Gramine processes know about it?

[dimakuv]

If I have entry in fs.mounts in manifest: { path = "/data", "file:/usr/var/data" } and create a socket at path /data/socket, where should the socket be on the host and how would other Gramine processes know about it?

• The host socket will be created as file:/usr/var/data/socket.
• Other Gramine processes will use the same manifest, so they will do something like connect("/data/socket") which will translate into connecting to file:/usr/var/data/socket.
  • Recall that in the currently discussed problem, we are in the same Gramine instance (two children want to talk to each other via UDS), so the manifest file is the same, and thus the FS view is the same.

[dimakuv]

I started implementing this, and came up with the following proposal:

1. libos/src/net/unix.c changes:
   • use a new URI_PREFIX_UDS_SRV instead of URI_PREFIX_PIPE_SRV;
   • do not modify the UDS name (do not perform SHA256 hash).
2. PAL changes:
   • still use g_pipe_ops for this new URI_PREFIX_UDS_SRV (so UDSes are still considered pipes)
   • in pipe_listen(), pipe_connect() etc. check:
     • if URI_PREFIX_UDS_SRV, then do not add /gramine/<instance_id>/ prefix (i.e., UDS name is created on host as-is)
     • otherwise, have the old logic of adding /gramine/<instance_id>

This seems to require the least amount of changes and hopefully is sufficiently clear. @boryspoplawski WDYT?

UPDATE: Actually, this significantly complicated the PAL-pipes logic and overloads pipes with Linux-specific stuff. Scratch this idea; I'll keep it here only for history.

[boryspoplawski]

The host socket will be created as file:/usr/var/data/socket

Should have given a better example without matching parts. like { path = "/data", "file:/usr/var/B" }, but I guess you just want to concatenate the paths? But how would you then run two Gramine instances at the same time? You want to disallow such cases?

[dimakuv]

But how would you then run two Gramine instances at the same time? You want to disallow such cases?

I'm not currently talking about this scenario (from Ying). In fact, I decided it is orthogonal to my problem.

[boryspoplawski]

Why? I meant just running them, not making them able to communicate with each other. Note that your proposed solution makes them use the very same host path, which will most likely cause 2nd instance to fail (e.g. because socket creation fails or 1st instance fails because 2nd shadows its socket).

[dimakuv]

Ok, now I understand your questions. I guess the best answer is to check my PR: https://github.com/gramineproject/gramine/pull/986

But how would you then run two Gramine instances at the same time? You want to disallow such cases?

Yes, this will be disallowed. Two Gramine instances creating the same named UDS will conflict.

[dimakuv]

I found one more problem with the current code (https://github.com/gramineproject/gramine/blob/094c4e6ad586467ec1e918e1734a41861b6e989b/libos/src/net/unix.c)

Imagine we have two subprocesses, one server and one client:

    // server subprocess
    struct sockaddr_un sa = {
        .sun_family = AF_UNIX,
        .sun_path = "/tmp/unix_socket",
    };
    bind(s, (void*)&sa, sizeof(sa));

    // client subprocess
    chdir("tmp/");
    struct sockaddr_un sa = {
        .sun_family = AF_UNIX,
        .sun_path = "unix_socket",
    };
    connect(s, (void*)&sa, sizeof(sa));

This works correctly on Linux, but fails in Gramine. This is because Gramine rewires the name of the UNIX socket to a new name by hashing the contents of sa.sub_path, irrespective of whether the contents are an absolute path or a relative path (in the latter case, it should have been concatenated with getcwd()): https://github.com/gramineproject/gramine/blob/094c4e6ad586467ec1e918e1734a41861b6e989b/libos/src/net/unix.c#L54-L65

---

I tried to create a quick fix for this. But it turned out very cumbersome -- Gramine never operates on "path strings" but always on dentries. So there are helper functions like dentry_abs_path(existing_dentry, &returned_abs_path) . But nothing to concatenate cwd + sa.sub_path.

Adding such a "string concatenation hack" is ugly. And when we implement proper Pathnames for UDSes, we'll use dentries anyway. So even if introduced, this "string concatenation hack" will go away.

I think this is another reason to add Pathnames for UDSes.

[dimakuv]

I'll keep my local git diff for reproducing the problem from the previous comment:

diff --git a/libos/test/regression/unix.c b/libos/test/regression/unix.c
index c6b33e9b..127f8edc 100644
--- a/libos/test/regression/unix.c
+++ b/libos/test/regression/unix.c
@@ -17,7 +17,11 @@

 #define SRV_ADDR_NONEXISTING "/tmp/nonexisting/nonexisting"
 #define SRV_ADDR_DUMMY       "tmp/dummy"
-#define SRV_ADDR             "tmp/unix_socket"
+
+
+#define SRV_ADDR_DIR  "tmp"
+#define SRV_ADDR_FILE "unix_socket"
+#define SRV_ADDR      SRV_ADDR_DIR "/" SRV_ADDR_FILE

 static const char g_buffer[] = "Hello from UDS server!";

@@ -86,10 +90,12 @@ static void client(int pipefd) {
     }
     CHECK(close(pipefd));

+    CHECK(chdir(SRV_ADDR_DIR));
+
 #if 0 /* FIXME: currently Gramine doesn't reflect named UNIX sockets in file system */
     /* named UNIX domain sockets must create FS files, verify it */
     struct stat statbuf;
-    CHECK(stat(SRV_ADDR, &statbuf));
+    CHECK(stat(SRV_ADDR_FILE, &statbuf));
     if (statbuf.st_uid != getuid() || statbuf.st_gid != getgid()) {
         errx(1, "unexpected UID/GID of file `%s`", SRV_ADDR);
     }
@@ -102,7 +108,7 @@ static void client(int pipefd) {

     struct sockaddr_un sa = {
         .sun_family = AF_UNIX,
-        .sun_path = SRV_ADDR,
+        .sun_path = SRV_ADDR_FILE,
     };
     CHECK(connect(s, (void*)&sa, sizeof(sa)));

This should be applied on top of https://github.com/gramineproject/gramine/pull/996