Libc, pthreads, and rust

[gparmer]

@Others, when porting Rust to composite, has run into an issue with musl and pthreads. To set up thread local storage (TLS) for each thread, we must execute pthread code. Parts of the rust standard library assume that they can use TLS. This starts taking us down a path where we just use libc interfaces for pretty much everything, instead of raw, low-overhead composite interfaces. So is there a design that gives us all benefits: use of TLS as expected for Rust and other run-times, efficient use of low-level composite interfaces when required, and low software eng. requirements (up-front, and in maintenance)?

A non-exhaustive set of options include:

1. Change __pthread_self in musl to avoid using TLS, and instead use a global variable. This can essentially just be a field in the current sl_thread. Unfortunately, this might fix pthreads in musl (as in, they won't use TLS anymore), but if Rust uses TLS in its own standard library (or in crate x, y, or z), then we aren't left with many options. Thus, this might work well for musl, but not necessarily for libraries stacked on top. Are these problems we want to solve separately? Perhaps.
2. Write our own pthread library. The API is pretty broad so this would be a pain. It isn't clear how much of musl would break if we do this. How much of it depends on the current implementation of pthreads? Or are threads kept somewhat isolated?
3. Use the pthread API for our components that require higher level services in languages such as rust. Unfortunately, the appeal of a language like rust is that it is low-level enough that we should be able to avoid a lot of POSIX BS when possible.

Are there any options that I'm missing?

This is complicated by the fact that components will execute in different environments. Sometimes with access to many resources, sometimes with access solely to invocations to a manger (see #251 for more details). The backends for how we handle different musl system calls will need to be different depending on the environment. This adds another dimension of complexity to this issue.

@phanikishoreg @ryuxin

[Others]

Notes on the options presented here: 1) This option sounds nice, but I don't think it really works. We have to call libc's thread initialization routines to set up pthread self correctly, and so we basically don't get anything out of doing it this way. 2) I favor this option. We can just stub out all the methods to fail, and then add them incrementally. I've spent a lot of time in libc internals, and I think I could do this. Since "pthreads" is a fairly high level API, we should just get composite level performance with compatibility. Additionally, we can make the api work with threads created outside the api. 3) If we pursue this option we basically have to fix all the current issues with pthreads. This means making the libc initialization routine work, perfectly emulating complex linux system calls, and generally a whole lot of annoying work. (I actually think this is more work than 2.)

[gparmer]

@Others thanks for the comment!

1. Agreed with option 1 that it doesn't solve the whole problem. We would still need to figure out how to work with pthreads. However, if we continue to have TLS issues, it is an orthogonal option. But lets not get side-tracked: you're right that this is a different thing.

2. Why do you think that option 3 is more work? I'd appreciate your context here. My concerns incluce: implementing condition variables alone would be a huge effort to get right; keys and all the varieties of join and attributes would be annoying; cancelling seems atrocious; RW locks seem painful. I think this requires actual evidence-based analysis, not solely high-level statements. It is 2700 LoC in musl. For your reference: http://cursuri.cs.pub.ro/~apc/2003/resources/pthreads/uguide/users-gu.htm#301162

3. For option three, emulating general system calls can often (but not always) be easier than emulating those with complex, particular semantics. Futex emulation, and clone emulate are the key parts of the Linux system call layer that would require emulation. Other smaller system calls for TLS, scheduling, etc, I'd imagine would be quite a bit easier (but I haven't though this through tempered with enough examples to say anything beyond a "feeling"). It is hard to convince myself this is more work than option 2. I recognize that there are fiddly bits around initialization and such, and getting the system calls right will not be nothing. But the amount of debugging involved in getting pthread emulation working is also pretty large.

Summary. I think that what we can say is that if parts of musl outside of the pthread code require pthread capabilities (i.e. if any part of the system outside of pthreads uses TLS), then option 2 is difficult. We'd end up in a situation where we have our own library, but libc code assumes pthreads are used. errno is one example where this might be an issue. I don't have faith in my imagination to predict others. There is some evidence that the rest of the library does rely on pthreads: pthread-specific code is executed at initialization.

If this is not an issue, and pthreads are a "drop in" part of the library (i.e. the library works completely without them), then we might be able to get a pared down version of option 2 working. I'd guess this is what @Others has in mind.

I very much don't like the requirement of TLS in each component, so Option 2 is very appealing to me. But there has to be an evidence-based discussion about the options. Given that, a list of quesitons:

• Can you describe the initialization problem? Is there an inherent problem, or does it just require hacking around edge cases?
• Do we have any idea how much of musl relies on pthreads? Do we have a way to find out?
• Do we know how much of the pthread API rust uses? Do we have a sense how much of it applications (e.g. nginx) use? @RobertGiff

[phanikishoreg]

Is TLS the main problem with many different runtimes, libraries that we foresee to support? I think we need to talk in a meeting to get broader context and be able to understand for ex: why we moved to musl from diet or even diet in the first place. Moreover, I don't know if this problem is easier to solve or requires less work if we use a different libc. I'm still trying to understand this problem thoroughly.

Given the above context and options, I think Option 3 is the most viable one. Though it's not as easy and we'll have obstacles, if we end up using pthread and fixing all of it's issues, we might be able to use it for broader runtimes, usecases etc.

[gparmer]

@phanikishoreg I agree a meeting is in order. Perhaps Wednesday?

[gparmer]

It feels like our options for a pthread emulation library vs. a syscall emulation library comes down to a single factor: what rust crates do we want to support, and how controlled is the rust code. I'll illuminate what I mean by this with examples:

1. We want to write a low-level component in rust. We want to use the standard containers, concurrency facilities (channels, Arcs, and Mutexes), and threads. We do not care about supporting general crate execution, and instead focus on a low-level environment for OS code.

2. We want to run most crates (where possible), and get full-featured Rust up and running. All rust, all the time. But because this environment will have a lot of dependencies on the surrounding system (memory management, FS, networking, scheduling, etc...), it is unlikely to be appropriate for OS-level components.

I believe our discussion has been conflating these environments. I believe these environments require different designs. Without thinking too deeply about the details, it seems to me that we'd want:

1. A custom version of many libc functions including but not limited to pthreads. In other words, libc emulation for some core aspects (though not all of it, memcpy, etc...). This can be implemented since musl uses weak aliases for many of its APIs (see https://git.musl-libc.org/cgit/musl/tree/src/mman/mmap.c#n35, and https://git.musl-libc.org/cgit/musl/tree/src/thread/pthread_create.c#n311).
2. System call emulation so that we can make maximal use of libc for all the crates that supports its broad functionality. This is implemented with the libposix we already have.

So in pictures, we have

+-------------+----------+------------+
| Rust crate  |Rust crate| native app | 
+---------+---+----------+--+---------+
| Option 1|                 |         |
+---------+                 +         |
|         | musl libc       |         |
+         +-----------------+         |
|         |  Opt 2/libposix |         |
+         +-----------------+         |
| cos_kernel_api/sl/...               |
+-------------------------------------+
| Composite Kernel/VK                 |
+-------------------------------------+

Please note that this discussion is broader than Rust. If we want to use our own threading libraries and libc, we'll have this issue (see the "native app" above). This is more of an issue of supporting general legacy code vs. supporting our own abstractions. Note that there is likely more interesting research for the latter.

[phanikishoreg]

@gparmer Wednesday sounds good to me!

[gedare]

In RTEMS we use newlib with custom threading. I don't know if switching your libc is an option for you to consider.