proot/chroot and termux-gfx-wrapper

[twaik]

It is known that we can not use Android (bionic) libraries inside glibc/uclibc/other environments.

We can use libhybris as a library loader, but still there is a problem with using EGL/GLES because in bionic EGL has dedicated TLS slots. Libhybris's libEGL is designed to be a wrapper around recompiled Android's libEGL with fixed (shifted) TLS slots and to use proper ANativeWindows designed to work with X11/Wayland/other (so they use Android's WSI to implement near-native X11/Wayland WSI implementation).

In the core Android's libEGL is a pretty simple loader which wraps all the objects it is passing to vendor's GL libraries (link). Termux-gfx-wrapper implements it's own WSI so in the case if we could dlopen vendor libraries (not with bionic linker, but with hybris) we could omit dlopening Android's libEGL and dlopen them directly to avoid a TLS+EGL headache.

But currently that is only my dream.

[Heasterian]

What about loading glibc GL libs directly with termux-gfx-wrapper glibc build directly in chroot/proot?

[twaik]

Why should we load glibc GL libs on Android even in glibc environment? it has no sence since vendors do not distribute glibc versions of their GL libs...

[Heasterian]

Some phones have Mesa drivers working fine with Android kernel, in this case it could work without dealing with bionic and glibc incompatibility.

[twaik]

So? Why not to load glibc Mesa libraries in glibc environment directly?

[twaik]

termux-gfx-wrapper is designed to load Android GL libraries. Built for bionic.

[Heasterian]

To use termux-gfx-wrapper WSI as devices don't have access to any Linux ones and software debug version is slow. Keep in mind that I'm just droping this idea here, not demanding anything. Termux-gfx-wrapper sounds like nice option to surpass lack of DRI3.

[twaik]

Do you mean using Android GL libraries inside bionic environment? That is the reason of starting this topic...

[tareksander]

I have a new idea for that: Have something akin to a virgl server running and listening on a path socket that is mounted in proot, the library inside the proot env opens the socket and forwards all commands, buffer reads and writes go through /proc/pid/mem for maximum performance. If only one process can be attached via ptrace, the server may have to lie in the proot binary. Should perform better than virgl, because you replace the mesa stack inside proot.

That solves the TLS problem while avoiding hybris, you just have to prevent the process from removing the dumpable flag, though proot should already do that so it can keep control.

You can get the pid of the peer socket to attach via ptrace, should work with chroot as long as the server runs as root (which chriot needs anyways) and SELinux doesn't interfere with that. Does the Android libEGL work with root?

[twaik]

If you want to implement pipe-based runner in different process only to avoid TLS issues there may be simpler solution (with using libhybris). You can run libhybris and dlopen/dlsym/run code in separate thread directly, without falling back to glibc functions and ensuring that your code will kill the thread after running to prevent running any glibc code (that usually runs after thread finishing). In this case you will get all bionic stuff mixed (kind of) in completely glibc environment with minimal overhead. You just must make sure that thread is started and killed with bionic functions, without running any glibc code.

[tareksander]

How do you build libhybris? Can it be included in the Termux repo?

[twaik]

Are you talking about building it for glibc-repo? Because there is no sense to build it for bionic. Also you should point that there are a few points that do not fit us.

1. Building recipes are designed to build only one version of linker at a time. We should patch it to build linkers for all android versions we support.
2. There are building recipes and source codes for supporting surfaceflinger, nfc, gps and other stuff which we can not use without root. And all this stuff requires ABI compatibility the availability of which we cannot guarantee. We should patch libhybris to not build all this stuff and restrict it only with base library and linkers.
3. Libhybris hooks some bionic functions with glibc alternatives which may cause TLS problems. We should disable it in the case if all stuff will run in the separate thread. Or we can keep it if it does not touch TLS.

[twaik]

Another solution will be to patch libhybris to backup glibc TLS and restore/create bionic TLS before every bionic function call and restore glibc and backup bionic TLS after finishing bionic function. This solution will have bigger copy overhead, but it will have less syscall/read_buffer/write_buffer/kernel_stuff/ipc_processing overhead so maybe it will be even more effective.

[tareksander]

Can you run programs from the glibc repo in proot or chroot? I think it should be in the repo and a setup script can copy it into proot/chroot env if needed. And yeah, the loaders would be enough. Are you sure the vendor EGL libs don't need TLS? In that case process separation is unavoidable.

[tareksander]

How does TLS work anyways?

[twaik]

TLS stands for thread local storage. It is a memory fragment specific for thread (allocated in pthread_start I think) for storing storage-specific data like GL interface pointer or errno. But the problem is that TLS slots in glibc and bionic have different purposes. No, both have errno, but it is stored on different offsets in these libcs.

That is the reason I suggested you to switch TLS storage contents before and after invoking bionic functions in glibc process. And to invoke them in separate thread. To avoid TLS mixing.

[twaik]

Can you run programs from the glibc repo in proot or chroot?

I do not think so. My fault, misunderstanding.

I think it should be in the repo and a setup script can copy it into proot/chroot env if needed.

I think we may have separate repo for building this project. CI can compile it for all needed archs and pack this into all package formats for installing in all distros we have in proot-distro. We should only write a workflow, it is not that complicated.

[tareksander]

In that case I think the best solution would be to have an amalgamation C file as an output similar to sqlite, include that in the package and the setup script can then compile it with the chroot/proot toolchain, so it works on any arch and distro. I think I'll try the server process way instead of hybris first, sounds like way less headaches.

[twaik]

Google already implemented it for using in AVD, you can check the code. It is a bit complicated, with a lot of serializations and deserializations. Probably not the fastest thing but can be fast enough for us.