Building from sources. I succeded to build Xvfb by using latest xserver source with latest NDK. Archive.tar.gz.
If you are planning to use latest xserver source you should port Xsdl to latest kdrive or use Xvfb source with some code fragments from Xwayland source (to change resolution of virtual framebuffer) and to push input events to input queue.
It is possible to create Unix domain socket in chroot/proot environment even without root access. For that you will need something like my shell loader (link). It can load code from main apk which will create unix socket and pass unix server socket via Binder to your server and then you will be able to add it to server using simple ListenOnOpenFD call.
ListenOnOpenFD
call.