Porting to webassembly

[saolsen]

Hey,

I'm thinking about trying to get blink running in the browser (via webassembly). My goal isn't just to run c code in the browser (could just use emscripten for that) but to actually run ape x64 executables in an interpreter where I can build a debugger and some visualization tools to see what's happening. (Similar to what blink already does). I'm wondering if anybody has tried to compile blink for webassembly or what you think some of the challenges would be.

[jart]

@duaneking could you attach your hello world binary to this issue tracker?

[duaneking]

Sure.

hello.c

#include <stdio.h>

int main(){
        printf("Hello World!");
        return 0;
}

Compile the a.out

cc -g -Os -no-pie -fno-pie -static hello.c -U_FORTIFY_SOURCE -fno-stack-protector -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wl,-z,common-page-size=65536 -Wl,-z,max-page-size=65536

Output from TUI:

I2023-01-13T15:02:08.705000:blink/syscall.c:2710:42 missing syscall 0x111
I2023-01-13T15:02:08.707000:blink/syscall.c:2710:42 missing syscall 0x14e
I2023-01-13T15:02:08.708000:blink/syscall.c:1942:42 getrandom() flags not supported yet
malloc(): corrupted top size
I2023-01-13T15:02:08.709000:blink/syscall.c:2414:42 tkill() failed: No such process
I2023-01-13T15:02:08.709000:blink/blink.c:66:42 terminating due to signal SIGABRT
Aborted(native code called abort())

Hope that helps.

[duaneking]

 file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=13a011f84eb1bf280499516bd405a19ec4c24c23, for GNU/Linux 3.2.0, with debug_info, not stripped

So gcc doesnt use sysv even when requested?

[duaneking]

hello.zip

[duaneking]

file tinyhello.elf tinyhello.elf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, no section header

[jart]

@duaneking It works for me:

$ o//blink/blink ./a.out |& less
I2023-01-13T15:12:30.686823:blink/syscall.c:2859:33689 missing syscall 0x111
I2023-01-13T15:12:30.686904:blink/syscall.c:2859:33689 missing syscall 0x14e
I2023-01-13T15:12:30.687197:blink/syscall.c:1960:33689 getrandom() flags not supported yet
I2023-01-13T15:12:30.688093:blink/xlat.c:524:33689 atf 4096 not supported yet
Hello World!

Could you tell us if it works for you if you run it with -m to disable linear memory optimization?

o//blink/blink -m ./a.out

If so, your platform toolchain is probably linking blink in such a way that it's conflicting with the loading of the executable you compiled. Chances are it's being loaded statically to the same addresses. In that case, it'd be really helpful if you could post the top ~50 or so lines of output of:

readell -Wa o//blink/blink

That will help me confirm my theory, in which case I'll get to finding a way to fix it right away.

[duaneking]

you and I are not testing the same things.

I'm using the TUI only. I'm not calling blink locally. I only came at this from the client perspective, using only the compiled version that was deployed online on github above.

The more I dig into this issue, it turns out that GCC just seems to do interesting things.

For the C application, I compiled that on Ubuntu via WSL, thinking it would get me a binary I could deploy at that link. I like the idea of being able to have a effectively virtual platform to deploy to but writing it in C is an interesting twist.

[duaneking]

Now pulling blink and running make...

[duaneking]

OK so if I compile it locally:

 o/blink/blink -m ~/a.out

I2023-01-13T15:41:18.581809:blink/syscall.c:2857:9325 missing syscall 0x111
I2023-01-13T15:41:18.581886:blink/syscall.c:2857:9325 missing syscall 0x14e
I2023-01-13T15:41:18.582101:blink/syscall.c:1960:9325 getrandom() flags not supported yet
I2023-01-13T15:41:18.583009:blink/xlat.c:524:9325 atf 4096 not supported yet
Hello World!

It works locally., just not at the linked above TUI.

Blink works perfectly locally for me with that binary. But what excited me was the browser part, and that's what I was trying to get working.

[duaneking]

Attaching the full readelf output of the binary. ;)

debug.txt

[duaneking]

I think that's everything I can give you. If you need anything else, just let me know.

[jart]

Thank you for the readelf output. What will most likely get you working locally right away is building Blink using our musl-cross-make toolchain. You can do that by running:

make -j8 o//x86_64/blink/blink

That will build it exactly the same way it builds on my machine. In this case, at HEAD you should see:

$ o//x86_64/blink/blink duaneking-hello.elf
I2023-01-13T17:04:59.550113:blink/syscall.c:2951:36025 missing syscall 0x111
I2023-01-13T17:04:59.550187:blink/syscall.c:2951:36025 missing syscall 0x14e
Hello World!

As for these:

I2023-01-13T15:41:18.582101:blink/syscall.c:1960:9325 getrandom() flags not supported yet
I2023-01-13T15:41:18.583009:blink/xlat.c:524:9325 atf 4096 not supported yet

I've just pushed a few changes adding support for fstatat(AT_EMPTY_PATH) and I've suppressed the getrandom() flags warning.

[trungnt2910]

You can #include <emscripten.h> and use EM_ASM(console.trace()); to print a backtrace.

Might it be useful to add this to the existing PrintBacktrace function?

[Vogtinator]

With -sASYNCIFY and some hacks (https://github.com/Vogtinator/blink/commit/12e08b8caf8dd01e4815084754042e9f0f737cfe) I was able to get blinkenlights.html to work somewhat:

https://vogtinator.github.io/blink/blinkenlights.html

By applying similar hacks to some syscalls (following commits on the master branch), interactive programs such as vi and sh work in blink.html as well to some extent:

https://vogtinator.github.io/blink/blink.html

[duaneking]

Just pulled the most recent changes and updated locally;

I'm loving that the only error output I get is the 0x14e syscall.

I think on linux that might be the compat_sendmmsg syscall?

Either way, Thanks for the fast update.

I reused the old client binaries with the new updated blink build, so nothing in terms of test input has changed outside of the upgrade.

$ o/blink/blink -m ~/a.out
I2023-01-15T14:23:39.729686:blink/syscall.c:3265:28627 missing syscall 0x14e
Hello World!

[jart]

Wow I'm really impressed to see Vi running in Blink in the browser. I'd love to be able to tweet that, once we get it up on https://jart.github.io/blink/blink.html

Since you're using BusyBox, I'd like to give you something even better. Here's what I call the "Blink Software Pack" which contains Actually Portable Executables from the Cosmopolitan project that are known to work well under Blink (because Blink was designed primarily with Cosmopolitan in mind). The listing is:

• qjs.com: Fabrice Bellard's QuickJS interpreter
• sqlite3.com: Standard SQLite command
• redbean.com: HTTP/HTTPS server with Lua and SQLite baked-in
• lua.com: Standard Lua interpreter with Redbean's UNIX module
• printimage.com: Renders GIF/JPG/PNG in terminal as XTERM UNICODE block characters
• finger.com: Finger protocol client, e.g. ./finger.com new_york@graph.no
• curl.com: Lightweight replacement for the curl HTTP client command
• unbourne.com: Fork of the Almquist shell with Bestline, a BSD-licensed GNU Readline replacement
• uname.com: Invokes the uname(2) system call
• ttyinfo.com: Interactive troubleshooting tool for ANSI escape codes
• sysinfo.com: Invokes the sysinfo(2) system call
• kilo.com: Antirez's famous 1000 LOC text editor
• hangman.com: The famous text-based game from UNIX Seventh Edition
• tidy.com: The HTML5 Tidy command
• ctags.com: Exuberant Ctags
• make.com: GNU Make w/ Justine's Landlock LSM security extensions
• python.com: Cosmopolitan's Python 3.6 interpreter in a single file
• awk.com: The One True Awk
• sha256sum.com: Replacement for the sha256sum command that goes 2x faster on modern x86
• mkdeps.com: Fast tool for generating Makefile code based on header file dependencies
• gzip.com: Compresses files using Zlib's DEFLATE implementation
• echo.com: Tool that writes a line of text to standard output
• rm.com: Tool for removing files
• mkdir.com: Tool for making directories
• pwd.com: Tool for printing current directory
• touch.com: Tool for updating timestamps on files
• cp.com: Tool for copying files
• dd.com: Tool for transferring blocks of data between files
• printf.com: Tool that formats a line of text to standard output
• chmod.com: Tool for changing file permissions

You can download the software pack here:

https://justine.lol/blinkenlights/blink-software-pack.tar.gz

One issue you may encounter with WASM in the browser, is many Cosmopolitan binaries assume they have the ability to open(argv[0]) so the WASM runtime should provide the executable access to read its own content. Otherwise programs like python.com won't be able to read the ZIP files contained within the executable.

[unicomp21]

Wow, you guys are amazing!

[duaneking]

I suggest that a sub-project be made to allow such software packs but as a more formal release, perhaps something such as blinkOS? ;)

[unicomp21]

I can't wait to see what happens w/ blinkOS, should be interesting!

[unicomp21]

Not sure if it's applicable here, but I know the leaningtech guys are pretty excited about having tail calls for webvm. Looks like they will land soon in both chrome and safari.

https://github.com/WebAssembly/tail-call/issues/15#issuecomment-1380699814

[jart]

I'm loving that the only error output I get is the 0x14e syscall. I think on linux that might be the compat_sendmmsg syscall?

That's rseq which is a deeply-Linux threading-related system call introduced a couple years ago that every Glibc binary seems to need. It doesn't have a man page aside this https://lwn.net/Articles/774098/ We should examine Glibc to confirm it has a fallback path for us returning ENOSYS, in which case we can suppress the warning.

https://github.com/jart/cosmopolitan/blob/be3e109309d7adc0824b8c22cf4fca3aa6433baa/libc/sysv/syscalls.sh#L370

[duaneking]

I'm loving that the only error output I get is the 0x14e syscall. I think on linux that might be the compat_sendmmsg syscall?

That's rseq which is a deeply-Linux threading-related system call introduced a couple years ago that every Glibc binary seems to need. It doesn't have a man page aside this https://lwn.net/Articles/774098/ We should examine Glibc to confirm it has a fallback path for us returning ENOSYS, in which case we can suppress the warning.

https://github.com/jart/cosmopolitan/blob/be3e109309d7adc0824b8c22cf4fca3aa6433baa/libc/sysv/syscalls.sh#L370

Yeah I was thinking an older version; https://thevivekpandey.github.io/posts/2017-09-25-linux-system-calls.html

Good catch.

[duaneking]

Linux rseq link for Reference: https://github.com/torvalds/linux/blob/master/kernel/rseq.c

I'm idling through code to see what I can find.

So it seems https://www.gnu.org/software/libc/manual/html_node/Restartable-Sequences.html confirms "The GNU C Library sets the cpu_id field to RSEQ_CPU_ID_REGISTRATION_FAILED if registration failed or was explicitly disabled." so I'm looking and https://github.com/torvalds/linux/blob/master/include/uapi/linux/rseq.h says thats defaulted to -2 currently.

This looks interesting:

https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=e3e589829d16af9f7e73c7b70f74f3c5d5003e45

[duaneking]

For future reference, a rseq search of the glibc code: https://sourceware.org/git/?p=glibc.git&a=search&h=HEAD&st=commit&s=rseq

[duaneking]

Ok found the rseq glibc code:

https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/sys/rseq.h;h=8533782cf4acf945a270abcbaa100bc806bf6cd8;hb=6c33b018438ee799c29486f21d43d8100bdbd597

[duaneking]

If its required is set by a value in the code as its checking ATTR_FLAG_DO_RSEQ, but of its needed and then doesn't init correctly, glibc calls __libc_fatal ,

https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/pthread_create.c;hb=569cfcc6bf35c28112ca8d7112e9eb4a22bed5b8#l376

[unicomp21]

"if" we'll soon have tail calls, which means emscripten will then support co_await and co_yield? Along with the compiler level optimizations they both provide? Does the need for emscripten "asyncify" go away?

https://github.com/jart/blink/issues/8#issuecomment-1383286674

[unicomp21]

actually, maybe only co_await requires tail calls? and co_yield already works? @tlively

[Niek]

Running busybox and vim is very impressive, a minimal Alpine would be even cooler (similar to https://bellard.org/jslinux/ / TinyEMU). Or is that out of scope of this project?

[unicomp21]

There are images here too ... https://github.com/copy/v86

[beriberikix]

Wow I'm really impressed to see Vi running in Blink in the browser. I'd love to be able to tweet that, once we get it up on https://jart.github.io/blink/blink.html

Since you're using BusyBox, I'd like to give you something even better. Here's what I call the "Blink Software Pack" which contains Actually Portable Executables from the Cosmopolitan project that are known to work well under Blink (because Blink was designed primarily with Cosmopolitan in mind). The listing is:

• qjs.com: Fabrice Bellard's QuickJS interpreter
• sqlite3.com: Standard SQLite command
• redbean.com: HTTP/HTTPS server with Lua and SQLite baked-in
• lua.com: Standard Lua interpreter with Redbean's UNIX module
• printimage.com: Renders GIF/JPG/PNG in terminal as XTERM UNICODE block characters
• finger.com: Finger protocol client, e.g. ./finger.com new_york@graph.no
• curl.com: Lightweight replacement for the curl HTTP client command
• unbourne.com: Fork of the Almquist shell with Bestline, a BSD-licensed GNU Readline replacement
• uname.com: Invokes the uname(2) system call
• ttyinfo.com: Interactive troubleshooting tool for ANSI escape codes
• sysinfo.com: Invokes the sysinfo(2) system call
• kilo.com: Antirez's famous 1000 LOC text editor
• hangman.com: The famous text-based game from UNIX Seventh Edition
• tidy.com: The HTML5 Tidy command
• ctags.com: Exuberant Ctags
• make.com: GNU Make w/ Justine's Landlock LSM security extensions
• python.com: Cosmopolitan's Python 3.6 interpreter in a single file
• awk.com: The One True Awk
• sha256sum.com: Replacement for the sha256sum command that goes 2x faster on modern x86
• mkdeps.com: Fast tool for generating Makefile code based on header file dependencies
• gzip.com: Compresses files using Zlib's DEFLATE implementation
• echo.com: Tool that writes a line of text to standard output
• rm.com: Tool for removing files
• mkdir.com: Tool for making directories
• pwd.com: Tool for printing current directory
• touch.com: Tool for updating timestamps on files
• cp.com: Tool for copying files
• dd.com: Tool for transferring blocks of data between files
• printf.com: Tool that formats a line of text to standard output
• chmod.com: Tool for changing file permissions

You can download the software pack here:

https://justine.lol/blinkenlights/blink-software-pack.tar.gz

One issue you may encounter with WASM in the browser, is many Cosmopolitan binaries assume they have the ability to open(argv[0]) so the WASM runtime should provide the executable access to read its own content. Otherwise programs like python.com won't be able to read the ZIP files contained within the executable.

Curious if anyone has tried these lately. They all are erroring for me, albeit is slightly different ways.

[trungnt2910]

Due to certain bugs in the CI process, the WASM preview has not been updated for a long time. You might have better luck trying a local build.

[RNRetailer]

Hi.

I'm trying to run a Linux binary in the browser.

It's written in Golang.

I can't compile for WASM because it has so many Linux dependencies.

I tried to run it here: https://trungnt2910.com/blink/blink.html

However, I got this error: "Uncaught RuntimeError: Aborted(Cannot enlarge memory arrays to size 1073811456 bytes (OOM). Either (1) compile with -sINITIAL_MEMORY=X with X higher than the current value 1073741824, (2) compile with -sALLOW_MEMORY_GROWTH which allows increasing the size at runtime, or (3) if you want malloc to return NULL (0) instead of this abort, compile with -sABORTING_MALLOC=0)"

Does this mean that emscripten must be compiled with -sALLOW_MEMORY_GROWTH?

I can pay if someone wants to help.

Thanks

[jart]

@RNRetailer If you can help fund further development on WASM integration, then please contact me by email.

[trungnt2910]

I can pay if someone wants to help.

I'm one of the authors of the WASM port and the owner of the page you mentioned. The page on my fork is a bit outdated compared to the official one, which itself has also been abandoned for quite a long time.

If you need my help updating the WASM version and fixing specific bugs, please reach out to me [at] trungnt2910.com or through my Discord.

[Vogtinator]

Does this mean that emscripten must be compiled with -sALLOW_MEMORY_GROWTH?

No, just build blink with one of the options.

However:

I can't compile for WASM because it has so many Linux dependencies.

Depending on the dependencies you might have to spend more time getting those to work with blink wasm. With the additional layer in between you'll also significantly lose performance.