Open mwcz opened 9 years ago
My bad, here's the source.
Glad to hear you're working on this!
First a question for you: you said you're using the onprogress
event: does that give you chunks of the downloaded file in ArrayBuffer
s? I thought this only worked if you set the responseType
to moz-chunked-arraybuffer
which isn't portable outside FF. I had been looking into this question and my novice conclusion was that the only way this would work was (1) WebSockets, (2) HTTP Range
requests, (3) splitting into multiple files. I'd be delighted to be wrong, though.
To answer your question: the basic way I was thinking this would work would be to break _asmjs_unpack
into two functions: _asmjs_start_unpack
which returned the State
object (instead of storing it on the stack), and _asmjs_continue_unpack
which took the State
object. _asmjs_start_unpack
would do everything up to function_definition_section
and then continue_unpack
would just unpack as many functions as it could and return when it ran out of input or finished. Ideally, function_definition_section
(which is just a simple loop), would know before starting to decode a function if it had enough bytes to finish decoding the function. This could be achieved by having the function_declaration_section
include the input size of each of function.
@lukewagner Very helpful, thanks for the info. :+1:
I suspect you're right about moz-chunked-arraybuffer
. I've only tested in Firefox so far, but FF passes into the onprogress
handler (among other things) a string containing the binary blob, not an ArrayBuffer
. The string can be converted to an ArrayBuffer, but that might be too expensive (then again it might be fast?). onprogress
fires for every byte received, or every 50ms, whichever is less frequent. We could easily throttle that even further though inside the onprogress
handler. Doing a string->ArrayBuffer conversion 5000 times during a download might be really slow, but 5 times might be okay (while still having a good early-decoding boost).
I think WebSockets is the ideal distribution protocol, but the best first step IMO is to implement something as basic as possible, which is compatible with all HTTP servers. That would make adoption pain-free.
I'll do some testing & research on the following things:
onprogress
and converting the response string to an ArrayBuffer (requires zero server config, pain-free for devs)ArrayBuffer
s with moz-chunked-arraybuffer
and finding out whether equivalents for other browsers have sprouted latelyGreat! I'd also check other browsers on whether they pass strings containing the binary data (is it base64 encoded?) to onprogress
.
The strings Firefox passed to onprogress
were plain JS utf-8 strings with plenty of non-printable characters. The XHR spec doesn't call for base64, but it does say "Responses must have the content-encodings automatically decoded."
Note to self: also try onprogress
with xhr.responseType = 'ArrayBuffer'
and see what the response looks like each tick.
Hey @lukewagner et al, I've been experimenting with chunked decoding and I have some questions.
My experimental setup is using the
onprogress
event ofXMLHttpRequest
for mid-transfer control. This is working well. I suspect WebSockets might be a better way to serve this stuff, but I'm looking at XHR first because it requires less up-front setup (every webserver supports GET, but ws:// takes more work). Anywho, on to the questions.Currently, the polyfill downloads the complete binary, then unpacks it. Unpacking involves a couple of mallocs and then a call to
_asmjs_unpack
.One malloc reserves space for the incoming bytes. No problem here, we've got access to the total bytes as early as the first
onprogress
event. There's another malloc that has me concerned, though. It reserves space for the unpacked asmjs code, as determined by_asmjs_unpacked_size
. Since I'm not sure how that function works, I'm not sure we can malloc that space before the download is complete (how does it know?). Perhaps it uses some theoretical maximum size of asmjs that can be generated by a given number of wasm bytes?Beyond the mallocs, I need some guidance on how
_asmjs_unpack
is going to work when unpacking an incomplete binary, a little bit at a time. The incoming binary slices will be at arbitrary positions, so_asmjs_unpack
(or a wrapper function) will need to intelligently handle things like partial opcodes, opcode without args, etc. Without source or docs, it's hard to know what to expect.My questions all boil down to this: what steps should be taken during each
onprogress
tick? and where can I find source or docs for the_asmjs_*
functions?