second-state / wasmedge-quickjs

A high-performance, secure, extensible, and OCI-complaint JavaScript runtime for WasmEdge.
Apache License 2.0
477 stars 59 forks source link

Files over certain size don't seem to return correctly #132

Open lastmjs opened 7 months ago

lastmjs commented 7 months ago

I haven't found a solution to this problem yet, but I've tracked the issue down to here: https://github.com/second-state/wasmedge-quickjs/blob/main/modules/internal/fs/stream.js#L175

In my Express app I'm trying to load a file that is just slightly bigger than the size of this buffer. I get this error:

"    at ERR_OUT_OF_RANGE (azle_main:6274)\n    at __node_internal_ (azle_main:6921)\n    at read (azle_main:20327)\n    at apply (native)\n    at <anonymous> (azle_main:7055)\n    at Promise (native)\n    at read (azle_main:7059)\n    at read (azle_main:21716)\n    at read (azle_main:19157)\n    at <anonymous> (azle_main:10176)\n    at resume_ (azle_main:9920)\n"

I will keep trying to track down a solution, but this is a major problem.

lastmjs commented 7 months ago

It seems like I have found the root cause of the issue. The file reading is supposed to be batch processed using buffers of length 16 * 1024, and the algorithm seems setup for that, but the parameters to this.file.read or very incorrect, leading to the batching algorithm not being used, so if you have a file over the original buffer size it will simply fail.

This is where the code needs to be changed: https://github.com/second-state/wasmedge-quickjs/blob/main/modules/internal/fs/stream.js#L218

Here's what I've done so far that seems to fix the issue:

const preLength = opts.end - opts.start - curPos + 1;

const { bytesRead: n } = await this.file.read(buffer, 0, preLength > buffer.byteLength ? buffer.byteLength : preLength, curPos === 0 ? opts.start : opts.start + curPos);
lastmjs commented 7 months ago

It seems that this code has not been well-tested...are there not Node.js or other test suites available to test against?