Open badgermole opened 2 months ago
sbc100
commented
2 months ago Is worker = new Worker(new URL("my.js", import.meta.url), workerOptions) a problem on its own? Of is it only when doing nested bundling? i.e. is webpack re-writing this line to be something else in the bundled output?
The worker = new Worker(new URL("my.js", import.meta.url), workerOptions) is, I believe, OK for webpack. I fact we recently fixed an issue with that line and added test for it in #22165
badgermole
commented
2 months ago I was thrilled when I saw #22165 because I was having the same issue, and that did resolve it in that case. So a qualified 'yes' to does #22165 fix the non-nested case. However, it's still an issue with the project I'm working on now, even when not doing nested bundling as it turns out.
What does seem to work most for me is passing down the URL from the application when instantiating the module object.
sbc100
commented
2 months ago If you do new Worker(Module["wasmJsURL"].href, workerOptions); then won't it be impossible for webpack to see the name of the file you are trying to load and therefore make it impossible for webpack to "bundle" said file? Itsn't the reason webpack is trying to parse the call to new Worker so that is can bundle the worker file and re-write the new Worker call appropriately?
@RReverser any insights here? I'm under understanding things right?
RReverser
commented
2 months ago @RReverser any insights here? I'm under understanding things right?
Sounds right. Webpack needs specifically the new Worker(new URL('...', import.meta.url)) pattern.
badgermole
commented
2 months ago You're right. That and the file: protocol getting used for import.meta.url led me to exclude my.js from the webpack bundle, putting it in the same directory as my.wasm. But that only works if I can pass down the URL so it can be picked up where the worker is created.
At this point I can say, for the non-nested case, new Worker(new URL('...', import.meta.url)) does work. But for the nested case, I still haven't seen my way through. I get:
my.js:1582 import.meta.url: file:///D:/.../wasm_export/bin_pt/dbg/my.js
my.js:1583 Uncaught (in promise) SecurityError: Failed to construct 'Worker': Script at 'file:///D:/.../wasm_import/mywasm/bin_pt_dbg_my_js.MyWasm.min.js' cannot be accessed from origin 'http://localhost:8088'.
at Object.allocateUnusedWorker (my.js:1583:14)I've put together as minimal an example as I could into repo emsdk_pthread_nested_webpack if anyone has time to take a look.
Another interesting note is that webpack duplicates my.js. It puts it once in a hash-named file, and once in MyWasm.min.js. And it also gives a circular dependency warning. However, if I exchange:
worker = new Worker(new URL("my.js", import.meta.url), workerOptions);
for
var my_js = "my.js";
worker = new Worker(new URL(my_js, import.meta.url), workerOptions);
it doesn't do that, but puts it only in MyWasm.min.js.
sbc100
commented
2 months ago Right, I guess when you do worker = new Worker(new URL(my_js, import.meta.url), workerOptions); then webpack doesn't know what file you are trying to load so it can't actually bundle said file and it gives up. You are effectively defeating the bundling process I believe that just falling back to using your original unbundled "my.js" file.
Not that in both cases don't you end up two seprate copies of the main js file? You either end up using a hash-named file or you end up using your original my.js that emscripten generated? Is that right?
This seems unfortunate and kind of defeats the point of bundles doesn't it? i.e. without bundling you just have a single my.js file, but with bundling that code in my.js exists in two different files that must be downloaded to the client separately. Is the right @RReverser ?
badgermole
commented
1 month ago I apologize for neglecting this - I had to switch tasks for a bit. I appreciate your comments and questions up to this point.
I should also try opening an issue in webpack's repo since I'm not convinced there isn't a webpack-based solution. But at the moment, I've worked around by post-processing to change
worker = new Worker(new URL("my.js", import.meta.url), workerOptions)
to
if ( Module[ "wasmJsURL" ] ) {
worker = new Worker( new URL( Module[ "wasmJsURL" ].href ), workerOptions );
}
else {
var jsFile = "my.js";
worker = new Worker( new URL( jsFile, import.meta.url ), workerOptions );
}@sbc100, like you said, I end up with two copies, one bundled and one not. So I'll still be probing for a real solution. But it does get me past this for the moment.
I'm not sure what your policy is for closing issues. I don't really consider this closed, but it's seeming like it may drag on a bit given the time I can spend on it.🫤
Version of emscripten/emsdk: 3.1.64
Please include the output
emcc -vhere:Failing command line in full: N/A
Full link command and output with
-vappended:Issue: With a pthread build, and bundled with webpack, creation of new Web Worker fails due to protocol=file: in import.meta.url.
The short description is that I have a project (project-A) where we are compiling our WASM target with
-pthread. Project-A is bundled with webpack. Project-A is in turn used by project-B, which also uses webpack as a bundler, so the bundling is nested. In the Module JS file, the lineworker = new Worker(new URL("my.js", import.meta.url), workerOptions);fails because the protocol ofimport.meta.urlis ‘file:’.I have not been able to find a webpack-based way to work through this issue. The workaround I have involves hand-editing my.js to do the following (minus error checking here):
worker = new Worker(Module["wasmJsURL"].href, workerOptions);, wherewasmJsURLis ascertained by the main application since it understands thelocation.originand path, and it can send the complete URL during instantiation of the WASM JS Module objectnew MyAppWASM( { instantiateWasm: onInstantiateWasm, noExitRuntime: true, wasmJsURL: this.wasmJsURL } )….Might there be another approach that I’m not aware of? I can put a minimal example in a github project if needed. Or is it a valid request to have Emscripten offer the ability for passing the WASM JS URL during instantiation for use in Worker creation?
Here are a few references I've collected that have, or may have, bearing:
22140
20580
#16878