dynamic loading of assets (full-offline support) (WIP)

[foxsouns]

todo here now.

this is the pull request for dynamic loading of assets in webretro. for discussion, and a general "problem tracker". goals and stuff are above.

[foxsouns]

@BinBashBanana need some help on this: could you explain, to the best of your knowledge, how the cores are loaded in this? i need to know where it is hitching on to get past where i am at.

[foxsouns]

just realized that, by default, index.html doesnt even work locally (with internet), meanwhile i have a menu working (without internet). so, at least i have progress towards something functioning.

[BinBashBanana]

I'm already working on this for the next release (v6.6), see #52. Since cores are 2-6 MB (per core), I don't want to bundle them all in one file, but instead download them once and use the Cache web API for offline use. I will bundle the essential assets (javascript, css, etc) into the offline html file.

To answer your question, each core has 2 parts; the .js file and the .wasm file. The .js file is loaded from base.js, and the .wasm file is loaded using fetch from the .js file.

[foxsouns]

@BinBashBanana still going to work on this to a functional state on my own: i need something to spend my free time on anyways. i will adapt my end goal to that though: ends up with less work on your end for v6.6.

[BinBashBanana]

Ok!

[foxsouns]

Got core: snes9x
stderr: failed to asynchronously prepare wasm: [object ProgressEvent]
stderr: [object ProgressEvent]
Succesfully read ROM file "Super Mario World [USA, headered].smc"

..it never even loads the core at least i know for sure now, time to work on that

[BinBashBanana]

This happens when you load the core from a file:// url.

[foxsouns]

@BinBashBanana: This happens when you load the core from a file:// url.

Which is, indeed, what my purpose here is: any idea on why they do not function? Thinking about potential workarounds: I guess I could try to implement indexdb loading to get around it, but that seems like a handful that I don't exactly want to attack until necessary.

[BinBashBanana]

Most browsers disable XHR/fetch to file:// urls, for security reasons. You could theoretically embed the .wasm file into the .js file and it could work. Or see here.

When doing quick local testing, another option than a local webserver is to bundle everything into a single file, using -s SINGLE_FILE (as then no XHRs will be made to file:// URLs).

[BinBashBanana]

(Doing this may increase load times, because modern WASM streaming won't work)

[foxsouns]

Ah, I see. Everything works from Github Pages: now I guess I have to figure out how to actually use indexeddb, now. And.. also do that with the BIOS files, too.

...Or just embed the BIOS files in the repo.. but that's a bit storage-wasteful. probably just do the above, as much as that may suck to program.

[BinBashBanana]

This is why I want to use a service worker cache, it will make it much easier for the asset bundle, cores, and BIOSes.

In v6.6 I plan to make a feature that uses the service worker to store cores for offline usage. It will have a "manager" so that you can choose which cores to cache.

Originally posted by @BinBashBanana in https://github.com/BinBashBanana/webretro/issues/52#issuecomment-1267841139

[BinBashBanana]

Although, now that I think about it, service workers may not work on file:// urls either. I'll check.

[foxsouns]

PR comment is now a "todo" sort of thing.

[BinBashBanana]

MDN Says:

Note that the Cache interface is exposed to windowed scopes as well as workers. You don't have to use it in conjunction with service workers, even though it is defined in the service worker spec.

Looks like this will be doable!

[BinBashBanana]

Just a heads up, this is not dynamic linking. Dynamic linking is something else I have planned for the future, see here.

[foxsouns]

Just a heads up, this is not dynamic linking. Dynamic linking is something else I have planned for the future, see here.

sorry: must have mixed up "linking" with "loading" somewhere, haha. to clarify: i always meant "loading". oops.

[foxsouns]

@BinBashBanana: thinking of different ways this could be implimented: i have a few i could work on, and wanted your opinion on it.

possible approaches

• no dl frontend at all, download cores/bios files on first load (results in no visible change to the user, but only way to remove them is via deleting site cache).
• dl frontend is unobtrusive, available at the bottom of the core-select page. downloads are preformed automatically (but doable within menu; deletion as well). user doesnt have to learn everything, but the tools are there. cores are always listed, regardless of installed status.
• dl frontend is obtrusive: available immediately upon page load. user has to select assets before progressing to the core-select page (which will only include downloaded cores). forces user to understand the new way before progressing.

[BinBashBanana]

I think automatically downloading everything would be a bad idea, so I want to have a manager (one of the modal windows) for it. Unobtrusive would probably be best (?)

[foxsouns]

@BinBashBanana this is roughly what i had in mind (ignore the quick paint art)

[BinBashBanana]

Something like that yeah. There are no per-core assets, besides the BIOS files.

So it could look like:

• Asset bundle
• Core: Beetle PSX HW
  • BIOS: Beetle PSX HW
• Core: Nestopia UE
• Core: Genesis Plus GX
  • BIOS: Genesis Plus GX
• Core: Snes9x

[foxsouns]

right, that makes sense was thinking shaders were game-specific for some reason, lol here's the site as is: going to be fun robbing, er, probably what i can salvage from the sram manager code, and then implementing service worker backend stuff, and generating downloadable cores and bios stuff from a list..

[foxsouns]

i havent forgotten about this: life has only taken priority for a bit. it will probably stay like that for a bit more.. taking a couple of days to myself, to recover and focus on myself, and also thanksgiving stuff.

(and now, to close this off with a very Stallman-esk "Happy Hacking!")

[foxsouns]

sorry about going kind of stale on this. life has been rough, and i haven't been feeling very motivated. this is still something i am willing to work on, it's just going to take a bit.

[BinBashBanana]

It's all good. I frequently feel demotivated as well.