search

anonhostpi / DBPF.js

A simple library for reading DBPF files in TypeScript/JavaScript from the browser and node
https://anonhostpi.github.io/DBPF.js/
MIT License
1 stars 1 forks source link

Using a new WebAssembly target in place of a JS reimplementation from-scratch? #2

Open chances opened 1 month ago

chances commented 1 month ago

See https://github.com/chances/dbpf-d/issues/4.

Justification

chances commented 1 month ago

I'd also like to discuss extracting your specification into it's own GitHub organization that we share.

anonhostpi commented 1 month ago

I like the idea, but I have a few concerns. Primarily, the most pressing concern is multithreading. I'm currently working on a multithreading implementation that leverages Web Workers (and Worker Threads in Node.js).

AFAIK the only way to achieve multithreading in WASM is through Web Workers. I believe this means that there would still have to be some level of interop between wasm and js to make multithreading usable.

Other concerns I have are:

anonhostpi commented 1 month ago

For multithreading, I'm currently using a client-server relationship between the main (client) and Worker Threads (servers) to match the Worker.postMessage design.

As for shared data, I'm primarily targeting Blob URLs, since Blobs are paged. For Node.js, I'm just using File URLs and fs ops for reading file data.

chances commented 1 month ago

Regarding your parallelism concerns:

D has a fairly robust infrastructure of primitives in the standard library to support both SMP Concurrency and full-blown Parallelism. However, I am not sure how the LLVM backend handles parallelism in compiled WASM modules. (Note to self: Research the parallelism story and its support (or lack thereof) in LDC.)

Re: Shared Data

Note to self: Regarding parallelism concerns, SharedArrayBuffer has limited support across JS runtimes.

chances commented 1 month ago

Or, perhaps, we just focus on the inverse, i.e. create a WebAssembly target in this library by compiling a shared library via AssemblyScript?

@anonhostpi How far along is SimCity 4 support in this library?

anonhostpi commented 1 month ago

Or, perhaps, we just focus on the inverse, i.e. create a WebAssembly target in this library by compiling a shared library via AssemblyScript?

I was thinking that a good starting point would be to implement wasm in isolation on each thread interoped with the main thread via JS. It would be a hybrid implementation of wasm and JS/TS. Where wasm is responsible for parsing Blobs and JS/TS is responsible for interop.

SharedArrays will likely not be necessary, as the file content is best handled/shared via Blob/File URLs, and the content of each parsed entry can be serialized and returned back to the main thread via postMessage (or similar)

@anonhostpi How far along is SimCity 4 support in this library?

So, I'm still working on writing handlers for the contents of each DBPF entry, but the library currently supports parsing the header and index table for any DBPF version up to v2.1 (v1.x included)

Support for DBPF version 3.x is in the works, but it requires that I dig into Spore (and possibly Darkspore)

chances commented 1 month ago

@anonhostpi How far along is SimCity 4 support in this library?

So, I'm still working on writing handlers for the contents of each DBPF entry, but the library currently supports parsing the header and index table for any DBPF version up to v2.1 (v1.x included)

Support for DBPF version 3.x is in the works, but it requires that I dig into Spore (and possibly Darkspore)

This is good to hear! My D library barely functions, if at all, to parse v1.x archives.

Since I'd rather not reinvent the wheel, I'm going to experiment in my fork by wrapping this library in an Extism plugin.