Build target for NPM with ESM-based Core that has TSC compliant .d.ts files and without the `.ts` source files.

cmidgley commented 1 week ago

Proposed change

Expand the NPM distro to include a dist style directory with ESM (such as _dist/esm) specified in tsconfig and generated by tsc. The dist output should include only the .js, .d.ts, and .map files (no .ts files). By using tsc (and not Deno) to compile, the resulting .d.ts files would be compliant with tsc.

This request is for all the repos associated with the nats.js client.

Use case

I am targeting the `Moddable XS' platform for microcontrollers. It is an excellent modern implementation of JS, but implements only ESM.

I am so far only working with Core (and its associated nkeys and nuid` packages), and I've tried using two versions:

npm install @nats-io/nats-core: While this is Node-compatible for ESM, the included js source files are CJS and use require. This will not work with Moddable.
npx jsr add @nats-io/nats-core: This is ESM, but the supporting .d.ts files are Deno compliant and specifically the header.d.ts file is incompatible with TSC. For example, line 18 has:

findKeys(k: string, match = Match.Exact): string[];

This has an initializer in the interface, which is not allowed. TSC gives this error:

 error TS2371: A parameter initializer is only allowed in a function or constructor implementation.

I've worked around the issue for now, by building a separate module that builds everything from the TypeScript source files located in the node_modules directory, but it would be cleaner if there was a dist/esm style directory that had tsc compiled files (including only the .js, .d.ts, and .map files, without the .ts files). This last point is because Moddable gets confused over the duplicate files in the same directory (the .ts and .js) and tries to compile both.

I'm still working through getting this to run, as I need to add some polyfills and Node API compatible libraries (such as TextEncoder/Decoder, WebSocket, and more) but it is compiling and loads/imports on the microcontroller, which is a great sign that it may work with the current source tree (using WebSockets). A custom transport would need to be written to use native IP, but that does not appear to be a big project (thanks to the excellent isolation in this architecture) and can be maintained in a separate repo.

Contribution

Yes, but I have no Deno experience and don't yet understand the build process involved.

aricart commented 1 week ago

The JSR libraries are built by JSR, so there's no control there.

I explored the possibility of doing esm and cjs on npm, but the curious thing is that while node supports different entry points (https://nodejs.org/api/packages.html#conditional-exports) tsc doesn't or at least didn't as of a couple months ago). But as you may know, tsc can build a library that is happy to work with cjs and esm. Have you tried that?

I would like to have an esm on npm, but as you are finding out the issue here is not a packaging issue, it is also a runtime issue. I don't specifically want to support another runtime unless we have the demand for it. With that said if there's a way of having the npm libraries be a peer library to the entry point and shims for moddable, that would be great. I would also encourage you to ask them (as they are a framework for building) to be the flexible thing. I can change my libs all I want, but always some bundler etc will not be happy.

aricart commented 1 week ago

Also I believe that soon enough deno will properly support node, which means that the deno libraries will be the node ones, as Deno is striving to be the runtime for node...

cmidgley commented 1 week ago

Appreciate the fast and detailed response and totally understand the lack of demand perspective - I doubt many (if any) others will have this issue.

You can tell tsc to build for esm vs. cjs via tsc --module esm --outdir ./dist/esm, or by a separate tsconfig.json (that extends the base and overrides module/outdir), but not sure if that addresses your concern. But as I said, I don't follow how Deno does this build process so I don't know if that's easy or not.

Feel free to close this issue out if it's not a dead simple thing to fix. It's working, I just have to have my own build setup to compile the .ts files, so any new files added by you will need me to adjust it, but it's forked so I have control over merge. But due to how I have to do it, it won't be something I can share back because it deeply tied into my own custom build system. Perhaps if Deno eventually generates tsc compatible types we can figure out a way to have a Moddable repo to share with others. It's a pretty amazing combination using NATS, your JS client, and Moddable for distributed microcontrollers (assuming it all works, which I won't fully know for a while!)

Thanks!

aricart commented 1 week ago

@cmidgley my suggestion is that you try @nats-io/nats-core from npm. That set of bundles is tsc generated... Are you using the we socket transport?

cmidgley commented 1 week ago

Thanks - I did try that, but the transpiled .js files use require (which works with both esm and cjs with Node). Moddable only supports import...

I am starting with the web socket transport. I'm just now finishing up a w3c compliant web socket client for Moddable now, and already added all the polyfills that nats.js uses from Node, so I'm hoping I'll start running it very soon. If I find success, I'll then move onto the other modules (jetstream, kv, services) and eventually look at creating a transport (though ws will hold the fort for a while).

I really appreciate the help, but I'm going to close this as it's working good enough and I'd hate to take up any more of your time. Thanks again.

nats-io / nats.js