Add an ESM build to @simplewebauthn/server to support non-Node runtimes

[rakeshpai]

I know I'm probably taking this lib to environments it wasn't designed for, but I thought I'd try my luck anyway.

I'm working with Miniflare, which is Cloudflare's local dev environment thing. When I invoke generateRegistrationOptions, I get the following error:

Error: Dynamic require of "stream" is not supported

All the stack frames are in my code, because the code is minified before it is passed into miniflare, so it's very hard to get to the root of the problem. However, from what I could find out from the minified code was that the issue is probably not with the SimpleWebAuthn code itself, but with the way it is packaged.

Here's a screenshot of the offending code, if it's useful.

My suspicion is that this could be fixed by packing up the server code as ESM, maybe? Right now, the @simplewebauthn/server:/server/dist/*.js is using node's require syntax, which requires me to fall back to miniflare's legacy handling of modules, which I'd rather not do.

Is there a chance that you could package up the server in an ESM format for consumption in modern ESM environments?

[MasterKale]

@rakeshpai Argh, I should have known it was too good to be true.

299 laid the ground work for an ESM build by refactoring away all dependency on NodeJS-specific data types or APIs. That PR did not refactor away the fact that the library is still CommonJS, and thus difficult to incorporate into ESM-based environments.

The next thing I plan on tackling is adding an ESM build, so that the project can continue to support CommonJS-based Node projects, while also working in Deno, Bun, Cloudflare Workers, etc... No ETA on that effort, but it's on my roadmap. In fact I'll make this issue the one about adding ESM support, and close this out when I finally deliver.

[Hexagon]

I've done it, but on another lib 😄

You might be able to pick up some goodies in these PRs

CJS to ESM: https://github.com/webauthn-open-source/fido2-lib/pull/83

Deno support: https://github.com/webauthn-open-source/fido2-lib/pull/84

[MasterKale]

You might be able to pick up some goodies in these PRs

@Hexagon You rock, this will save me so much time 🤩

[nightah]

@MasterKale given the title of this issue would you prefer another one to also track CJS to ESM for the other packages like @simplewebauthn/browser or are you happy to expand the scope of this issue to cover the full suite of packages?

[MasterKale]

@simplewebauthn/browser is already ESM with optional UMD builds too. I don't publish a CJS version of it because it never made sense to.

I'm assuming that when I undertake the effort of adding an ESM build to @simplewebauthn/server I'll end up doing the same with typescript-types and iso-webcrypto too because server won't work without those being ESM either.

[james-d-elliott]

So our issue (me and nightah) is related to #293. We can fix it locally by changing index.js (and package.json to reflect this) to index.mjs, with the jest ignore pattern of "[/\\\\]node_modules[/\\\\].+\\.(js|jsx|cjs)$".

I think it's prudent for us to discuss this here as it may affect the future of ESM builds for the server portion. But it's your repository and we'll obviously take your guidance on that.

I personally feel the pattern more closely reflects an evergreen build environment as it most closely reflects what files should be transpiled by jest in normal conditions. Having to manually add each module which shouldt be transpiled is a rather ugly solution as if you can only do this with a reverse negative lookahead and you have to have all of these in the same ignore pattern.

All that being said I'm NOT someone who should be implicitly trusted to be giving accurate advice as I find the whole cjs/esm and general node landscape to be incredibly wild and confusing. EDIT: In summary I feel like if index.js was instead transpiled to index.mjs (explicitly setting it as a ECMAScript Module rather than just implicitly) it may be a good result, but I'm not sure if this has other issues associated with it.

We used the following in trying to figure all of this out (short read, though it doesn't cover jest): https://bnb.im/series/esm-in-node-js/

[MasterKale]

The NodeJS docs are pretty clear about how Node attempts to determine which module system to use:

https://nodejs.org/docs/latest/api/packages.html#determining-module-system

Node.js will treat the following as ES modules when passed to node as the initial input, or when referenced by import statements or import() expressions:

1. Files with an .mjs extension.
2. Files with a .js extension when the nearest parent package.json file contains a top-level "type" field with a value of "module".
3. Strings passed in as an argument to --eval, or piped to node via STDIN, with the flag --input-type=module.

@simplewebauthn/browser should satisfy Scenario 2:

1. The non-UMD build outputs .js files
2. Its package.json includes "type": "module"

And this has been the case for a while, since #237 added "type": "module" back in August 2022 (and went out as v5.4.1)

Reviewing #293, which was reported two months later in October 2022, I'm not sure why Jest wasn't processing the files. I wonder, @james-d-elliott, if you'd be willing to help create a basic reproduction of a NextJS project that's unable to run tests because @simplewebauthn/browser is still not sufficiently identifying itself as an ESM module.

EDIT: Or if not NextJS then a basic reproduction of whatever setup you and @nightah are having trouble running Jest in seemingly because of browser.

[james-d-elliott]

Not that it matters a great deal but I agree that setting the type to module does satisfy that and @simplewebauthn/browser does that. I hope my response didn't come over abrasive in any way.

I suspect that this a jest issue the more I think about it and delve into it:

• https://github.com/facebook/jest/issues/9430#issuecomment-617656277
• https://github.com/facebook/jest/issues/9860
• https://jestjs.io/docs/ecmascript-modules

I have no trouble setting up an example.

[nightah]

Just to add to @james-d-elliott's comments and clear up my misunderstanding earlier, it seems that Jest by default expects CJS. You can turn on ESM support within Jest (experimentally) but that alone doesn't seem to fix the specific issue we're having (which again seems related to Jest).

I think our options here are:

1. Transform the ESM to CJS for @simplewebauthn/browser with a negative lookahead regex in transformIgnorePatterns.
2. Get ESM support within Jest working (though this option could require a lot more work in the broader codebase).

Also just for clarity the example that we're working with isn't a NextJS project, it's Vite project which could also be part of the problem. I think we need to explore getting a minimal reproduction with Vite/Vitest.

[MasterKale]

I apologize if I sounded annoyed, I'm not at all.

To get on my soapbox a bit, it's things like "CJS vs ESM" that are annoying to deal with, and I don't want to end up chasing down related rabbit holes trying to solve "an issue with SimpleWebAuthn" that is actually another library's (understandable) struggle to adapt to a bifurcated module system. Thankfully everything is moving towards ESM, and I haven't heard of anything (yet) coming along that might try to usurp ESM, so I feel confident in pursuing ESM support for server.

I don't want to add CJS support to browser when everything else front end is now ESM, and it'd only be to paper over some issue with an unrelated library that wants to blend CJS and ESM in otherwise ESM front end projects.

Thanks for explaining your situation more. Is "updating Jest to better support ESM" something you might actually try to help with? I wish I had that time anymore, it's all I can do sometimes to keep SimpleWebAuthn moving along 🫠

[MasterKale]

@rakeshpai I'm planning on starting this work on ESM support. Is there a CF worker config/setup you'd suggest I test with that'll ensure I'm actually using the library via ESM, so I don't repeat my earlier "success" that somehow got things working with @simplewebauthn/server's current CJS output?

[MasterKale]

Note to self: I guess I'll have to add .js to all imports, even in .ts files 🤔

https://github.com/TypeStrong/ts-node/issues/1833

And tsc isn't happy with my extension-less imports when I try to specify "module": "esnext" and "moduleResolution": "node16":

src/services/settingsService.ts:9:40 - error TS2835: Relative import paths need
explicit file extensions in EcmaScript imports when '--moduleResolution' is
'node16' or 'nodenext'. Did you mean './defaultRootCerts/apple.js'?

9 import { Apple_WebAuthn_Root_CA } from './defaultRootCerts/apple';
                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~

And I think I remember this being an issue when I tried using this library in Deno too. Probably from its ESM-first architecture. Lots of work ahead, it seems...

[Hexagon]

Not sure of you need it 😅

But here is a proven method of laying out package.json for esm+cjs. Don't forget to include package.json itself, needed by some build tools.

https://github.com/gildas-lormeau/zip.js/issues/396

[rakeshpai]

@rakeshpai I'm planning on starting this work on ESM support. Is there a CF worker config/setup you'd suggest I test with that'll ensure I'm actually using the library via ESM, so I don't repeat my earlier "success" that somehow got things working with @simplewebauthn/server's current CJS output?

Would have shared my project, but it has a lot of fluff with TS, Nx, Astro etc that isn't relevant here. That said, let me list out what I think are the relevant parts of my setup.

1. I use esbuild to minify the server code, so that it's easy to package up for CF workers. Let's call this command npm run build
2. In my wrangler.toml, I have the following config (relevant bits only)

   [build]
   command = "npm run build"
   [build.upload]
   format = "modules"

   This tells miniflare what my build command is, and that I want to use ESM.

3. I keep miniflare running in a terminal with miniflare --watch --debug ./path/to/esbuild/output
4. I then have an index.ts which has the following code:

   export default {
     fetch(request: Request): Promise<Response> {
       // use `request` to return a `new Response(...)`
     }
   };

   This uses the CF workers' module syntax to define the entry-point. In this entrypoint file, all you need to do is invoke generateRegistrationOptions to get the error. You don't even need a UI app and make a valid request, since the error happens even before generateRegistrationOptions is run - right in the bootstrapping phase of miniflare.

5. Finally, I have set my tsconfig's compilerOptions.types to include @cloudflare/workers-types if you're using TS, though I don't think this is too relevant for this situation.

Let me know if I can help with more clarifications.

[MasterKale]

Thank you for that rundown, that's enough for me to revisit my initial attempt and update it accordingly.

[build]
command = "npm run build"
[build.upload]
format = "modules"

I'll bet it's these toml file properties that I need to include in my wrangler.toml when I try this again to make sure it's trying to run everything as ESM. Mine was the bare minimum name, main, and compatibility_date that probably allowed it to flip to its CJS support when it encountered my library the first time.

[netgusto]

I'm able to use server's verifyRegistrationResponse() in a Cloudflare Worker (module format, ie default format) using Wrangler 2, and it's working fine without any fiddling (npm add -D @simplewebauthn/server and that's it)

[spendres]

I am unable to load simplewebauthn/sever in deno deploy. You can run snippets in deno deploy playground to recreate the problem. From the logs: gcp-us-east5TypeError: Cannot read properties of undefined (reading '0') at Function.a.path (https://esm.sh/v128/node-gyp-build-optional-packages@5.0.3/denonext/node-gyp-build-optional-packages.mjs:7:1746) at a (https://esm.sh/v128/node-gyp-build-optional-packages@5.0.3/denonext/node-gyp-build-optional-packages.mjs:7:1358) at https://esm.sh/v128/cbor-extract@2.1.1/denonext/cbor-extract.mjs:4:934 at https://esm.sh/v128/cbor-extract@2.1.1/denonext/cbor-extract.mjs:4:452 at https://esm.sh/v128/cbor-extract@2.1.1/denonext/cbor-extract.mjs:4:1028

The following import returns the same errors locally or on deno deploy: import SimpleWebAuthnServer from "https://esm.sh/@simplewebauthn/server";

The following import runs locally but returns errors on deno deploy: import SimpleWebAuthnServer from "npm:@simplewebauthn/server";

[Hexagon]

@spendres, What errors do you get when using npm specifiers? Esm.sh is black magic and works if it works 😄

[spendres]

No, I would rather use npm specificiers.

On Thu, Jul 6, 2023 at 2:24 PM Hexagon @.***> wrote:

@spendres https://github.com/spendres, I'm curious. Do you have specific reasons for using esm.sh instead of npm specifiers?

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1624130852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYBFQH3XDC642MR2DODXO37GHANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[spendres]

No, I would rather use npm specificiers. (sorry for any duplication from not using reply all initially)

On Thu, Jul 6, 2023 at 2:24 PM Hexagon @.***> wrote:

@spendres https://github.com/spendres, I'm curious. Do you have specific reasons for using esm.sh instead of npm specifiers?

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1624130852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYBFQH3XDC642MR2DODXO37GHANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[spendres]

When using import SimpleWebAuthnServer from "npm:@simplewebauthn/server@7"; // I've tried 7,7.2,7.2.0,7.3.1

Deno reports that module npm:@simplewebauthn/server is not found.

When using import SimpleWebAuthnServer from "https://esm.sh/@simplewebauthn/server@7.2.0";

Deno reports that an internal server error occured. (Cypto?)

[Hexagon]

Oooh, i was just curious, but yeah, seems like deno deploy doesn't support npm-specifiers at all (yet)...

[MasterKale]

Thanks for the trip report @spendres. I've got some time off coming up mid-August that I plan on dedicating some time to finally crack this ESM nut. I'll definitely be testing with Deno as my ideal place to get @simplewebauthn/server running.

[spendres]

Breadcrumbs from Deno: Import dist/main.js from a trusted source. Below is an example using the official deno.land repository. It is recommended to enable integrity checking.

import { Fido2Lib } from "https://deno.land/x/fido2@$VERSION/dist/main.js";

fido2-lib repo import deploys to Deno and works with no errors. This may serve as an example to update simplewebauthn lerna/nx scripts to export to deno.land/x.

[spendres]

I was able to deploy version 7.3.1 using docker and so check in with me when you start to work on this issue. The issue seems to be CBOR detecting which runtime it's running on to include the proper native cbor-extract library.

You are very close!

On Fri, Jul 7, 2023 at 12:19 PM Matthew Miller @.***> wrote:

Thanks for the trip report @spendres https://github.com/spendres. I've got some time off coming up mid-August that I plan on dedicating some time to finally crack this ESM nut. I'll definitely be testing with Deno as my ideal place to get @.**/server running.

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1625648830, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYGHP245KMLXJV2GDATXPAZJXANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[spendres]

MasterKale,

I have bumped cbor-x library to 1.5.3 in a local repo and the regression test pass. This version of cbor-x works on Deno(both simple test and used by Fido2-lib).

How can I test an npm deployment as a candidate v7.3.2 of @simplewebauthn/server?

[MasterKale]

How can I test an npm deployment as a candidate v7.3.2 of @simplewebauthn/server?

If you npm init -y to create a basic Node project, you can then npm install ../../packages/server/ the relative path to your local SimpleWebAuthn clone to use the local copy of the library any time you import { ... } from '@simplewebauthn/server. Any time you make a change to packages/server/ package you'll need to run npm run build before re-running whatever is importing the library.

If you're writing JavaScript, a simple node ./index.js then will suffice. If you're writing TypeScript then I prefer to call npx ts-node ./index.ts from within packages/server/ when I'm writing one-off scripts to test something.

[MasterKale]

I have bumped cbor-x library to 1.5.3 in a local repo and the regression test pass. This version of cbor-x works on Deno(both simple test and used by Fido2-lib).

I dug a bit into how cbor-x has evolved since November when I pulled it in as part of the v7.0.0 isomorphic rewrite. I wonder if this PR over there is why cbor-x v1.5.3 helped get things working in Deno:

https://github.com/kriszyp/cbor-x/pull/67

I'll look to upgrade cbor-x first if that's the case.

[spendres]

I wonder if this PR over there is why cbor-x v1.5.3 helped get things working in Deno:

https://github.com/kriszyp/cbor-x/pull/67

I think your hunch must be right, because Deno won't import cbor-x below version 1.5.3. If you type deno info "npm:@simplewebauthn/server@3.7.1 they download cbor-x@1.5.3 (instead of v1.4.2) along with the various c libraries for cbor-extract: ... ├─┬ npm:cbor-x@1.5.3 (1.41MB) │ └─┬ npm:cbor-extract@2.1.1 (11.72KB) │ ├── npm:@cbor-extract/cbor-extract-darwin-arm64@2.1.1 (103.1KB) │ ├── npm:@cbor-extract/cbor-extract-darwin-x64@2.1.1 (unknown) │ ├── npm:@cbor-extract/cbor-extract-linux-arm@2.1.1 (unknown) │ ├── npm:@cbor-extract/cbor-extract-linux-arm64@2.1.1 (unknown) │ ├── npm:@cbor-extract/cbor-extract-linux-x64@2.1.1 (unknown) │ ├── npm:@cbor-extract/cbor-extract-win32-x64@2.1.1 (unknown) │ └── npm:node-gyp-build-optional-packages@5.0.3 (12.95KB) ├─┬ npm:cross-fetch@3.1.8 (73.38KB) │ └─┬ npm:node-fetch@2.6.12 (158.34KB) │ └─┬ npm:whatwg-url@5.0.0 (48.72KB) │ ├── npm:tr46@0.0.3 (262.1KB) │ └── npm:webidl-conversions@3.0.1 (12.08KB) └─┬ npm:debug@4.3.4 (41.36KB) └── npm:ms@2.1.2 (6.68KB)

[MasterKale]

Leaving a note for myself to check out dnt as a potential refactor target:

https://deno.com/blog/publish-esm-cjs-module-dnt

[Hexagon]

@MasterKale

Leaving a note for myself to check out dnt as a potential refactor target:

https://deno.com/blog/publish-esm-cjs-module-dnt

Do it 😍

Sharing my experience from converting fido2-lib to esm (with native Deno-support). I took the route of finding libraries that work in both Node and Deno. I then used the node-modules as a base, mapping the Deno equivalents using their node package name through import_map.json. I ended up having one build/test pipeline for Node, and one for Deno.

See these two files for the "mapping":

https://github.com/webauthn-open-source/fido2-lib/blob/master/lib/toolbox.js https://github.com/webauthn-open-source/fido2-lib/blob/master/import_map.json

But if you go Deno-first and generate the Node module with dnt, it'll save you a vast amount of trouble and development dependencies. You won't need any other tools except Deno and dnt for the build/type/lint/formatting process. I guess a separate testing pipeline using native Node tools could be usable (?), though, but you could skip TypeScript, linting, etc...

As the codebase will be Deno-first doing this, I would have gone all in with the recommended mod.ts/deps.ts approach.

[spendres]

@Hexagon & MasterKale

I started this process to see how close fido2-lib's import_map would take it. I'm still extracting the external dependencies, but here are a few other libraries:

@./asn1-ecc": @*.**@*.", @./asn1-schema": @.**@.", @./asn1-android": @*.**@*.***",

On Mon, Jul 31, 2023 at 3:15 PM Hexagon @.***> wrote:

@MasterKale https://github.com/MasterKale

Leaving a note for myself to check out dnt as a potential refactor target:

https://deno.com/blog/publish-esm-cjs-module-dnt

Do it 😍

Sharing my experience from converting fido2-lib. I took the route of finding libraries that work in both Node and Deno. I then used the node-modules as a base, mapping the Deno equivalents using their node name through import_map.json. I ended up having one build/test pipeline for Node, and one for Deno.

See these two files for the "mapping":

https://github.com/webauthn-open-source/fido2-lib/blob/master/lib/toolbox.js

https://github.com/webauthn-open-source/fido2-lib/blob/master/import_map.json

But if you go Deno-first and generate the Node module with dnt, it'll save you a vast amount of trouble and development dependencies. You won't need any other tools except Deno and dnt for the build/type/lint/formatting process. I guess a separate testing pipeline for Node would be needed, though, but you could skip TypeScript, linting, etc...

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1658989210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYBVSVSM2LHIK3ETXFDXS7745ANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[spendres]

@Hexagon & MasterKale

Initial take at dynamically detecting the runtime across Browser, Node, Deno, Bun, Cloudflare – to determine which webcrypto to load, if any. // https://developer.mozilla.org/en-US/docs/Glossary/Global_object // assert global namespace //console.log(globalThis === globalThis.globalThis); // true (everywhere) try { if( typeof(global) != "undefined" ){ if(global === global.global){ console.log("Node,Bun?") } // true (in Node.js but not Deno. Bun?, CFW?)

} if( typeof(Deno) != "undefined") { console.log("Deno"); } } catch{ console.log('Not a server'); try{ console.log(window === window.window); // true (in a browser) console.log(frames === frames.frames); // true (in a browser) console.log('browser'); console.log(self === self.self); // true (in a browser or a Web Worker) console.log('browser or webworker'); } catch { console.log('Not a browser'); } }

On Mon, Jul 31, 2023 at 6:16 PM Steven Endres @.***> wrote:

@Hexagon & MasterKale

I started this process to see how close fido2-lib's import_map would take it. I'm still extracting the external dependencies, but here are a few other libraries:

@./asn1-ecc": @*.**@*.", @./asn1-schema": @.**@.", @./asn1-android": @*.**@*.***",

On Mon, Jul 31, 2023 at 3:15 PM Hexagon @.***> wrote:

@MasterKale https://github.com/MasterKale

Leaving a note for myself to check out dnt as a potential refactor target:

https://deno.com/blog/publish-esm-cjs-module-dnt

Do it 😍

Sharing my experience from converting fido2-lib. I took the route of finding libraries that work in both Node and Deno. I then used the node-modules as a base, mapping the Deno equivalents using their node name through import_map.json. I ended up having one build/test pipeline for Node, and one for Deno.

See these two files for the "mapping":

https://github.com/webauthn-open-source/fido2-lib/blob/master/lib/toolbox.js

https://github.com/webauthn-open-source/fido2-lib/blob/master/import_map.json

But if you go Deno-first and generate the Node module with dnt, it'll save you a vast amount of trouble and development dependencies. You won't need any other tools except Deno and dnt for the build/type/lint/formatting process. I guess a separate testing pipeline for Node would be needed, though, but you could skip TypeScript, linting, etc...

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1658989210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYBVSVSM2LHIK3ETXFDXS7745ANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[spendres]

@Hexagon worked this out already...

// Import webcrypto import as platformCrypto from "crypto"; import as peculiarCrypto from @.***/webcrypto"; let webcrypto; if ((typeof self !== "undefined") && "crypto" in self) { // Always use crypto if available natively (browser / Deno) webcrypto = self.crypto;

} else { // Always use node webcrypto if available ( >= 16.0 ) if(platformCrypto && platformCrypto.webcrypto) { webcrypto = platformCrypto.webcrypto;

    } else {
            // Fallback to @peculiar/webcrypto
            webcrypto = new peculiarCrypto.Crypto();
    }

}

On Mon, Jul 31, 2023 at 6:33 PM Steven Endres @.***> wrote:

@Hexagon & MasterKale

Initial take at dynamically detecting the runtime across Browser, Node, Deno, Bun, Cloudflare – to determine which webcrypto to load, if any. // https://developer.mozilla.org/en-US/docs/Glossary/Global_object // assert global namespace //console.log(globalThis === globalThis.globalThis); // true (everywhere) try { if( typeof(global) != "undefined" ){ if(global === global.global){ console.log("Node,Bun?") } // true (in Node.js but not Deno. Bun?, CFW?)

} if( typeof(Deno) != "undefined") { console.log("Deno"); } } catch{ console.log('Not a server'); try{ console.log(window === window.window); // true (in a browser) console.log(frames === frames.frames); // true (in a browser) console.log('browser'); console.log(self === self.self); // true (in a browser or a Web Worker) console.log('browser or webworker'); } catch { console.log('Not a browser'); } }

On Mon, Jul 31, 2023 at 6:16 PM Steven Endres @.***> wrote:

@Hexagon & MasterKale

I started this process to see how close fido2-lib's import_map would take it. I'm still extracting the external dependencies, but here are a few other libraries:

@./asn1-ecc": @*.**@*.", @./asn1-schema": @.**@.", @./asn1-android": @*.**@*.***",

On Mon, Jul 31, 2023 at 3:15 PM Hexagon @.***> wrote:

@MasterKale https://github.com/MasterKale

Leaving a note for myself to check out dnt as a potential refactor target:

https://deno.com/blog/publish-esm-cjs-module-dnt

Do it 😍

Sharing my experience from converting fido2-lib. I took the route of finding libraries that work in both Node and Deno. I then used the node-modules as a base, mapping the Deno equivalents using their node name through import_map.json. I ended up having one build/test pipeline for Node, and one for Deno.

See these two files for the "mapping":

https://github.com/webauthn-open-source/fido2-lib/blob/master/lib/toolbox.js

https://github.com/webauthn-open-source/fido2-lib/blob/master/import_map.json

But if you go Deno-first and generate the Node module with dnt, it'll save you a vast amount of trouble and development dependencies. You won't need any other tools except Deno and dnt for the build/type/lint/formatting process. I guess a separate testing pipeline for Node would be needed, though, but you could skip TypeScript, linting, etc...

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1658989210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYBVSVSM2LHIK3ETXFDXS7745ANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[spendres]

@MasterKale,

RE: Refactoring for ESM/Deno

1. I have refactored the repo to support ESM. a. edge-db created a nice tool called 'compile-for-deno' to help in migrating a node project to deno. See https://www.edgedb.com/blog/how-we-converted-our-node-js-library-to-deno-using-deno#dependencies b. I've 'denoified'/esm'd simplewebauthn with the 'compile-for-deno' tool and can share when you are ready. c. Deno favors importing external dependencies from one root file, like 'toolbox' on fido2-lib shared by @Hexagon. I created deps.ts/mod.ts files at each layer to resolve module imports – like index files. I used the import {x} from 'toolbox' to point to src/deps.ts that holds the external dependencies.
2. Deno deploy now works, fed from a private github repo. I can register and authenticate without any hurdles. Git push auto deploys to deno.dash.
3. I had DNT working but broke it somewhere in the process of getting the repo to deploy to deno.dash.
4. I use the "imports" option in the deno.jsonc config file at the project root directory to anchor urls. The fido2-lib project uses an import_map, but the compile-for-deno tool has its own import_map plus it hoists the dependencies to the common toolbox.
5. There should be a way to pass the resolved imports from compile-for-deno to DNT... but I'm out of time for DNT.
6. Your example works in deno using express, but Deno has its own dotenv that doesn't play well with npm:dotenv.
7. My repo started with your most recent v7.4.0.
8. Still cannot access the modules using raw.githubusercontent.com, ... starts fine but stops at typescript-types mod.ts.

@spendres

On Mon, Jul 31, 2023 at 6:44 PM Steven Endres @.***> wrote:

@Hexagon worked this out already...

// Import webcrypto import as platformCrypto from "crypto"; import as peculiarCrypto from @.***/webcrypto"; let webcrypto; if ((typeof self !== "undefined") && "crypto" in self) { // Always use crypto if available natively (browser / Deno) webcrypto = self.crypto;

} else { // Always use node webcrypto if available ( >= 16.0 ) if(platformCrypto && platformCrypto.webcrypto) { webcrypto = platformCrypto.webcrypto;

    } else {
            // Fallback to @peculiar/webcrypto
            webcrypto = new peculiarCrypto.Crypto();
    }

}

On Mon, Jul 31, 2023 at 6:33 PM Steven Endres @.***> wrote:

@Hexagon & MasterKale

Initial take at dynamically detecting the runtime across Browser, Node, Deno, Bun, Cloudflare – to determine which webcrypto to load, if any. // https://developer.mozilla.org/en-US/docs/Glossary/Global_object // assert global namespace //console.log(globalThis === globalThis.globalThis); // true (everywhere) try { if( typeof(global) != "undefined" ){ if(global === global.global){ console.log("Node,Bun?") } // true (in Node.js but not Deno. Bun?, CFW?)

} if( typeof(Deno) != "undefined") { console.log("Deno"); } } catch{ console.log('Not a server'); try{ console.log(window === window.window); // true (in a browser) console.log(frames === frames.frames); // true (in a browser) console.log('browser'); console.log(self === self.self); // true (in a browser or a Web Worker) console.log('browser or webworker'); } catch { console.log('Not a browser'); } }

On Mon, Jul 31, 2023 at 6:16 PM Steven Endres @.***> wrote:

@Hexagon & MasterKale

I started this process to see how close fido2-lib's import_map would take it. I'm still extracting the external dependencies, but here are a few other libraries:

@./asn1-ecc": @*.**@*.", @./asn1-schema": @.**@.", @./asn1-android": @*.**@*.***",

On Mon, Jul 31, 2023 at 3:15 PM Hexagon @.***> wrote:

@MasterKale https://github.com/MasterKale

Leaving a note for myself to check out dnt as a potential refactor target:

https://deno.com/blog/publish-esm-cjs-module-dnt

Do it 😍

Sharing my experience from converting fido2-lib. I took the route of finding libraries that work in both Node and Deno. I then used the node-modules as a base, mapping the Deno equivalents using their node name through import_map.json. I ended up having one build/test pipeline for Node, and one for Deno.

See these two files for the "mapping":

https://github.com/webauthn-open-source/fido2-lib/blob/master/lib/toolbox.js

https://github.com/webauthn-open-source/fido2-lib/blob/master/import_map.json

But if you go Deno-first and generate the Node module with dnt, it'll save you a vast amount of trouble and development dependencies. You won't need any other tools except Deno and dnt for the build/type/lint/formatting process. I guess a separate testing pipeline for Node would be needed, though, but you could skip TypeScript, linting, etc...

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1658989210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYBVSVSM2LHIK3ETXFDXS7745ANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[MasterKale]

I've got #425 up that refactors everything but browser to use dnt. server's tests have all been refactored and are finally passing, but browser is having issues resolving @simplewebauthn/typescript-types because I refactored it too to use dnt, but that broke something perhaps in how Lerna helps packages fine local packages by their NPM name.

So that'll be the next thing for me to tackle, before I try to figure out a new monorepo publication process. I think the way Lerna updates packages' package.json files as part of lerna publish my dnt build scripts won't be able to know the updated version to put into their generated package.json because the build happens before the package.json gets updated. Maybe it's time to move on from Lerna...

Anyway I'd appreciate some eyes on #425 if any of you dnt fans have a spare moment. I'm generally pleased with the direction things are going, and am happy I finally had a reason to learn some Deno, but I'm still nervous about if this'll truly achieve the goal of making SimpleWebAuthn available.

Signs of success, though, include my being able to call generateAuthenticationOptions() in a Node-specific file and a Deno-specific file from the Example project after an npm install of the local new server build, as well as being able to do the same in a local CloudFlare worker via Wrangler w/node_compat = false 👀

test_node.js

const { generateAuthenticationOptions } = require('../packages/server/npm');

generateAuthenticationOptions().then(console.log);

test_deno.ts

import { generateAuthenticationOptions } from '../packages/server/src/index.ts';

console.log(await generateAuthenticationOptions());

[spendres]

This version does not deploy on deno since they don't support npm:specifiers. See next email. Your example does work on local deno by:

1. awaiting the now async versions of generateAuthenticationOptions and generateRegistrationOptions
2. changing the https,http,and fs imports and adding an import for the global node process... to look like: import https from "node:https"; import http from "node:http"; import fs from "node:fs"; import process from "node:process";
3. importing your SimpleWebAuthn library from github url as: import { // Authentication generateAuthenticationOptions, // Registration generateRegistrationOptions, verifyAuthenticationResponse, verifyRegistrationResponse, } from " https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/index.ts ";
4. Imported the helpers from github url: import { isoBase64URL, isoUint8Array, } from " https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/helpers/ index.ts";
5. I used the same github url used in 3. for the two type import statements: import type { GenerateAuthenticationOptionsOpts, GenerateRegistrationOptionsOpts, VerifiedAuthenticationResponse, VerifiedRegistrationResponse, VerifyAuthenticationResponseOpts, VerifyRegistrationResponseOpts, } from " https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/index.ts "; //from @.***/server";

import type { AuthenticationResponseJSON, AuthenticatorDevice, RegistrationResponseJSON, } from " https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/index.ts "; //from @.***/typescript-types";

6. Added the .ts suffix to example-server.d import: import { LoggedInUser } from "./example-server.d.ts";

[image: image.png]

On Thu, Aug 17, 2023 at 7:02 PM Matthew Miller @.***> wrote:

I've got #425 https://github.com/MasterKale/SimpleWebAuthn/pull/425 up that refactors everything but browser to use dnt. server's tests have all been refactored and are finally passing, but browser is having issues resolving @simplewebauthn/typescript-types because I refactored it too to use dnt, but that broke something perhaps in how Lerna helps packages fine local packages by their NPM name.

So that'll be the next thing for me to tackle, before I try to figure out a new monorepo publication process. I think the way Lerna updates packages' package.json files as part of lerna publish my dnt build scripts won't be able to know the updated version to put into their generated package.json because the build happens before the package.json gets updated. Maybe it's time to move on from Lerna...

Anyway I'd appreciate some eyes on #425 https://github.com/MasterKale/SimpleWebAuthn/pull/425 if any of you dnt fans have a spare moment. I'm generally pleased with the direction things are going, and am happy I finally had a reason to learn some Deno, but I'm still nervous about if this'll truly achieve the goal of making SimpleWebAuthn available.

Signs of success, though, include my being able to call generateAuthenticationOptions() in a Node-specific file and a Deno-specific file from the Example project after an npm install of the local new server build, as well as being able to do the same in a local CloudFlare worker via Wrangler w/node_compat = false 👀

test_node.js

const { generateAuthenticationOptions } = require('../packages/server/npm'); generateAuthenticationOptions().then(console.log);

test_deno.ts

import { generateAuthenticationOptions } from '../packages/server/src/index.ts'; console.log(await generateAuthenticationOptions());

[image: Screenshot 2023-08-17 at 4 00 26 PM] https://user-images.githubusercontent.com/5166470/261458485-fcc962ae-06ee-45f9-8f98-41ba1df06f4e.png

[image: Screenshot 2023-08-17 at 4 01 20 PM] https://user-images.githubusercontent.com/5166470/261458493-d2afc4d8-2416-4e5d-851b-a3f4aa3c5411.png

— Reply to this email directly, view it on GitHub https://github.com/MasterKale/SimpleWebAuthn/issues/338#issuecomment-1683087413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEVLYHBKNMVF2VKHW536STXV2PJHANCNFSM6AAAAAAUCYTSBE . You are receiving this because you were mentioned.Message ID: @.***>

[spendres]

Deno deploy does not yet support npm:specifiers. I used the substitutions below to work around this limitation. But there should be a way to use an import_map.json with a github action to substitute out the following npm:specifiers with their wrapped esm module url –– as part of the dnt migration.

{ /cbor-x/ , @.***/index.js?module' },

{ @.\/base64/ @./dist/base64.min.mjs"},

{ /debug/ , @.***/denonext/debug.mjs' },

{ @.\/asn1-rsa/ , ' @*.**@*.***/denonext/asn1-rsa.mjs' },

{ @.\/asn1-ecc/ , ' @*.**@*.***/denonext/asn1-ecc.mjs' },

{ @.\/asn1-schema/ , ' @*.**@*.***/denonext/asn1-schema.mjs' },

{ @.\/asn1-android/ , ' @*.**@*.***/denonext/asn1-android.mjs' },

{ @.\/asn1-x509/ , ' @*.**@*.***/denonext/asn1-x509.mjs' },

[MasterKale]

@spendres I don't know why this is but so many of your comments here have had some kind of aggressive email filtering applied to them so those package names are all but illegible. Is there any chance you can revisit your message as it appears on GitHub.com and help provide some clarity on your suggestions?

[spendres]

That is odd. It looks like markup/markdown conversion. Here is what I sent using markup in github.

Deno deploy does not yet support npm:specifiers. I used the substitutions below to work around this limitation. But there should be a way to use an import_map.json with a github action to substitute out the following npm:specifiers with their wrapped esm module url –– as part of the dnt migration. This deployment step would be only used for deno.land.

{ "npm:cbor-x" , 'https://deno.land/x/cbor@v1.5.2/index.js?module' },
{ "npm:@hexagon/base64" ,'https://deno.land/x/b64@1.1.26/dist/base64.min.mjs'},
{ "npm:debug" , 'https://esm.sh/v130/debug@4.3.4/denonext/debug.mjs' }, 
{ "npm:@peculiar/asn1-rsa" , 'https://esm.sh/v128/@peculiar/asn1-rsa@2.3.6/denonext/asn1-rsa.mjs' },
{ "npm:@peculiar/asn1-ecc" , 'https://esm.sh/v130/@peculiar/asn1-ecc@2.3.6/denonext/asn1-ecc.mjs' },
{ "npm:@peculiar/asn1-schema" , 'https://esm.sh/v130/@peculiar/asn1-schema@2.3.6/denonext/asn1-schema.mjs' },
{ "npm:@peculiar/asn1-android" , 'https://esm.sh/v128/@peculiar/asn1-android@2.3.6/denonext/asn1-android.mjs' },
{ "npm:@peculiar/asn1-x509" , 'https://esm.sh/v130/@peculiar/asn1-x509@2.3.6/denonext/asn1-x509.mjs' },

[MasterKale]

Oh sweet, thanks for mapping those remaining NPM imports. I knew there are many websites like https://esm.sh, but wasn't sure which one to pick.

One thing I don't know the answer to is how Deno helps projects incorporate security updates. In deps.ts the URLs point to exact versions of packages, while in Node projects NPM specifiers like "^1.2.6" in package.json let projects that consume libraries like SimpleWebAuthn pull in package updates without the library needing to be republished. How do minor sub-dependency updates work in Deno for Deno applications?

[Hexagon]

Deno libraries are always frozen to a specific dependency version, with additional integrity check through the lockfile. But so do most node-packages too, normally freezing third party dependency versions through package-lock.json. The "weak" specifiers are mostly used to allow packages to be auto updated by the package maintainer, i guess (?).

Ignoring that, dnt allows for shimming, where you map a specific dependency name to the node equivalent with an additional version specifier which is transfered to the finished package.json, but I think that require one to use import-maps.

[MasterKale]

Ignoring that, dnt allows for shimming, where you map a specific dependency name to the node equivalent with an additional version specifier which is transfered to the finished package.json, but I think that require one to use import-maps.

I'm using mappings in my dnt config but specify the "^1.2.6" style of package version in the generated package.json. This project has had people ask for me to unpin when I tried pinning dependencies in the past which is why I'm keeping it for Node projects. I just wanted to make sure that Deno projects had some way of updating their project's dependencies without me having to cut a new release every time on of SimpleWebAuthn's dependencies get updated.

[spendres]

I just wanted to make sure that Deno projects had some way of updating their project's dependencies without me having to cut a new release every time on of SimpleWebAuthn's dependencies get updated.

see: (https://deno.land/manual@v1.36.1/basics/import_maps#overriding-imports)

My understanding is that deno consumers of SimpleWebAuthn would be able to override a version by placing an import_map.json(or deno.json) file at the root of 'their' project, that would override the versions that you specify in package.json – locked down from your automated testing and in the "publish to npm and deno.land" portions of your release.

As an example, let's say that you use ^1.0.0 for dependency A, then publish the package to npm and deno.land. When an application imports SimpleWebAuthn, they would use https://deno.land/x/@SimpleWebAuthn/server@7.4.1/index.ts which would use ^1.0.0 for dependency A. If a patch to dependency A comes out, and downstream users want to update to ^1.0.1 of A, then they would create a version of import_maps.json in their path where they invoke deno(or pass in as command line arg), that would have an entry for A, like:

// import_maps.json { "A", "https://esm.sh/A/X@1.0.1/index.ts" }

Deno would detect the local import_map.json(or deno.json) file and look for 'import {X} from A' while loading and substitute A with the URL for A from the import mapping above.

[spendres]

The earlier post:

This version does not deploy on deno since they don't support npm:specifiers. See next email. Your example does work on local deno by:

1. awaiting the now async versions of generateAuthenticationOptions and generateRegistrationOptions
2. changing the https,http,and fs imports and adding an import for the global node process... to look like: import https from "node:https"; import http from "node:http"; import fs from "node:fs"; import process from "node:process";
3. importing your SimpleWebAuthn library from github url as: import { // Authentication generateAuthenticationOptions, // Registration generateRegistrationOptions, verifyAuthenticationResponse, verifyRegistrationResponse, } from "https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/index.ts";
4. Imported the helpers from github url: import { isoBase64URL, isoUint8Array, } from "https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/helpers/index.ts";
5. I used the same github url used in 3. for the two type import statements: import type { GenerateAuthenticationOptionsOpts, GenerateRegistrationOptionsOpts, VerifiedAuthenticationResponse, VerifiedRegistrationResponse, VerifyAuthenticationResponseOpts, VerifyRegistrationResponseOpts, } from "https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/index.ts"; //from "@simplewebauthn/server";

import type { AuthenticationResponseJSON, AuthenticatorDevice, RegistrationResponseJSON, } from "https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/index.ts"; //from "@simplewebauthn/typescript-types";

6. Added the .ts suffix to example-server.d import: import { LoggedInUser } from "./example-server.d.ts";

[spendres]

Μy understanding was correct. I used the deno.lock file to find the npm:specifiers. I kept running:

deno info server.ts | grep npm

until there were no more npm:imports.

The following import works on deno deploy: import * as SimpleWebAuthnServer from "https://raw.githubusercontent.com/MasterKale/SimpleWebAuthn/feat/server-esm-take-2-dnt/packages/server/src/index.ts";

I am not sure why, but there are three versions of npm:debug and two versions of npm:types/debug in the release.

Here is the deno.jsonc I use in the same directory as the server.ts that loads the SimpleWebAuthnServer import above:

{
  "$schema": "https://deno.land/x/deno/cli/schemas/config-file.v1.json",
  "tasks": {
    "start": "deno run --allow-all --unstable server.ts",
    "start:watch": "deno run --allow-all --unstable --watch server.ts",
    "start-redis:watch": "deno run --allow-all --unstable --watch server-redis.ts"
  },
  "imports": {
    "npm:@hexagon/base64@^1.1.26": "https://deno.land/x/b64@1.1.26/dist/base64.min.mjs",
    "npm:@peculiar/asn1-android@^2.3.3": "https://esm.sh/v128/@peculiar/asn1-android@2.3.6/denonext/asn1-android.mjs",
    "npm:@peculiar/asn1-ecc@^2.3.4": "https://esm.sh/v130/@peculiar/asn1-ecc@2.3.6/denonext/asn1-ecc.mjs",
    "npm:@peculiar/asn1-rsa@^2.3.4": "https://esm.sh/v130/@peculiar/asn1-rsa@2.3.6/denonext/asn1-rsa.mjs",
    "npm:@peculiar/asn1-schema@^2.3.3": "https://esm.sh/v130/@peculiar/asn1-schema@2.3.6/denonext/asn1-schema.mjs",
    "npm:@peculiar/asn1-x509@^2.3.4": "https://esm.sh/v130/@peculiar/asn1-x509@2.3.6/denonext/asn1-x509.mjs",
    "npm:cbor-x@v1.5.2": "https://deno.land/x/cbor@v1.5.2/index.js?module",
    "npm:cross-fetch@^3.1.5": "./not-used-by-deno.ts",
    "npm:@types/debug@4.1.8": "https://esm.sh/v131/@types/debug@4.1.8/index.d.ts",
    "npm:@types/debug@^4.1.7": "https://esm.sh/v131/@types/debug@4.1.8/index.d.ts",
    "npm:debug@4.3.4": "https://esm.sh/v130/debug@4.3.4/denonext/debug.mjs",
    "npm:debug@^4.3.2": "https://esm.sh/v130/debug@4.3.4/denonext/debug.mjs",
    "npm:debug": "https://esm.sh/v130/debug@4.3.4/denonext/debug.mjs"
  }
}

[MasterKale]

This morning I ended up manually finding out the long-form esm.sh URL for each of the packages still using npm:, and now everything is properly mapped to Deno-compatible URL, with corresponding entries in dnt's mappings property to their NPM package names 🎉

As a matter of fact CI just passed for the first time after refactoring almost the entire monorepo https://github.com/MasterKale/SimpleWebAuthn/actions/runs/5907029108 👀

[MasterKale]

Note to self: I'm so far off the Lerna reservation by moving two of three packages to using dnt for builds. Lerna's "algorithm" or detecting build order doesn't know anymore to build typescript-types before browser or server, and it can't influence the version number specified for @simplewebauthn/typescript-types in the package.json that gets generated by dnt's build.

This PR will also migrate the project to using npm's workspaces, which means Lerna's primary value now is its lerna publish flow. I might have to drop Lerna completely, for something hand-written I guess, if I can't figure out how to get its publish command to update things like it could pre-use-of-dnt.

The biggest issue right now is that the dnt build does an npm install, and if I try to use the latest types package's version (that isn't yet published to NPM) then the install will fail.

For example, here's what happens during a lerna run build after I set the version to 8.0.0-alpha.0 in a previous lerna publish that errored out when it tried building everything to publish:

$> npx lerna run build
lerna notice cli v7.1.5

    ✔  @simplewebauthn/browser:build  [existing outputs match the cache, left as is]
    ✔  @simplewebauthn/typescript-types:build (10s)

    ✖  @simplewebauthn/server:build
       > @simplewebauthn/server@8.0.0-alpha.0 build
       > deno task build

       Task build deno run -A build_npm.ts
       Building for testing...
       [dnt] Transforming...
       [dnt] Running npm install...
       npm ERR! code ETARGET
       npm ERR! notarget No matching version found for @simplewebauthn/typescript-types@^8.0.0-alpha.0.
       npm ERR! notarget In most cases you or one of your dependencies are requesting
       npm ERR! notarget a package version that doesn't exist.

       npm ERR! A complete log of this run can be found in: /Users/matt/.npm/_logs/2023-08-18T21_56_36_714Z-debug-0.log
       error: Uncaught (in promise) Error: npm install failed with exit code 1
             throw new Error(
                   ^
           at runCommand (https://deno.land/x/dnt@0.38.0/lib/utils.ts:56:13)
           at eventLoopTick (ext:core/01_core.js:183:11)
           at async build (https://deno.land/x/dnt@0.38.0/mod.ts:242:5)
           at async file:///Users/matt/Developer/simplewebauthn/packages/server/build_npm.ts:30:1

[spendres]

Note to Matt: Consider claiming https://deno.land/x/@simplewebauthn ... and setting up the webhook: https://deno.com/add_module

[MasterKale]

Is that something I can do before publishing anything so I know the name is available?