Investigate switch from node to deno

[dae]

Some issues we're likely to hit that will need investigating before it becomes practical:

• Ensuring we can still edit Svelte files in VS Code with proper code completion/checking
• Ensuring we can still build Svelte files and use svelte-check in CI.
• Figuring out if it's possible to port our existing Jest tests over to Deno's test runner. Some of our tests rely on jsdom.
• Ensuring protobuf generation still works. We could potentially fall back on #2043 if protobufjs proves problematic, and I have a half-complete port lying around somewhere.
• We copy files from node_modules as part of the build; that will no longer work with deno AFAIK, and we'll need to manually pull the files from npm packages. I have a prototype implementation that does this.

Related: #2430

[andrewpeters9]

Happy to take this on if you'd like me to as I have decent availability - feel free to assign it to me if you're comfortable with me taking this on. Will also be able to help with the actual conversion.

Just some other trivial things I think could be added to the checklist:

• [x] Ensure all Node packages used have an equivalent Deno Package (this was previously an issue with packages such as mongoose etc.)
  • [x] For those that don't, evaluate workarounds
• [x] Ensure that deno packages can be patched
  • [x] Appears that deno vendor can help with this - https://deno.land/manual@v1.31.2/tools/vendor
• [ ] Ensure WebStorm IDE support for those using Intellij editors
• [ ] Quickly hack up small pieces of the conversion to uncover unknown-unknowns that may pop up
• [ ] Ensure @dae 's personal VS Code setup works with the changes, as VS Code configurations can sometimes be fiddly person-to-person

[dae]

If you'd like to look into this, that would be much appreciated. A couple of points:

• the VS code configuration is in .vscode.dist, and I think the other devs are using it too
• I found Svelte support in IDEA to not be very good in the past, but perhaps things have improved or I had set things up wrong.
• I'd prefer not to have protobufjs files directly in the repo, but we should be able to put them in a submodule and then use vendor with that.

[andrewpeters9]

Took a bit of look today @dae .

• svelte IDE integration seemed to work in VS Code with the Deno plugin installed and "deno.enable": true, added to .vscode/settings.json (as well as in WebStorm) - but this may be a different story when node is completely ripped out
• esbuild seems to generally work OK with deno granted some small changes
  • This is not the case for some plugins however, e.g. esbuild-sass-plugin - this can be fixed by using this however: https://github.com/LePichu/esbuild-plugin-sass-deno
• deno respects package.json which alleviates some of the pain in conversion, it doesn't however respect yarn.lock
  • E.g. deno will go ahead and use esbuild@0.15.18 as it's the latest version that complies with package.json, but this doesn't comply with yarn.lock which requires ^0.15.13 to resolve to 0.15.13
  • I have a couple of ideas in mind for a fix, not a show stopper - but may involve either strict versions or resolutions built into the package.json
• Have yet to investigate protobuf and Jest test migration. Will look into that when I have the time to.
• I'm also coming across some things such as yarn.sh that will have be written etc. Deno integration might be something that coexists with node for a while and then yarn and node could be slowly phased out - up to you though.

[dae]

Thanks Andrew.

Currently package.json is rather lax compared to the lock file; after 2.1.61 is released (hopefully not long) I'll do a dependency bump and update package.json at the same time so they're closer in sync.

I don't mind a temporary period where we have both node and deno being used, as long as there's a clear path to full migration. What would not be ideal is if the deno migration stalls, and we end up stuck requiring both of them.

[andrewpeters9]

@dae Gotcha. I'll dig deeper into these testing Jest and protobuf issues over the weekend - although I may need a hand with ascerning if protobuf will still work, as I'm new to Rust + protobuf.

I'll also start to put together a list of tasks required, and how they could be potentially batched up.

[dae]

A smoke test for the protobuf code is opening the graphs screen - the graphs will fail to render if protobufjs is not working. If you run into an issue with it and need help, please let me know.

[andrewpeters9]

A smoke test for the protobuf code is opening the graphs screen - the graphs will fail to render if protobufjs is not working. If you run into an issue with it and need help, please let me know.

Thanks for the tip. I'm still working on this by the way, I've just been smashed with other work recently. Will be progressing this weekend.

[dae]

No rush - this is not a blocking task. Appreciate your contributions.

[dae]

Still no rush on this, but just wanted to let you know I've gotten around to porting to protobuf-es, which should be more deno friendly, and reduces the number of dependencies we need. This also removed the need for patch-package.

https://github.com/ankitects/anki/pull/2547

[andrewpeters9]

Hey @dae , I'm still wanting to do this, have just been smashed at work recently. Jumped back on this over the weekend, here's what I've found/did:

Some recent developments to Deno:

• Now supports node_modules
  • Makes transition for some Rust steps easier
• Now dependencies from npm can be directly pulled in by npm:
  • This isn't to say that all npm deps will work out of the box, however.
• Support for lock files
  • But, it doesn't support yarn.lock files. Potentially there's some script that will convert yarn.lock to package-lock.json
  • This is also dependent on how precise you are with your lock file. E.g. do you just rebuild lock files frequently, or are you always trying to keep the lock file as lean as possible
  • We could potentially just recreate the lock file with Deno (Deno has a native lock file called deno-lock.json and we could optimise from there?

• Now supports local references in package.json

  General work:

• I got a compile working for the easiest ESBuild scripts, the task can be run via this task
  • I haven't tested the validity of this build, but I'll do so when I get around to running a full build
• Jest appears to be an issue, but given that there aren't a tonne of Jest tests, maybe they could just be rewritten in Deno's native testing library (fairly similar syntax to Jest)
  • This might also be a good MR to first introduce Deno in via? I.e., just strip out Jest and replace with Deno testing, but leave everything as is

• svelte intellisense appears to be an issue (at least for me on WebStorm, I'll try VSCode soon)

  • Problem lies with non-default exports from <script context="module"></script>
  • EDIT: intellisense for svelte appears to be all good on VSCode ✅, some of the @tslib/* intellisense is broken on VSCode, but I believe it wouldn't be too hard to fix

  Merge Request:

• Here's a little PR that I opened up for testing, feel free to play around with it: https://github.com/andrewpeters9/anki/pull/1

[dae]

Thanks for all your work on this Andrew.

Jest pulls in quite a lot of transitive dependencies IIRC, so if it's practical to replace with Deno's built-in runner, that would be a nice win. Some of our Jest tests use it with the jsdom environment, so we can run tests against browser APIs (eg ts/html-filter relies on document.getElement() & .nodeType), so we'd need some way to still support those APIs (maybe deno-dom + some reworking of the tests?).

Re lock files, in larger Anki updates, I try to update all deps to their latest. In smaller updates, I want the versions to be locked, and the ability to update single deps (eg pull in a security update to a package without updating everything else). I presume such a workflow can be done using the traditional Deno approach of listing out versions in a dependencies.ts file, which the rest of the codebase imports? A move to deno would come as part of a larger update, so I'm not too concerned about matching the existing yarn lockfile. I briefly Googled the deno lock documentation, but it seemed pretty barebones - I'm guessing it doesn't currently provide something akin to 'yarn upgrade' or 'yarn upgrade some_package'?

I gave the math-jax task a try using your PR, and was initially surprised at how slow deno was to pull in all the packages, but then I realised that I'm just used to yarn runs where the packages have already been cached.

Re @tslib imports in VS Code, while we could reference the non-generated files via relative paths instead, all our protobuf code and translations are also referenced that way, so we'd definitely need to fix this before a switch is practical.

I'm open to starting with deno as just a Jest replacement, once we've at least prototyped using it for our other components and/or can easily roll it back. What I'd like to avoid is a situation where we've partially switched to deno and then discovered showstopper issues that require us to keep using yarn, and we're unable to roll back deno either, leaving us with a build system that is twice as slow :-)

[andrewpeters9]

Jest pulls in quite a lot of transitive dependencies IIRC, so if it's practical to replace with Deno's built-in runner, that would be a nice win.

Sounds like a good MR to start with. I'll start work on this over the weekend.

I presume such a workflow can be done using the traditional Deno approach of listing out versions in a dependencies.ts file, which the rest of the codebase imports?

Correct, but we can also just keep the package.json to manage versions, now that deno has support for this: https://deno.com/manual@v1.34.3/node/package_json

NOTE: deno does still recommend using a deno.json file for import mapping, so maybe migration from package.json → deno.json becomes a house-keeping task once the conversion is fully complete. I'll make a GH issue for this so that I don't forget.

I briefly Googled the deno lock documentation, but it seemed pretty barebones - I'm guessing it doesn't currently provide something akin to 'yarn upgrade' or 'yarn upgrade some_package'?

For upgrading all packages in one command, I'm unsure if this exists, but individual packages can be updated while still being careful not to disrupt the lockfile. So I don't think there's a problem here.

I gave the math-jax task a try using your PR, and was initially surprised at how slow deno was to pull in all the packages, but then I realised that I'm just used to yarn runs where the packages have already been cached.

Necessary packages are cached upon the first call to deno, so subsequent builds should be a lot quicker. Feel free to re-run that command to check the speed - as I don't want your workflow to be impacted by a command that I imagine you're running plenty of times a day.

Re @tslib imports in VS Code, while we could reference the non-generated files via relative paths instead, all our protobuf code and translations are also referenced that way, so we'd definitely need to fix this before a switch is practical.

Noted.

I'm open to starting with deno as just a Jest replacement, once we've at least prototyped using it for our other components and/or can easily roll it back.

Sounds good to me!

[dae]

For upgrading all packages in one command, I'm unsure if this exists, but individual packages can be updated while still being careful not to disrupt the lockfile. So I don't think there's a problem here.

Having to manually update all packages would be rather cumbersome, and I'm a bit surprised they don't provide a first-party solution for this, but it looks like https://github.com/hayd/deno-udd will accomplish it.

so subsequent builds should be a lot quicker.

Yep, they are :-)

[dae]

Closing, as it was determined that it's not practical for now.