Runtime features to support Quokka.js

[smcenlly]

Hi Deno team – firstly, I wanted to say thanks for all your work creating deno. It’s impressive and I am personally looking forward to seeing more community adoption.

Our company creates a couple of popular JavaScript/TypeScript developer tools (Wallaby.js and Quokka.js).

We’ve had a number of our Quokka users asking us to add support for deno. If you are not familiar with Quokka, Quokka provides an in-editor scratchpad experience for JavaScript/TypeScript. Runtime values and results are displayed in the IDE right next to your code. Quokka re-executes code as soon as you start typing and provides real-time/immediate feedback. For VS Code alone, Quokka has 733,000+ installations so we’re keen to add support for deno if we can; this may also help drive adoption of deno, or at the very least would help developers explore the deno runtime.

We spent the last week exploring deno and have identified some blockers to being able to support deno (without us forking the deno code base, which we don’t want to do). We were hoping to work with you so we can get to the point of supporting deno.

The issues we have are:

1. Need to be able to cancel Program unload. (as per your docs: […unload event] cannot be cancelled)

Providing real-time feedback while developers are typing means that Quokka has to be fast. To achieve that, we recycle the runner process. The way we do this in node.js is we have a network socket that we unref() before we execute user code and then we ref() it again in process.once('beforeExit'); this stops node.js from terminating between runs. For small projects with no dependencies, this probably doesn’t matter too much, but for larger projects, this will stop the effective use of Quokka.

2. Need to be able to invalidate local files on re-execution

This requirement falls out of (1). If we are recycling the runner process, we need to be able to invalidate any cached local files that may have changed when they are subsequently imported again on a fresh execution.

3. Ability to modify input files prior to execution

This is really the “no public API for compilers” issue (https://github.com/denoland/deno/issues/1739) but we can/have solved lack of API for compilers in other ways in the past that may be lower cost to implementing a full API for compilers with exposed AST, etc. In some other scenarios we have achieved similar functionality by duck-punching fs.readFileSync that is used by node.js when modules are required.

Obviously we appreciate that the paradigm for deno is different to node.js and with the focus on providing a secure by default runtime, we can’t do things the same way so may need first class support for some of our requirements vs. doing things in a hacky way.

---

Finally, we do know that we could get Quokka working with deno today without any of these features, but to do this, we’ll need to make some pretty big internal architectural changes and more importantly, we don’t think our approach will scale for larger projects, at least not until we have a solution to (1) and (2).

[kitsonk]

Specifically with #2 it feels like Quokka is doing that because no one else is, and actually in these 3 requirements it feels a bit like these needs are there because of what is built on top of it. With #2 it is effectively a controlled watch mode.

With number #3, what sort of modifications do you need to need to do to files? None of the module reading before being inserted into the isolate in Deno occurs inside the sandbox. (Which is why a public for API for compiler becomes interesting, but the challenge is we have been iterating and re-iterating on the whole mechanism of managing modules and caching them in Deno that it hasn't really ever been stable enough figure out a good way to expose the right APIs).

Overall it feels like the integration with Node.js is from within the sandbox, but ultimately to have that sort of control with Deno, it would require integration from the outside. I don't know if the debug port with the right APIs would work (for both sides) but that feels like something worth exploring.

[ArtemGovorov]

@kitsonk Test runners with (an efficient) watch mode in node land use #2 (mocha, ava, jest - a bit differently but still). Right now implementing something like mocha in watch mode in deno means reloading deno process on every file save. In node land the same process (with loaded testing framework and optionally other loaded third party modules) can be reused, because changed file(s) can be removed from the requre.cache, and it's making things much faster for subsequent test runs.

With number #3, what sort of modifications do you need to need to do to files?

We instrument files to collect various runtime information, including but not limited to code coverage and runtime values for user logged objects.

Example of another tool in node land that changes files on the fly when they are loaded is @babel/register. All of the above mentioned node test runners also need to change files on the fly (when compiling files with Babel prior to execution or/and when instrumenting files with Istanbul for code coverage).

[kitsonk]

For coverage, it is ultimately inefficient to use code to instrument, when we can get it from v8 directly (see: #106). Additional observability/instrumentation of the running code in theory should be available via the debugging protocol.

I can understand how things are done in Node.js, that doesn't specifically make following the same pattern good for Deno. Primarily, allowing sandboxed code to modify modules makes security a lot harder, so it isn't something that should be taken lightly. Hooking the module loading makes things even more complex. The vision of #1739 is that specific media types (maybe pattern matches) would be registered with a "compiler" that is loaded in a worker. The worker would receive requests to modify modules and would have APIs to request additional resources to be fetched and would be able to return "compiled" versions of the code. It would have, intentionally, no visibility of if that code ever gets loaded our executed. We have been making enhancements to the cache to be able to better know if a module needs to be "recompiled". I don't know the specifics of replacing modules in a running isolate, but I think I remember it being said that it was hard or impossible, due to the nature of how ES modules work.

Personal opinion here, but we need to shift the mental model with Deno in that we should focus on but figuring out a strategy to observe and control things outside the sandbox.

[ArtemGovorov]

I can understand how things are done in Node.js, that doesn't specifically make following the same pattern good for Deno.

Sorry if it sounded like I was suggesting that Deno should follow the same pattern as Node, I didn't mean to. We've only started to look at Deno after spending years developing tools for Node, so our comments/questions may sound a bit like we want this or that Node feature in Deno. However, our goal is to understand how to make things work for our users in Deno using Deno way and make sure that the way is not less performant or less attractive to our users than for the same features in Node.

For coverage, it is ultimately inefficient to use code to instrument, when we can get it from v8 directly (see: #106). Additional observability/instrumentation of the running code in theory should be available via the debugging protocol.

Unfortunately, neither v8 code coverage, nor observability/instrumentation of the running code via the debugging protocol is sufficient for the things we do. Having the full freedom over how to modify the code on the fly prior to execution unlocks a lot of possibilities for our tools (as opposed to being limited to v8/debugging protocol capabilities), such as implementing Time Travel Debugger. As for the API, we don't need anything much different from what would Babel (or any other compile-to-JS-on-the-fly language hook) need to be able to work in Deno.

Primarily, allowing sandboxed code to modify modules makes security a lot harder, so it isn't something that should be taken lightly.

Maybe it makes sense to consider 2 different scenarios:

• Deno as a production server runtime environment,
• Deno as a development tool running on a developer's machine.

For the first scenario, I agree that security is a very important concern. For the second scenario however, I think having more flexibility (maybe even by sacrificing some security if required) is what can allow many tools to exist and flourish in Deno's ecosystem, making it attractive for developers, and thus helping the first scenario.

As a developer coming to use Deno, I probably care if I can run my units tests faster (especially when I run then continuously in watch mode) rather than about whether or not I am running them in a more secure environment than I am in node.

Maybe it would helpful/useful to have some sort of dev flag/mode, that relaxes certain limitations and unlocks certain APIs (otherwise unavailable)?

[caspervonb]

Is there anything actionable for us here? Seems like most of this, if not all should be done via the inspector protocol.

[smcenlly]

@caspervonb - as @ArtemGovorov mentioned:

Unfortunately, neither v8 code coverage, nor observability/instrumentation of the running code via the debugging protocol is sufficient for the things we do.

[bartlomieju]

@smcenlly @ArtemGovorov I have a question regarding 2) and especially ESM integration in Node, how do you invalidate the cache in this situation? Does Node expose API that allow to unload ES modules and execute them again?

Point 3) is this hard to address as ES modules loading in done purely in Rust and doesn't expose any user hooks that would allow to override what's being fetched.

[smcenlly]

how do you invalidate the cache in this situation? Does Node expose API that allow to unload ES modules and execute them again?

When we reuse the worker process and reimport the file, we add a cache breaker (see below). Doing this causes the file and its downstream imports to be reloaded:

  const fileUrl = url.pathToFileURL(file);
  fileUrl.href = fileUrl.href + "?update=" + new Date().getTime();
  return await import(fileUrl);

[bartlomieju]

how do you invalidate the cache in this situation? Does Node expose API that allow to unload ES modules and execute them again?

When we reuse the worker process and reimport the file, we add a cache breaker (see below). Doing this causes the file and its downstream imports to be reloaded:

  const fileUrl = url.pathToFileURL(file);
  fileUrl.href = fileUrl.href + "?update=" + new Date().getTime();
  return await import(fileUrl);

@smcenlly this is surprising - I agree this will cache bust main imported modules, but how does it cache bust its dependencies. AFAICT dependencies don't get annotated with ?update= in which case dependencies would be the same. BTW this trick works in Deno too, though i causes a memory leak and previous version of imported module is never unloaded.

[edouardmisset]

Hi all, Thank you all for your work and taking the time to explore this issue.

For VS Code alone, Quokka has 733,000+ installations so we’re keen to add support for deno if we can; this may also help drive adoption of deno, or at the very least would help developers explore the deno runtime.

It's more than 3 millions today... ><

Is it still something you might be pursuing today (I know it's been 4 years...) ?