psFried
commented
1 year ago I've done some research and playing around with Deno and deno_core and wanted to share some notes on how I see the major pieces fitting together.
Build-time operations
In addition to executing the lambdas at runtime, there's a number of other things that we need to do during builds.
- Typechecking
- Resolve dependencies
- Packaging
Currently, those operations are handled by a mix of npm and tsc. Deno provides all of those operations in a single subcommand called bundle, which emits a single Javascript file with all dependencies inlined. This would be stored in the build database with a DENO_MODULE type in the resources table. Integrating Deno as a library gives a much more low level interface where all those operations are separate. I'd propose that, at least for starting out, we use the CLI for build-time operations. The Deno CLI would be added to our docker image, and the builds would shell out to deno bundle to produce a fully inlinded JS file for each derivation.
Changing the scope of the bundle
Our current NPM modules are scoped to the build. That is, there is a single npm module that includes the code for all the derivations within a given build. As part of our move to Deno, we should change the scope of those so that we end up with a separate module (and a separate V8 isolate) for each derivation. Having a separate isolate per derivation allows for better isolation, as the name implies, and also allows multiple derivations from the same build to run in parallel on the same reactor.
- Direct invocation seems more efficient, anyway.
- It seems desirable to isolate the modules from different tasks, anyway, for a variety of reasons.
- We could potentially also use a separate "realm" ("context" in V8 terminology) for each lambda within an isolate, to guard against users who may try to update some global JS state in one lambda invocation and then read it in another.
Runtime integration
- Deno doesn't yet support unix sockets for HTTP, so we can't simply re-use the existing integration model from NodeJS. We probably wouldn't really want to, anyway, so NBD.
- Integrating with Deno as a library is very different from running Deno as a CLI. The
deno_corelibrary is much more low level than the CLI. Even things likeconsole.logneed to be explicitly configured. - Deno requires an async executor. It's unclear whether our existing "haxecutor" will work for this purpose. It can probably work well enough if we disable file and network permissions, but requires testing.
- I think we'll want to disallow network access from derivations, anyway, at least for now. In the future, we might be able to allow access to only the public internet or something. I'm not taking a position on whether that's a good idea or not, but just trying to limit scope in the short term.
- It seems like the design of Deno doesn't really afford "pushing" data. You can't just ask Deno (or V8, for that matter) "please invoke function X with parameters Y and Z", at least not directly. You can pass it a string and say "please execute this script", but that seems like a poor way to pass data into the runtime.
- Instead, the prevailing pattern seems to be that you define
ops, which are registered at runtime creation, and then you expose eachopas a JS function that client code can call. - It seems desirable to have Deno-based lambdas use exactly the same function signatures that our Node-based lambdas do today. In other words, the Typescript code that's written by users for NodeJS transforms ought to be accepted as-is for Deno.
I think the basic form of the integration would be an Extension that we implement. This extension would provide ops that allow JS code to read in documents from the runtime and send results back to the runtime. We would then have an automatically generated main.ts module that drives the execution of user-provided transform functions using those primitives. Something like this extremely hand-wavey illustrative example:
while (true) {
let next_doc = Flow.getNextSourceDocument();
let userFunction = lookupUserFunction(next_doc.source_meta);
let result = userFunction(next_doc.document);
Flow.sendResult(result);
}A particular detail I'd like to point out is the lookupUserFunction function. We need some way to associate source documents with specific update/publish lambdas. There's lots of different ways we could do that, and I'm not suggesting any particular implementation just yet (and don't know what exactly source_meta is). But some options that I see there are:
- Use and integer to identify each source binding, and a flag that says whether it's an update or publish lambda. Use those to lookup the proper function in an automatically generated data structure in JS.
- Have a separate loop per lambda, where each loop runs concurrently and passes the binding id to
getNextSourceDocument - Have the runtime pass in the name of each function directly on the
next_doc - Something else?
Some other things to note about runtime integration:
- I haven't yet measured how much overhead there is for an
opcall (though I suspect it's tiny). I say this only to point out that there could be some benefit to batching documents as they transit the Rust<=>V8 boundary, and thus the actual API might change a bit. - Our extension would also provide
console.logandconsole.errorimplementations that directly forward to our log publisher. - There's no reason why the
main.jshas to be generated at build time. This could be done at runtime, though I don't necessarily think that's what we should do. - V8 has the concept of "Snapshots", which essentially just allow you to make creating new context objects much faster (anecdotally, ~40ms to ~2ms). I'm ignoring those, because creating a new context should only be done once per task term, so those ~38ms don't seem very important to us.
- The high-level integration model of having the script call back into the runtime could afford a lot of potential features in the future. For example, we could allow the derivation more flexible access to the rocksdb-backed KV store
jgraettinger
dhammika
The goal is to use Deno instead of NodeJS for executing transform lambdas. The primary motivation for this is security and isolation. The current NodeJS implementation is difficult to sandbox, and Deno has a much better story around that.