node / typescript workflows

[JAForbes]

Better node / typescript workflows

Today, it is possible to call arbitrary node code server side within a template literal.

I think this is cool but in practice it's not that ergonomic to work with, it's very stringly typed and you might need to escape your code (e.g. is there a template literal inside your template literal).

It also completely removes the ability to have typed languages because all that type information that would normally be analyzed by a compiler is "trapped" inside a string.

I suggest we build in support for hashed modules.

Hashed modules

Hashed modules simply copy a module's subtree in the same way we copy source code. But hashql generates a transformed replica of those modules (updating relative imports for example).

From an editors perspective it's just a dynamic import with an async default export. Both JS and TS will be able to find the original source file and identify the type information from that source file.

So if you have some js module, you can just dynamically import it and call it:

import('./my-lambda.js').then(
  ({ default:f }) => f({ a: 1, b: 2 })
)

Which is compiled to something like:

hashql.module(<hash-of-compiled-subtree>).then(
  ({ default:f }) => f({ a: 1, b: 2 })
)

The cool thing to note is, this is 100% typed. An IDE will chug along not knowing we've done anything at all to the source, and it will simply look at the originally referenced path and gather type information itself.

Typescript

The above will work with typescript, but hashql doesn't yet have an example of transpilation of source in the hashql handler. I think this is pretty easy to do as a build step, but it'd be good to have in /examples so people can figure it out. It'd also be good to have some ootb modules for rigging up common workflows (e.g. hashql-rollup-typescript or hashql/rollup/typescript).

Static Usage

This can definitely work for static imports too, with a few restrictions, all exported values are async.

So if you have a module:

export const a = 1

And you import it, a will be a Promise

import * as example from './example'

example.a.then(console.log) // logs 1

In practice that won't be too much of an issue, if you want code to run on the server it's likely async anyway. But I think it'd be valuable for the compile step to warn that a hashed module was imported with non async imports - to avoid confusion.

This problem is mitigated with dynamic imports because async is enforced, But I think we shouldn't exclude static imports, it just feels nice to do something like import * as api from './api' and then call api.createUser(user) from the client as if it was all in the same process.

Questions

How do we identify which imports should be hashed, and which shouldn't is an open question. We could have a comment, or we could have a config file with path globs. Because this doesn't use template literals it requires some new interface design work in that department.

Why?

This feature will give hashql editor support for free. There's no reason we can't take this same technique and use it with other language's module systems (e.g. python/rust/go), and those editors will all already know how to get the type information from those imports.

This relieves a lot of pressure from the library to do special custom things like generating tsd's or the like.

[JAForbes]

I just had a thought on this front. I'm still really excited by hashed modules.

We don't need to generate client side modules, we can simply have a proxy for all hashed modules and when you import server side code a proxy module is returned instead.

The proxy acts similarly to hashql, when a function is invoked from the proxy module object a request is made with the query being a hash of the imported path and the values being the JSON serialized parameters passed to the function.

The proxy will let you arbitrarily access any deeply nested sub property and invoke and such property as a function.
VSCode et al won't know it is a proxy, so it will offer whatever intellisense would normally be provided for that import.

Another benefit is, you can import whatever languages you want as long as your server can intercept and handle that request server side. So typescript for example should work out of the box without any extra work.

The build story for production would be basically the same as the existing rollup module. There would just need to be an AST rewrite for static/dynamic imports at certain paths instead of searching for tagged template functions.

And a sensible default could be any import for files that are not in shared or child directories of the client side entry point will use hashed modules.

So

// entry point to client code

// not a hashed module
import component from './component/example.js'

// also not a hashed module
import styles from './styles.js'

// by default, a hashed module
import server from '../server/example.js'

// server is a proxy object, when we invoke a function
// the call is translated to a POST request like so:
// POST { query: md5({ "file": /server/example.js', "path": 'nested.someFunction' }), values: [1,2] }
await server.nested.someFunction(1,2)

No eval is required, the server just resolves the request via a dynamic import, the path will have been resolved to the project root. And pass the values into the executed function. That would be the default server side handler. But you could also run that server side module in a lambda, a cloudflare worker, or on some other server.

[porsager]

This is great James!

I'm not even sure we'd need proxies? We can just convert "server calls" in the client output to calls like we do currently.

So eg.

import server from '../server/example.js'
await server.nested.someFunction(1,2)

would just be compiled into

import hql from 'hql' // which can be bundled further
await hql(hash, ...args)

The relevant info for lookup (file path and object path) would go into the output hashql uses in the server, and the client lib would expose an option to handle these "global" requests.

[JAForbes]

Ah that's even better!

[JAForbes]

I'm juggling several big things at the moment so I can't tackle it right now but I'm keen to in the next few weeks as long as you're not planning on it @porsager?

[JAForbes]

I'm not even sure we'd need proxies? We can just convert "server calls" in the client output to calls like we do currently.

I'd really like this to work without having to transform the client side source at all for local development. And for that to work, I think there needs to be some kind of "fake module" sent back from a dev server for local development. Not necessarily using proxies, but with the same export interface as the real module.

That way the dev server can respond to normal native imports. And import maps could be conceivably used to configure which paths target the hashed modules' server side handler.

[JAForbes]

Just leaving a note to say I'm still intending to continue this work, just got a bit on my plate at the moment. Should be able to pick it up again in a few weeks.