POC

[kiwicopple]

Proof of concept

Describe the chore

The task is to come up with a design or POC that will allow Postgres & Deno to work together.

[kiwicopple]

From @J0

The project as a whole seems more complicated than I anticipated. I'm not sure how I'll interact with Deno itself since it's written in Rust and Postgres extensions seem to need to be in C/C++ . I noticed that the Deno internals has C++ bindings via libdeno which wraps V8 -- I'm not sure how to interact with the Deno process as a whole.

@steve-chavez this one is probably up your alley. Do you have a good idea how to approach this task in a "hello world" type approach?

[steve-chavez]

I'm not sure how I'll interact with Deno itself since it's written in Rust and Postgres extensions seem to need to be in C/C++

Not necessarily. There's zombodb/pgx which allows writing extensions in Rust.

Do you have a good idea how to approach this task in a "hello world" type approach?

@kiwicopple Maybe calling a deno process from a pg background worker could be a good start. (Would have to give it more thought to be certain)

[eeeebbbbrrrr]

Author of https://github.com/zombodb/pgx here. If you need anything, don't hesitate to reach out. We have a public discord server.

[eeeebbbbrrrr]

Hi, I wanted to circle back with a link to something I'll be releasing soon that could serve as an inspiration for this project: https://github.com/zombodb/plrust

There's no docs yet, but it's a standard pgx extension so you need cargo install cargo-pgx to build it.

After that you need to add 2 settings to your target postgresql.conf file:

plrust.work_dir = '/path/to/where/plrust/compiles/functions'
plrust.pg_config = '/full/path/to/pg_config'

Once you've built and installed the extension (cargo pgx run pgXX for testing) then you can do:

CREATE EXTENSION plrust;
CREATE FUNCTION my_lower_case(input text) RETURNS text STRICT LANGUAGE plrust AS $$
   Some(input.to_lower_case())
$$;
SELECT my_lower_case('HI THERE');

This might serve as a useful place for you to get started. I haven't released it yet for three reasons (that I'm working on):

• pgx lacks trigger function support, so making a plrust trigger function isn't yet supported. I've got WIP code for this locally
• At least on MacOS, you can't dlclose() a shared library that uses thread local storage and then get the new version on a subsequent dlopen() -- you get the previously loaded .so as in this case dlclose() is a no-op. This means CREATE OR REPLACE FUNCTION doesn't do what you expect. I think there's a solution here, but I haven't found it yet
• Documentation

If you find this useful, great. If supabase.io is interested in working with me on plrust or pgx, definitely let me know -- sponsorships or work-for-hire contracts are all doable.

[kiwicopple]

Hey @eeeebbbbrrrr, great to see you here - we love what you're doing with Zombo

If supabase.io is interested in working with me on plrust or pgx

I think it would be amazing to collaborate on one of these. We're still dealing with the aftermath of our beta launch, so I'll reach out next week.

For this repo, we're still brainstorming the implementation (and if it's worthwhile from a developer perspective). @inian I think you had some ideas here - it could be run as a layer on top of plv8 right?

[eeeebbbbrrrr]

we love what you're doing with Zombo

Hey, thanks. Me too! The past ~6yrs have been a blast.

dealing with the aftermath of our beta launch

I get it. ZomboDB 3000 (ha!) just entered alpha a few weeks ago, and I'm up to my eyeballs in work with that. Fortunately, I'm kinda short.

we're still brainstorming the implementation

I've got a suggestion. Hear me out. It's not as crazy as it sounds...

Forget things like pl/v8 and javascript -- write a Typescript-to-Rust transpiler. I haven't really written any Typescript, but this doesn't sound too difficult.

Then use pgx to make that available to Postgres. I'm serious. We're only talking about functions that are as long-lived as the evaluation of a database row in Postgres' executor. You don't need a full JS engine to do that -- and probably don't want one anyways.

One of the issues I foresee with p/rust (which will be released soon!) is that it won't be a "trusted" Postgres language -- it's pure, unadulterated, 100% Rust. Wanna write an extern "C" declaration into Postgres' DropTableSpace() function? Go for it! That's gonna hinder adoption a bit. It's not gonna keep me from finishing it, but I've got my expectations leveled.

With a pl/typescript thing that's actually Rust (which is then compiled to machine code), you can be trusted, as type-safe as Postgres, and execute (nearly) as fast as a function written in C.

it would be amazing to collaborate on one of these

Hit me up. eebbrr at gmail dot com. I'd love to chat.

[soedirgo]

Thanks for looping in @eeeebbbbrrrr!

Forget things like pl/v8 and javascript -- write a Typescript-to-Rust transpiler.

One problem I have with this is that TS and Rust are both fast moving targets, it'll be a challenge to keep a transpiler up to date hand in hand with pldeno.

Actually, a smaller scope would be a JS-to-Rust transpiler (since TS is allegedly a superset of JS). Problems with dynamic typing also apply here. I don't think it's impossible, but if it's worth doing I'd expect someone to have attempted this already (e.g. Cloudflare for their Workers or AWS for Lambda).

Maybe I'm missing something, I'm just a budding Rustacean 🦀. Would love to hear your thoughts on this!

[kiwicopple]

Adding options here - I chatted to the Wasmer team about using their WASM binding: https://github.com/wasmerio/wasmer-postgres

Wasm brings a lot of benefits (any language you want), although it doesn't fit our use-case because the binary needs to be installed on the server. Deno is more of a "JIT", where our users can add easily functions and import modules from any location on the web, at least that's the goal

[eeeebbbbrrrr]

TS and Rust are both fast moving targets

I think that's overblown for Rust. Target a Rust language edition (say 2018) and you're gtg.

I didn't look hard, but I couldn't find anything official about TS language versions or plans or whatever.

There's probably a few months of work on such a thing.

smaller scope would be a JS-to-Rust transpiler

JS being JS, I wonder if it would be smaller scope. The naive implementation would probably generate a "maps of maps"-type structure for each JS object -- wonder how performant that would be?

[eeeebbbbrrrr]

I chatted to the Wasmer team about using their WASM binding

I've seen that. Haven't used it. Looks like it only supports 3 integer data types and is limited to PG10. I'm sure it'll improve over time tho. I know it ain't easy doing this stuff with Postgres

where our users can add easily functions and import modules from any location on the web

If that's the goal then you're not looking at a "trusted" PL. I mean, that's totally fine. Just pointing it out.

Does Deno use v8 behind the scenes or is Deno a full JS runtime implemented in Rust? I haven't been able to figure that out?

[inian]

Hey @eeeebbbbrrrr , yup the goal is not to build a trusted extension. We are looking at deno because it lets you import from URLs directly and this way developers get the power of the node ecosystem without directly bundling it with their function.

Deno does use v8 behind the scenes. It is the wrapper that is built in Rust.

To get a trusted typescript extension, it might be easier to integrate tsc on top of plv8, but the long term goal is have deno programs working in the extension.

[syrusakbary]

It will be super exciting to see Wasm support in Supabase. We have talked with a few DBs that are also prototyping it... including Redis! So it will be really great to see it widely adopted :)

Let me know if I can help in any way!

[shah]

I was curious if there has been any further progress on this extension. We're using Deno quite a bit for database functionality (outside of PostgreSQL) but would love to use it internally for stored routines as well.

PLV8 v3.0.0 was released about 11 days ago (it's a "Procedural Language in Javascript powered by V8"). We're exploring the latest version of PLv8 but would much prefer a Deno-native capability inside PostgreSQL.

Thanks for all the hard work!

[kiwicopple]

We didn't make much progress here but @burggraf created this extensions which extends PLV8 with a lot of cool functionality: https://github.com/burggraf/SupaScript (i.e. import libraries from the web)

[shah]

Thanks @kiwicopple - that does look like a fantastic add-on to PLV8 and importing libraries from the web is one of the best features of Deno so have that in PLV8 may meet many of my needs. What I'm salivating about with respect to this postgres-deno extension, though, is native TypeScript support :-)

[burggraf]

@shah I guess we could look at a compile step that turns TypeScript into JS first. I'm a huge TypeScript fan, too, so that'd be pretty cool.

[shah]

Thanks @burggraf that would be fantastic -- I'd be less nervous about PL/v8 if we could take our TypeScript into the database!

[burggraf]

Just remember that TypeScript compiles down to JavaScript, so you'd never really be running TypeScript :)

[shah]

Agreed @burggraf - and you're 100% right, but I always feel better writing in Rust or TypeScript to help catch simple type errors as early in the process as possible. I think your idea of taking the compile step of TypeScript to JS is perfect ... the best of both worlds: type-safety as well as ubiquity through JS!