Hyperdrive support via deno-postgres

mlanza commented 2 years ago

I have a index.js script which imports postgres drivers:

import { Client, Pool } from "https://deno.land/x/postgres/mod.ts";

When running I get:

mlanza@Marios-MacBook-Air df-ex % denoflare serve hello-local
Compiling https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli-webworker/worker.ts into worker contents...
Compiled https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli-webworker/worker.ts into worker contents in 630ms
runScript: index.js
worker: start
Error running script TypeError: Requires net access to "deno.land", run again with the --allow-net flag
    at blob:null/33e03125-26c1-4b7b-88c2-5e4888b766d3:1:30
    at async Function.create (blob:null/2b6f1dc4-863b-4125-87f2-b6f05e8dcbbb:428:9)
    at async Function.start (blob:null/2b6f1dc4-863b-4125-87f2-b6f05e8dcbbb:496:125)
    at async blob:null/2b6f1dc4-863b-4125-87f2-b6f05e8dcbbb:1547:22
    at async RpcChannel.receiveMessage (blob:null/2b6f1dc4-863b-4125-87f2-b6f05e8dcbbb:34:36)
    at async self.onmessage (blob:null/2b6f1dc4-863b-4125-87f2-b6f05e8dcbbb:1587:13)

This was after confirming that serving using the vanilla .ts example worked. I am using .js. I tried passing in the --allow-net flag myself but I'm not sure the baton got passed.

I ignored the local failure and trudged on:

mlanza@Marios-MacBook-Air df-ex % denoflare push hello-local                                     
bundling hello-local into bundle.js...
Download https://deno.land/x/postgres/mod.ts
Warning Implicitly using latest version (v0.14.2) for https://deno.land/x/postgres/mod.ts
Download https://deno.land/x/postgres@v0.14.2/mod.ts
Download https://deno.land/x/postgres@v0.14.2/client.ts
Download https://deno.land/x/postgres@v0.14.2/client/error.ts
Download https://deno.land/x/postgres@v0.14.2/connection/connection_params.ts
Download https://deno.land/x/postgres@v0.14.2/pool.ts
Download https://deno.land/x/postgres@v0.14.2/query/query.ts
Download https://deno.land/x/postgres@v0.14.2/query/transaction.ts
Download https://deno.land/x/postgres@v0.14.2/connection/message.ts
Download https://deno.land/x/postgres@v0.14.2/query/decode.ts
Download https://deno.land/x/postgres@v0.14.2/query/encode.ts
Download https://deno.land/x/postgres@v0.14.2/utils/utils.ts
Download https://deno.land/x/postgres@v0.14.2/utils/deferred.ts
Download https://deno.land/x/postgres@v0.14.2/connection/connection.ts
Download https://deno.land/x/postgres@v0.14.2/connection/packet.ts
Download https://deno.land/x/postgres@v0.14.2/deps.ts
Download https://deno.land/x/postgres@v0.14.2/query/decoders.ts
Download https://deno.land/x/postgres@v0.14.2/query/oid.ts
Download https://deno.land/x/postgres@v0.14.2/connection/auth.ts
Download https://deno.land/x/postgres@v0.14.2/connection/message_code.ts
Download https://deno.land/x/postgres@v0.14.2/connection/scram.ts
Download https://deno.land/x/postgres@v0.14.2/query/array_parser.ts
Download https://deno.land/x/postgres@v0.14.2/query/types.ts
Download https://deno.land/std@0.114.0/async/mod.ts
Download https://deno.land/std@0.114.0/bytes/mod.ts
Download https://deno.land/std@0.114.0/datetime/mod.ts
Download https://deno.land/std@0.114.0/encoding/base64.ts
Download https://deno.land/std@0.114.0/fmt/colors.ts
Download https://deno.land/std@0.114.0/hash/mod.ts
Download https://deno.land/std@0.114.0/hash/sha256.ts
Download https://deno.land/std@0.114.0/io/buffer.ts
Download https://deno.land/std@0.114.0/async/deadline.ts
Download https://deno.land/std@0.114.0/async/debounce.ts
Download https://deno.land/std@0.114.0/async/deferred.ts
Download https://deno.land/std@0.114.0/async/delay.ts
Download https://deno.land/std@0.114.0/async/mux_async_iterator.ts
Download https://deno.land/std@0.114.0/async/pool.ts
Download https://deno.land/std@0.114.0/async/tee.ts
Download https://deno.land/std@0.114.0/datetime/formatter.ts
Download https://deno.land/std@0.114.0/bytes/equals.ts
Download https://deno.land/std@0.114.0/bytes/bytes_list.ts
Download https://deno.land/std@0.114.0/io/types.d.ts
Download https://deno.land/std@0.114.0/hash/_wasm/hash.ts
Download https://deno.land/std@0.114.0/hash/hasher.ts
Download https://deno.land/std@0.114.0/datetime/tokenizer.ts
Download https://deno.land/std@0.114.0/encoding/hex.ts
Download https://deno.land/std@0.114.0/hash/_wasm/wasm.js
bundle finished in 1444ms
computed bindings in 0ms
putting script hello-local... (328.9kb) (93.7kb compressed)
error: Uncaught (in promise) Error: putScript failed: status=400, errors=10021 Uncaught ReferenceError: FinalizationRegistry is not defined
  at line 0

    at execute (https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/common/cloudflare_api.ts:164:15)
    at async putScript (https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/common/cloudflare_api.ts:40:13)
    at async buildAndPutScript (https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli/cli_push.ts:52:9)
    at async push (https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli/cli_push.ts:60:5)
    at async https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli/cli.ts:15:13

I tried to deploy another version of the same script using wrangler and it failed to resolve and bundle all the webpack deps. I never got it working. I got the script, same import, working on Deno Deploy in short order so I know it's conceptually sound.

I am trying to repeat that success on Cloudflare's platform using your tool. I am excited about it. Having a simple tool for bundling/deploying ES modules directly to Cloudflare will be a huge blessing. Thank you for sharing your work.

PS: I haven't crossed the TypeScript divide yet so having working JavaScript examples would be helpful for stragglers like me. I looked at the https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/common/config.schema.json and I was able to ascertain that I should use standard Worker syntax, but it took an extra couple steps to figure that out.

mlanza commented 2 years ago

I may have been mistaken in thinking it reasonable to connect to and interact with an RDMBS from a FaaS, e.g. Cloudflare Workers. But I'm not entirely certain. I am getting mixed signals.

THE SERVERLESS SERIES — Mistakes You Should Avoid | by Nicolas Dao | Neap For Startups Query Postgres from Workers using a database connector · Cloudflare Workers docs

This post gives the impression this is a problem not yet solved: Making connections with TCP and Sockets for Workers

Anyway, your call. Close if you have the sense that RDBMSes were not intended to be called from a FaaS. Been web developing a while, but serverless is new to me.

johnspurlock-skymethod commented 2 years ago

You are on the cutting edge - Cloudflare does not support TCP clients yet (although it's coming soon [1]), but you can use a websocket proxy with a tunnel today (although I've not tried it). It is something on their radar that they want to support.

I'll try to get access to client TCP from a Cloudflare worker and look at a postgres client scenario. Do you have the sample code you were using?

[1] https://blog.cloudflare.com/introducing-socket-workers/

johnspurlock-skymethod commented 2 years ago

And as far as making direct network calls when running locally in denoflare serve, you might try to run without isolation, i.e. with localIsolation: "none" in your config. This way whatever Deno permissions you give the denoflare process should apply to your script, but you won't be able to hot reload - and obviously it doesn't match the runtime permissions in Cloudflare production as you've already found out.

[1] https://denoflare.dev/cli/configuration#script

mlanza commented 2 years ago

Here's code which worked in Deno Deploy with the exception that my connection string is PRIVATE:

import { listenAndServe } from "https://deno.land/std@0.111.0/http/mod.ts";
import { Pool } from "https://deno.land/x/postgres/mod.ts";
const databaseUrl = `XXXXXX`;
const pool = new Pool(databaseUrl, 4, true); // `true` indicates lazy connections

async function exec(text) {
  const client = await pool.connect();
  let result;
  try {
    result = await client.queryObject(text);
  } finally {
    client.release();
  }
  return result;
}

async function handler(req, e) {
  const url = new URL(req.url);
  const result = await exec('SELECT s.*, u.email_address FROM sessions s JOIN users u ON u.id = s.owner');
  const body = JSON.stringify(result.rows, null, 2);
  return new Response(body, {
    status: 200,
    headers: {"content-type": "application/json"},
  });
}
await listenAndServe(":8080", handler);

While Deno Deploy and Cloudflare Workers may not be identical they are both built to use v8 isolates. Also, I am not sure whether the idea of using a FaaS from these platforms is sound with an RDBMS like Postgres. I was hopeful, but I may be mistaken.

I adapted it to the Cloudflare Workers pattern and could not get it working, which may be okay if it's not a right fit. And even though Cloudflare does offer some documentation on connecting to Postgres via a worker I'm not convinced it's practical:

Query Postgres from Workers using a database connector · Cloudflare Workers docs cloudflare/worker-template-postgres: Reference demo and modified PostgreSQL driver to connect Cloudflare Workers to a relational database.

hansottowirtz commented 2 years ago

Looked into this issue recently as well, and basically bundled the template into a single module here: https://github.com/bubblydoo/cloudflare-workers-postgres-client. Although I'm not using Denoflare yet, I think you can use:

import { Client } from "https://esm.sh/@bubblydoo/cloudflare-workers-postgres-client@0.0.8"

You will need a Cloudflare Tunnel though, but it's pretty easy to set up if you're familiar with Docker.

johnspurlock-skymethod commented 1 year ago

Now that Cloudflare has released Hyperdrive, I think it's time to take another look at this. Would love to get basic Client access working over the cf-specific Socket interface, without pools (Hyperdrive takes care of this)

Will need to slightly modify deno-postgres to abstractify the Deno.* stuff from connection.ts and remove the need for FinalizationRegistry, which is not implemented in workers and causes the script upload to fail.

Too bad the deno-postgres module looks abandoned, will need to maintain a fork in the short term.

johnspurlock-skymethod commented 1 year ago

ok, got a working proof of concept connecting to Hyperdrive (to Supabase) from within a Cloudflare worker using a slightly modified deno-postgres:

forked deno-postgres to https://github.com/skymethod/deno-postgres
added a minimal patch to trim out unnecessary crypto dep and allow an alternate Deno.Conn factory - will try to upstream this
started an alternate Deno.Conn factory implementation using Cloudflare sockets as a reusable helper in Denoflare. common/cloudflare_sockets_deno_conn_factory.ts

mlanza commented 1 year ago

I am no longer pursuing this issue. If someone else is, please reopen.

johnspurlock-skymethod commented 1 year ago

Hehe - I am, clearly

skymethod / denoflare

Hyperdrive support via deno-postgres #6