Open kelvich opened 1 year ago
Indeed, this would be awsome! 🚀 I tried to play with the wsproxy alone for a couple hours (https://github.com/neondatabase/wsproxy) but I suppose this is a bit outdated as it only serves websockets (and prometheus metrics) but the neon serverless client now seem to do fetch requests rather than websocket connections...
Something very simple for a start would be great (for instance, Docker images are great, but if you target development, this should be flexible enough as to permit all dev setups).
--edit: found additional info in this issue... In particular, the fact that websockets are only used if you use Client or Pool, not the neon() function
I already commend on the closed thread (https://github.com/neondatabase/serverless/issues/33#issuecomment-1710331914) but I think it makes sense to share it here too.
I've taken the steps from https://github.com/neondatabase/serverless/issues/33#issuecomment-1634853042 and created a Docker compose project to simplify the setup process for a local neon proxy + PostgreSQL DB when using the neon()
http proxy setup.
You can find the repo here: https://github.com/TimoWilhelm/local-neon-http-proxy.
@kelvich I would consider this a high impact.
We almost didn't consider neon because our developers prefer to have a fully offline localhost development option. Installing docker and running docker is a huge burden for those of us who have simple infrastructure (node+postgres only in our case).
For a lot of folks considering neon, having neon appear like a drop-in replacement for postgres would remove a lot of hesitation. It's almost there except for the localhost experience :)
Perhaps this could be designed to work outside of the box? For example, the library could auto-detect whether a connection string is a neon string or a regular postgres string and use a built-in proxy to reduce the number of moving parts. Maybe wsproxy could be ported to javascript to reduce the number of moving parts and steps to installation.
A bit more context: https://github.com/neondatabase/serverless/issues/33
I tried to get pretty far on this with a local postgres database running in docker with a NextJS edge function, using a bunch of different ws:// -> tcp://
proxy implementations but ended up getting FATAL: invalid frontend message type 112
errors back on the postgres side... I haven't figured out yet where things got tripped up, but wanted to leave this here in case its useful for anyone else
For those arriving here, I did get this working... Here is my setup (big thanks for @kincaidoneil who cracked the code with disabling pipelineConnect
):
import {
Pool as NeonPool,
neonConfig,
} from "@neondatabase/serverless";
import {
PrismaNeon,
} from "@prisma/adapter-neon";
import { PrismaClient } from "@prisma/client";
import ws from "ws";
neonConfig.webSocketConstructor = typeof WebSocket !== "undefined" ? WebSocket : ws;
function buildNeonClient(databaseUrl: string, isLocalDevelopment: boolean): PrismaClient {
// https://gist.github.com/kincaidoneil/bc2516111f0ec8850cd6020b8191b27b
if (isLocalDevelopment) {
neonConfig.pipelineConnect = false;
neonConfig.useSecureWebSocket = false;
neonConfig.wsProxy = () => "localhost:5433";
}
const neon = new NeonPool({ connectionString: databaseUrl });
const adapter = new PrismaNeon(neon);
const prisma = new PrismaClient({ adapter });
return prisma;
}
Then, to make this work, run a websocket -> TCP proxy like this:
# Your database URL would point to this proxy, assuming postgres is at localhost:5432
# example: DATABASE_URL=postgresql://postgres:password@localhost:5433/postgres
wstcp --log-level debug --bind-addr 127.0.0.1:5433 127.0.0.1:5432
For those arriving here, I did get this working... Here is my setup (big thanks for @kincaidoneil who cracked the code with disabling
pipelineConnect
):import { Pool as NeonPool, neonConfig, } from "@neondatabase/serverless"; import { PrismaNeon, } from "@prisma/adapter-neon"; import { PrismaClient } from "@prisma/client"; import ws from "ws"; neonConfig.webSocketConstructor = typeof WebSocket !== "undefined" ? WebSocket : ws; function buildNeonClient(databaseUrl: string, isLocalDevelopment: boolean): PrismaClient { // https://gist.github.com/kincaidoneil/bc2516111f0ec8850cd6020b8191b27b if (isLocalDevelopment) { neonConfig.pipelineConnect = false; neonConfig.useSecureWebSocket = false; neonConfig.wsProxy = () => "localhost:5433"; } const neon = new NeonPool({ connectionString: databaseUrl }); const adapter = new PrismaNeon(neon); const prisma = new PrismaClient({ adapter }); return prisma; }
Then, to make this work, run a websocket -> TCP proxy like this:
# Your database URL would point to this proxy, assuming postgres is at localhost:5432 # example: DATABASE_URL=postgresql://postgres:password@localhost:5433/postgres wstcp --log-level debug --bind-addr 127.0.0.1:5433 127.0.0.1:5432
I've noticed this works well when accessing the db from the front end (for example taking some action on the front end that calls an API route) but for some reason when I use Postman to hit the api endpoints directly it doesn't proxy the request correctly--the db call fails in the middleware.ts file. This is what the error looks like:
⨯ Error: All attempts to open a WebSocket to connect to the database failed. Please refer to https://github.com/neondatabase/serverless/blob/main/CONFIG.md#websocketconstructor-typeof-websocket--undefined. Details: fetch failed
at eval (webpack-internal:///(middleware)/./node_modules/@neondatabase/serverless/index.mjs:1359:74)
at async PrismaNeon.performIO (webpack-internal:///(middleware)/./node_modules/@prisma/adapter-neon/dist/index.mjs:385:79)
at async PrismaNeon.queryRaw (webpack-internal:///(middleware)/./node_modules/@prisma/adapter-neon/dist/index.mjs:347:17)
at async eval (webpack-internal:///(middleware)/./node_modules/@prisma/client/runtime/wasm.js:2:7537)
proxy logs:
Jun 20 04:58:13.460 DEBG Header: host: localhost:5433
connection: keep-alive
upgrade: websocket
user-agent: Next.js Middleware
x-middleware-subrequest: src/middleware
accept: */*
accept-language: *
sec-fetch-mode: cors
accept-encoding: gzip, deflate
, client_addr: 127.0.0.1:50588, server_addr: 127.0.0.1:55433, proxy_addr: 127.0.0.1:5433
Jun 20 04:58:13.460 WARN Invalid WebSocket handshake request: InvalidInput (cause; assertion failed: `values.any(|v| v.trim() == "Upgrade")`; value="keep-alive")
HISTORY:
[0] at /Users/beebs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wstcp-0.2.1/src/channel.rs:162
[1] at /Users/beebs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wstcp-0.2.1/src/channel.rs:81
, client_addr: 127.0.0.1:50588, server_addr: 127.0.0.1:55433, proxy_addr: 127.0.0.1:5433
I don't have a solution yet but I'll update this if I figure something out
For those arriving here, I did get this working... Here is my setup (big thanks for @kincaidoneil who cracked the code with disabling
pipelineConnect
):import { Pool as NeonPool, neonConfig, } from "@neondatabase/serverless"; import { PrismaNeon, } from "@prisma/adapter-neon"; import { PrismaClient } from "@prisma/client"; import ws from "ws";
neonConfig.webSocketConstructor = typeof WebSocket !== "undefined" ? WebSocket : ws;
function buildNeonClient(databaseUrl: string, isLocalDevelopment: boolean): PrismaClient { // https://gist.github.com/kincaidoneil/bc2516111f0ec8850cd6020b8191b27b if (isLocalDevelopment) { neonConfig.pipelineConnect = false; neonConfig.useSecureWebSocket = false; neonConfig.wsProxy = () => "localhost:5433"; } const neon = new NeonPool({ connectionString: databaseUrl }); const adapter = new PrismaNeon(neon); const prisma = new PrismaClient({ adapter }); return prisma; } Then, to make this work, run a websocket -> TCP proxy like this:
Your database URL would point to this proxy, assuming postgres is at localhost:5432
example: DATABASE_URL=postgresql://postgres:password@localhost:5433/postgres
wstcp --log-level debug --bind-addr 127.0.0.1:5433 127.0.0.1:5432
followed all your steps and still get "Error: PrismaClient is unable to run in this browser environment, or has been bundled for the browser (running in unknown
)." any tips
Motivation
With more adoption of serverless driver we got more requests for local and CI setups for proxy.
DoD
We have local setup that is:
Implementation ideas
So far we had two ideas:
neon dev
commandSecond approach is intriguing but it is way more work (e.g. we should distribute binaries in npm, like esbuild does) so it seems better to proceed with docker image approach for now.
For the local experience to be end-to-end we need the following things:
So we can target for the following flow:
To make things easier to use I suggest to embed postgres into the image. At the same time it would be nice to support env var that can supply external postgres URL for cases when postgres is already configured.