[🐛 Bug]: `pg` does not work

janpio commented 10 months ago

next-on-pages environment related information

System:
        Platform: linux
        Arch: x64
        Version: #202312122204 SMP PREEMPT_DYNAMIC Tue Dec 12 22:43:56 UTC 2023
        CPU: (16) x64 AMD EPYC 7B13
        Memory: 63 GB
        Shell: Unknown
Package Manager Used: npm (10.2.3)

Relevant Packages:
        @cloudflare/next-on-pages: 1.8.2
        vercel: 33.0.0
        next: 14.0.4

Description

pg has support for Cloudflare Workers and Pages. This project makes it possible to run Next.js projects on Cloudflare Pages.
Unfortunately something gets lost along the way, so that pg triggers webpack errors during build and dev:

$ npm run build

> next-pg@0.1.0 build
> next build

   ▲ Next.js 14.0.4

Failed to compile.

./node_modules/pg-connection-string/index.js:76:69
Module not found: Can't resolve 'fs'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pg/lib/connection-parameters.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/helper.js:3:11
Module not found: Can't resolve 'path'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pgpass/lib/index.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/helper.js:4:13
Module not found: Can't resolve 'stream'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pgpass/lib/index.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/index.js:3:11
Module not found: Can't resolve 'path'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/index.js:4:9
Module not found: Can't resolve 'fs'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

> Build failed because of webpack errors

I theorize that the problem is because pg was made to use a adapter version of some code when running in workerd. But during the Next.js build this is not set, so it thinks it is running in a normal Node environment and includes the wrong code - which then would not work in edge-light which the Next.js bundler assumes for edge functions - and hence fails.

Reproduction

https://github.com/janpio/cloudflare-next-prisma/tree/plain-pg

Pages Deployment Method

Direct Upload (wrangler pages publish or the @cloudflare/pages-action GitHub Action)

Pages Deployment ID

No response

Additional Information

No response

Would you like to help?

[ ] Would you like to help fixing this bug?

VanThanh12195 commented 6 months ago

Hello is there any workaround for this issue? I try to use pg and prisma on cloudflare next-on-pages with nextjs application. Observe similar issue.

james-elicx commented 6 months ago

Hello is there any workaround for this issue? I try to use pg and prisma on cloudflare next-on-pages with nextjs application. Observe similar issue.

This build error comes from Next.js and is something we have no control over.

webdeb commented 5 months ago

@janpio / @VanThanh12195 have you found a workaround?

webdeb commented 5 months ago

@james-elicx can you provide tips what a developer can/should do fix it? next-on-pages allows to develop next inside wranger context, if its ok to call it like this. pg is supported by cloudflare, and those errors are coming from loading pg dependencies inside the wrangler context. I mean, you guys are probably dealing with it on the cloudflare side, let it make work on the dev side, or show us how, please.

james-elicx commented 5 months ago

@james-elicx can you provide tips what a developer can/should do fix it? next-on-pages allows to develop next inside wranger context, if its ok to call it like this. pg is supported by cloudflare, and those errors are coming from loading pg dependencies inside the wrangler context. I mean, you guys are probably dealing with it on the cloudflare side, let it make work on the dev side, or show us how, please.

Module not found: Can't resolve is a Next.js build error because they throw errors when libraries import Node.js builtins that aren't supported according to their edge runtime implementation. It has nothing to do with us or how our dev mode works, and we can't stop it from happening.

webdeb commented 5 months ago

Module not found: Can't resolve is a Next.js build error because they throw errors when libraries import Node.js builtins that aren't supported according to their edge runtime implementation. It has nothing to do with us or how our dev mode works, and we can't stop it from happening.

@james-elicx I understand that the build error is coming from nextjs. However, lets focus on a possible solution or a workaround to use pg in a next-on-pages project.

Or just declare that next-on-pages does not support pg and call it a day..

huhan-123 commented 4 months ago

Has anyone solved this problem?

janpio commented 4 months ago

There might be a way to make it work by configuring Next.js and patching pg. Super hacky, but it shows there is a way forward: https://github.com/brianc/node-postgres/issues/3206#issuecomment-2154419602

protonys commented 3 months ago

Not working pg connection, MariaDB connection and MySQL connection in to middleware.ts How then use middleware.ts with db connections? May be use pg connection in layout dashboard for auth?

lancerace commented 4 weeks ago

facing the same exact problem due to wanting to use cloudflare and vercel deployment with edge runtime.

Below are the summary i am facing,

The following code could work with next build and @cloudflare/next-on-pages deploy

import { Prisma, PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();
export const runtime = 'edge';

//schema.prisma
generator client {
  provider        = "prisma-client-js"
  previewFeatures = []
}

in a cloudflare preview deployed environment. If a page call an api that requires prisma, the following error occurs

Next, i replace it with prisma/adapter-pg in attempt to resolve Next edge runtime

The following code fails at next build.

import { Pool } from 'pg'
import { PrismaPg } from '@prisma/adapter-pg'
import { Prisma, PrismaClient } from '@prisma/client';
const connectionString = `${process.env.DATABASE_URL}`
const pool = new Pool({ connectionString })
const adapter = new PrismaPg(pool)
const prisma = new PrismaClient({ adapter } as any)
export const runtime = 'edge';

//schema.prisma
generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["driverAdapters"]
}

wonder if anyone resolved this

irvinebroque commented 3 weeks ago

If you're trying to get pg working — it's worth experimenting with @opennextjs/cloudflare, which lets you use a much wider set of Node.js APIs, which DB drivers like pg typically depend on. (The edge "runtime" in the Next.js compiler intentionally constrains which Node.js APIs your app can use to a much narrower set than what Cloudflare Workers supports)

https://opennext.js.org/cloudflare

maltekuehl commented 3 weeks ago

If you're trying to get pg working — it's worth experimenting with @opennextjs/cloudflare, which lets you use a much wider set of Node.js APIs, which DB drivers like pg typically depend on. (The edge "runtime" in the Next.js compiler intentionally constrains which Node.js APIs your app can use to a much narrower set than what Cloudflare Workers supports)

https://opennext.js.org/cloudflare

Has anybody yet been able to get the Cloudflare version of OpenNext to work with prisma?

The following prisma configuration works fine with next dev, builds without errors with npx cloudflare but fails when using wrangler dev or wrangler deploy:

import { Pool } from "pg";
import { PrismaPg } from "@prisma/adapter-pg";
import { PrismaClient } from "@prisma/client";

const pool = new Pool({ connectionString: connectionString, max: 1 })
const adapter = new PrismaPg(pool)
const prisma = new PrismaClient({ adapter })

Error messages:

Could not resolve "#main-entry-point"

    .worker-next/.next/standalone/node_modules/.prisma/client/default.js:1:30:
      1 │ module.exports = { ...require('#main-entry-point') }
        ╵                               ~~~~~~~~~~~~~~~~~~~

  The module "./wasm.js" was not found on the file system:

    .worker-next/.next/standalone/node_modules/.prisma/client/package.json:108:19:
      108 │         "workerd": "./wasm.js",
          ╵                    ~~~~~~~~~~~

and

ould not resolve "pg-cloudflare"

    .worker-next/.next/standalone/node_modules/pg/lib/stream.js:10:41:
      10 │     const { CloudflareSocket } = require('pg-cloudflare')
         ╵                                          ~~~~~~~~~~~~~~~

  The module "./dist/index.js" was not found on the file system:

    .worker-next/.next/standalone/node_modules/pg-cloudflare/package.json:13:15:
      13 │     "workerd": "./dist/index.js",
         ╵                ~~~~~~~~~~~~~~~~~

  You can mark the path "pg-cloudflare" as external to exclude it from the bundle, which will remove this error. You can also surround this "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.

Adding pg-cloudflare as a dependency does also not seem to help. Wondering if anybody has managed to set it up successfully.

ianthpun commented 4 days ago

having same issue as above ^

ianthpun commented 4 days ago

@maltekuehl were you able to figure this out?

@irvinebroque do you have an example of using @opennextjs/cloudflare with prisma to get it working?

karthikjn01 commented 4 days ago

Btw - not an expert by any means, but I had managed to get supabase (postgres) working with the neon driver from prisma, not sure if that's helpful, but might be worth a shot until pg is working?

ianthpun commented 4 days ago

@karthikjn01 could you give me an example of how you set it up?

karthikjn01 commented 4 days ago

Unfortunately don't have the code on hand. Have a look here: https://www.prisma.io/docs/orm/overview/databases/neon

ianthpun commented 4 days ago

@karthikjn01 i just tried it using the neon drivers and got this error

same issue as above. What does your package.json look like?

karthikjn01 commented 4 days ago

Ah yeah, I was running into that on opennext - try next-on-pages, I believe that's where it was working.

ianthpun commented 2 days ago

hi yes thank you @karthikjn01 i got it working with next-on-pages, thank you. It looks like it sstill bugged on opennext

ianthpun commented 1 day ago

I seem to have gotten it compiling and running with prisma pool, but ther seems to be approx 1 second overhead for each function call made, as I think the setting up of the PrismaClient with the adapter takes around a second each time to instantiate. @karthikjn01 have you had any of those issues? The moment i moved it back to a singleton (to test this theory) it worked fine

cloudflare / next-on-pages