search

tursodatabase / libsql-client-ts

TypeScript/JavaScript client API for libSQL
https://docs.turso.tech/sdk/ts/quickstart
MIT License
199 stars 30 forks source link

@libsql/client prints several module warnings in Next.js #74

Open rosszurowski opened 11 months ago

rosszurowski commented 11 months ago

When using @libsql/client with Next.js (I've only tested in the app router), the client spits out a number of warnings about resolving modules.

Here's a small sample:

- warn ./node_modules/node-fetch/lib/index.js
Module not found: Can't resolve 'encoding' in '/workspace/node_modules/node-fetch/lib'

Import trace for requested module:
./node_modules/node-fetch/lib/index.js
./node_modules/cross-fetch/dist/node-ponyfill.js
./node_modules/@libsql/client/lib-esm/http/http-driver.js
./node_modules/@libsql/client/lib-esm/http/index.js
./app/db.ts
./app/page.tsx

You can see them in this reproduction sandbox. I've also copied the full set of warnings I'm seeing on a Vercel build below:

Show warnings from Vercel build output 

[10:19:14.090] ./node_modules/.pnpm/node-fetch@2.6.12/node_modules/node-fetch/lib/index.js
[10:19:14.091] Module not found: Can't resolve 'encoding' in '/vercel/path0/node_modules/.pnpm/node-fetch@2.6.12/node_modules/node-fetch/lib'
[10:19:14.091] 
[10:19:14.091] Import trace for requested module:
[10:19:14.091] ./node_modules/.pnpm/node-fetch@2.6.12/node_modules/node-fetch/lib/index.js
[10:19:14.091] ./node_modules/.pnpm/@libsql+isomorphic-fetch@0.1.6/node_modules/@libsql/isomorphic-fetch/node.mjs
[10:19:14.091] ./node_modules/.pnpm/@libsql+hrana-client@0.5.0/node_modules/@libsql/hrana-client/lib-esm/index.js
[10:19:14.091] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/http.js
[10:19:14.097] ./src/lib/db.ts
[10:19:14.091] ./src/lib/auth.tsx
[10:19:14.092] 
[10:19:14.092] ./node_modules/.pnpm/node-fetch@2.6.12/node_modules/node-fetch/lib/index.js
[10:19:14.092] Module not found: Can't resolve 'encoding' in '/vercel/path0/node_modules/.pnpm/node-fetch@2.6.12/node_modules/node-fetch/lib'
[10:19:14.092] 
[10:19:14.092] Import trace for requested module:
[10:19:14.092] ./node_modules/.pnpm/node-fetch@2.6.12/node_modules/node-fetch/lib/index.js
[10:19:14.092] ./node_modules/.pnpm/@libsql+isomorphic-fetch@0.1.6/node_modules/@libsql/isomorphic-fetch/node.mjs
[10:19:14.092] ./node_modules/.pnpm/@libsql+hrana-client@0.5.0/node_modules/@libsql/hrana-client/lib-esm/index.js
[10:19:14.092] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/http.js
[10:19:14.097] ./src/lib/db.ts
[10:19:14.092] ./src/lib/auth.tsx
[10:19:14.092] 
[10:19:14.092] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/buffer-util.js
[10:19:14.092] Module not found: Can't resolve 'bufferutil' in '/vercel/path0/node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib'
[10:19:14.092] 
[10:19:14.092] Import trace for requested module:
[10:19:14.092] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/buffer-util.js
[10:19:14.093] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/receiver.js
[10:19:14.093] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/wrapper.mjs
[10:19:14.093] ./node_modules/.pnpm/@libsql+isomorphic-ws@0.1.3/node_modules/@libsql/isomorphic-ws/node.mjs
[10:19:14.093] ./node_modules/.pnpm/@libsql+hrana-client@0.5.0/node_modules/@libsql/hrana-client/lib-esm/index.js
[10:19:14.093] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/http.js
[10:19:14.097] ./src/lib/db.ts
[10:19:14.093] ./src/lib/auth.tsx
[10:19:14.093] 
[10:19:14.093] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/buffer-util.js
[10:19:14.093] Module not found: Can't resolve 'bufferutil' in '/vercel/path0/node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib'
[10:19:14.093] 
[10:19:14.093] Import trace for requested module:
[10:19:14.093] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/buffer-util.js
[10:19:14.093] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/receiver.js
[10:19:14.093] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/wrapper.mjs
[10:19:14.094] ./node_modules/.pnpm/@libsql+isomorphic-ws@0.1.3/node_modules/@libsql/isomorphic-ws/node.mjs
[10:19:14.094] ./node_modules/.pnpm/@libsql+hrana-client@0.5.0/node_modules/@libsql/hrana-client/lib-esm/index.js
[10:19:14.094] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/http.js
[10:19:14.097] ./src/lib/db.ts
[10:19:14.094] ./src/lib/auth.tsx
[10:19:14.094] 
[10:19:14.094] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/validation.js
[10:19:14.094] Module not found: Can't resolve 'utf-8-validate' in '/vercel/path0/node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib'
[10:19:14.094] 
[10:19:14.094] Import trace for requested module:
[10:19:14.094] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/validation.js
[10:19:14.094] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/receiver.js
[10:19:14.094] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/wrapper.mjs
[10:19:14.094] ./node_modules/.pnpm/@libsql+isomorphic-ws@0.1.3/node_modules/@libsql/isomorphic-ws/node.mjs
[10:19:14.094] ./node_modules/.pnpm/@libsql+hrana-client@0.5.0/node_modules/@libsql/hrana-client/lib-esm/index.js
[10:19:14.094] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/http.js
[10:19:14.097] ./src/lib/db.ts
[10:19:14.095] ./src/lib/auth.tsx
[10:19:14.095] 
[10:19:14.095] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/validation.js
[10:19:14.095] Module not found: Can't resolve 'utf-8-validate' in '/vercel/path0/node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib'
[10:19:14.095] 
[10:19:14.095] Import trace for requested module:
[10:19:14.095] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/validation.js
[10:19:14.095] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/lib/receiver.js
[10:19:14.095] ./node_modules/.pnpm/ws@8.13.0/node_modules/ws/wrapper.mjs
[10:19:14.095] ./node_modules/.pnpm/@libsql+isomorphic-ws@0.1.3/node_modules/@libsql/isomorphic-ws/node.mjs
[10:19:14.095] ./node_modules/.pnpm/@libsql+hrana-client@0.5.0/node_modules/@libsql/hrana-client/lib-esm/index.js
[10:19:14.095] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/http.js
[10:19:14.097] ./src/lib/db.ts
[10:19:14.095] ./src/lib/auth.tsx
[10:19:14.095] 
[10:19:14.095] ./node_modules/.pnpm/better-sqlite3@8.5.2/node_modules/better-sqlite3/lib/database.js
[10:19:14.096] Critical dependency: the request of a dependency is an expression
[10:19:14.096] 
[10:19:14.096] Import trace for requested module:
[10:19:14.096] ./node_modules/.pnpm/better-sqlite3@8.5.2/node_modules/better-sqlite3/lib/database.js
[10:19:14.096] ./node_modules/.pnpm/better-sqlite3@8.5.2/node_modules/better-sqlite3/lib/index.js
[10:19:14.096] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/sqlite3.js
[10:19:14.096] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/node.js
[10:19:14.096] ./node_modules/.pnpm/drizzle-orm@0.28.5_@libsql+client@0.3.2_@opentelemetry+api@1.4.1/node_modules/drizzle-orm/libsql/index.mjs
[10:19:14.097] ./src/lib/db.ts
[10:19:14.096] ./src/lib/auth.tsx
[10:19:14.096] 
[10:19:14.096] ./node_modules/.pnpm/better-sqlite3@8.5.2/node_modules/better-sqlite3/lib/database.js
[10:19:14.096] Critical dependency: the request of a dependency is an expression
[10:19:14.096] 
[10:19:14.096] Import trace for requested module:
[10:19:14.096] ./node_modules/.pnpm/better-sqlite3@8.5.2/node_modules/better-sqlite3/lib/database.js
[10:19:14.096] ./node_modules/.pnpm/better-sqlite3@8.5.2/node_modules/better-sqlite3/lib/index.js
[10:19:14.096] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/sqlite3.js
[10:19:14.097] ./node_modules/.pnpm/@libsql+client@0.3.2/node_modules/@libsql/client/lib-esm/node.js
[10:19:14.097] ./node_modules/.pnpm/drizzle-orm@0.28.5_@libsql+client@0.3.2_@opentelemetry+api@1.4.1/node_modules/drizzle-orm/libsql/index.mjs
[10:19:14.097] ./src/lib/db.ts
[10:19:14.097] ./src/lib/auth.tsx
[10:19:14.097]

It seems like these warnings all stem from the ws, node-fetch, and better-sqlite3 packages. I suspect it has something to do with how Webpack (which Next.js uses under the hood) resolves these modules, but I've never seen these errors for other packages.

One recommendation I might suggest for the node-fetch issues is consider using the new built-in fetch which is available since Node 18, or use undici, which is an official Node.js project, and I believe can be used to bring that fetch API to older versions.

zaingz commented 10 months ago

I am also facing the same issue. In my case I am not able to run the nextjs dev server. Here are the logs:

- wait compiling /page (client and server)...
- error ./node_modules/@libsql/client/LICENSE
Module parse failed: Unexpected token (1:4)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> MIT License
|
| Copyright (c) 2023 libSQL

Import trace for requested module:
./node_modules/@libsql/client/LICENSE
./node_modules/@libsql/ sync ^\.\/.*$
./node_modules/libsql/index.js
./node_modules/@libsql/client/lib-esm/sqlite3.js
./node_modules/@libsql/client/lib-esm/node.js
./src/lib/db/index.ts
./src/app/page.tsx
- wait compiling...
- warn ./node_modules/@neon-rs/load/dist/index.js
Critical dependency: the request of a dependency is an expression

Import trace for requested module:
./node_modules/@neon-rs/load/dist/index.js
./node_modules/libsql/index.js
./node_modules/@libsql/client/lib-esm/sqlite3.js
./node_modules/@libsql/client/lib-esm/node.js
./src/lib/db/index.ts
./src/app/page.tsx

./node_modules/node-fetch/lib/index.js
Module not found: Can't resolve 'encoding' in '/Users/zainzafar/workplace/talking-sites/node_modules/node-fetch/lib'

Import trace for requested module:
./node_modules/node-fetch/lib/index.js
./node_modules/cross-fetch/dist/node-ponyfill.js
./node_modules/cross-fetch/dist/node-polyfill.js
./node_modules/@pinecone-database/pinecone/dist/v0/index.js
./node_modules/@pinecone-database/pinecone/dist/index.js
./src/lib/db/pinecone.ts
./src/app/page.tsx

./node_modules/ws/lib/buffer-util.js
Module not found: Can't resolve 'bufferutil' in '/Users/zainzafar/workplace/talking-sites/node_modules/ws/lib'

Import trace for requested module:
./node_modules/ws/lib/buffer-util.js
./node_modules/ws/lib/websocket.js
./node_modules/ws/wrapper.mjs
./node_modules/@libsql/isomorphic-ws/node.mjs
./node_modules/@libsql/hrana-client/lib-esm/index.js
./node_modules/@libsql/client/lib-esm/http.js
./node_modules/@libsql/client/lib-esm/node.js
./src/lib/db/index.ts
./src/app/page.tsx

./node_modules/ws/lib/validation.js
Module not found: Can't resolve 'utf-8-validate' in '/Users/zainzafar/workplace/talking-sites/node_modules/ws/lib'

Import trace for requested module:
./node_modules/ws/lib/validation.js
./node_modules/ws/lib/sender.js
./node_modules/ws/wrapper.mjs
./node_modules/@libsql/isomorphic-ws/node.mjs
./node_modules/@libsql/hrana-client/lib-esm/index.js
./node_modules/@libsql/client/lib-esm/http.js
./node_modules/@libsql/client/lib-esm/node.js
./src/lib/db/index.ts
./src/app/page.tsx
K1NXZ commented 10 months ago

Same issue here.

penberg commented 10 months ago

@rosszurowski @zaingz @K1NXZ can you please test 0.3.5-pre.2 (that switched from better-sqlite3 to libsql) with the following additional configuration:

https://github.com/libsql/libsql-client-ts#using-the-library-with-nextjs

The extra config is not needed if you use a version of Next with https://github.com/vercel/next.js/pull/55266 included.

K1NXZ commented 10 months ago

@penberg

@rosszurowski @zaingz @K1NXZ can you please test 0.3.5-pre.2 (that switched from better-sqlite3 to libsql) with the following additional configuration:

https://github.com/libsql/libsql-client-ts#using-the-library-with-nextjs

The extra config is not needed if you use a version of Next with vercel/next.js#55266 included.

I just tested it with 0.3.5-pre.2 on nextjs version 13.4.20-canary.28 & 13.4.19 and get following compilation error now:

Failed to compile
./node_modules/.pnpm/@libsql+client@0.3.5-pre.2/node_modules/@libsql/client/LICENSE
Module parse failed: Unexpected token (1:4)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> MIT License
| 
| Copyright (c) 2023 libSQL

Also running it with or without the experimental flag in the next config serverComponentsExternalPackages: ["libsql"] did not change anything.

john-prutton commented 10 months ago

On @libsql/client: 0.3.4 and next: 13.4.19, with or without the Nextjs experimental flags, I get errors from not being able to find the following modules, but can still connect and use the database: encoding, bufferutil, utf-8-validate.

Here's an example stack trace of the errors:

Module not found: Can't resolve 'encoding' in 'C:\...\{workspace}\node_modules\.pnpm\node-fetch@2.7.0\node_modules\node-fetch\lib'

Import trace for requested module:
./node_modules/.pnpm/node-fetch@2.7.0/node_modules/node-fetch/lib/index.js
./node_modules/.pnpm/@libsql+isomorphic-fetch@0.1.10/node_modules/@libsql/isomorphic-fetch/node.js
./node_modules/.pnpm/@libsql+hrana-client@0.5.5/node_modules/@libsql/hrana-client/lib-esm/index.js
./node_modules/.pnpm/@libsql+client@0.3.4/node_modules/@libsql/client/lib-esm/http.js
./node_modules/.pnpm/@libsql+client@0.3.4/node_modules/@libsql/client/lib-esm/node.js
./src/lib/db/index.ts

When upgrading to @libsql/client: 0.3.5-pre.9, with or without Nextjs experimental flags, I get a more sever error and my app is unusable: 

Module parse failed: Unexpected token (1:4)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> MIT License
|
| Copyright (c) 2023 libSQL

Import trace for requested module:
./node_modules/.pnpm/@libsql+client@0.3.5-pre.9/node_modules/@libsql/client/LICENSE
./node_modules/.pnpm/node_modules/@libsql/ ./node_modules/@libsql/ ./node_modules/.pnpm/node_modules/@libsql/ sync ^\.\/.*$
./node_modules/.pnpm/libsql@0.1.14/node_modules/libsql/index.js
./node_modules/.pnpm/@libsql+client@0.3.5-pre.9/node_modules/@libsql/client/lib-esm/sqlite3.js
./node_modules/.pnpm/@libsql+client@0.3.5-pre.9/node_modules/@libsql/client/lib-esm/node.js
./src/lib/db/index.ts

Hope this helps.

Nnadivictory25 commented 10 months ago

Any solutions to this yet??

typeofweb commented 10 months ago

Unfortunately serverComponentsExternalPackages: ["libsql"], doesn't fix the issue.

linyiru commented 10 months ago

After extensive testing, I noticed that this error occurs when using pnpm. No issues when using yarn or npm. Quick fix: use yarn or npm.

If you're using a Turbo repo (monorepo) like me, add the following to your .npmrc:

public-hoist-pattern[]=*libsql*

Not entirely sure this is the correct solution, but at least it's working for me now.

john-prutton commented 10 months ago

As @linyiru has mentioned above, the error Failed to compile occurs when using pnpm. I can add that it specifically occurs when pnpm is used without --shamefully-hoist.

Here is a minimal reproduction repo: my test repo. The ReadMe contains information on how to compare the effects of npm i, pnpm i, and pnpm i --shamefully-hoist.

Here is a link to a Codesandbox for the same repo. Initially it should run npm i and successfully dev env, but I encourage you to clear node_modules by running rm -rf node_modules and first pnpm i to see the error, then clear node_modules again and run pnpm i --shamefully-hoist to see that the dev env starts up successfully.

I will probably open a new issue regarding this error.

john-prutton commented 10 months ago

As for dealing with the main purpose of this issue thread: the warnings. I recommend taking a look at what this repo is doing to solve these errors, spoiler: it involves editing the webpack config in your next.config.ts. (Source line). This stopped the warning messages for me.

webpack: (config) => {
    // config.externals is needed to resolve the following errors:
    // Module not found: Can't resolve 'bufferutil'
    // Module not found: Can't resolve 'utf-8-validate'
    config.externals.push({
        bufferutil: 'bufferutil',
        'utf-8-validate': 'utf-8-validate',
    });

    return config;
},
kevinmitch14 commented 10 months ago

Adding the following stopped the warning messages. Using "@libsql/client": "^0.3.5"

const nextConfig = {
    experimental: {
        serverComponentsExternalPackages: ["@libsql/client"]
    }
}

It looks like this will be automatically handled for you soon - https://github.com/vercel/next.js/pull/56192

mnpqraven commented 8 months ago

Adding the following stopped the warning messages. Using "@libsql/client": "^0.3.5"

const nextConfig = {
    experimental: {
        serverComponentsExternalPackages: ["@libsql/client"]
    }
}

It looks like this will be automatically handled for you soon - vercel/next.js#56192

this is still giving me errors when used directly in a server component

UPDATE: apparently this does resolves the issue, but if you somehow call libsql (or drizzle()) in a server component but incorrectly marked with use client, you will still get this module errors instead of the usual error that is along the lines of cannot use server components in files marked with use client. Granted this is not something you'll ever do but it took a lot of my time looking at my next.config files when there's just one line of use client that i forgot to delete when i'm migrating a client component into a server one.

TL;DR: If anyone still gets the error after the above solutions, double check your files and make sure components calling libsql are not marked with use client since you won't get the normal errors if you do