Help request: Loading *.proto files fails on nextjs / serverless function

[Richard87]

Hi!

I was hoping one of you could give me an advice, runnning lightning client as a serverless function fails in Nextjs, because of file not found. Do any of you have a tip/quickfix/hack to load the proto files?

Like copying them to the project and telling lighning where to find them?

I made a simple repo here to illustrate the issue: https://github.com/richard87/test-grpc

or the discussion about the root issue: https://github.com/vercel/next.js/issues/8251

And feel free to close the issue, as its probably out of scope for the project!

[alexbosworth]

I'm not sure which file is not found, if it's the proto file i'm not sure how to construct the proto object without reading the proto file

[Richard87]

Sorry for the bad message, I can't load any proto files: Error: ENOENT: no such file or directory, open '/Users/richard/Projects/test-grpc/.next/serverless/grpc/protos/autopilot.proto'

Error: ENOENT: no such file or directory, open '/Users/richard/Projects/test-grpc/.next/serverless/grpc/protos/autopilot.proto'
    at Object.openSync (node:fs:585:3)
    at Object.readFileSync (node:fs:453:35)
    at fetch (/Users/richard/Projects/test-grpc/.next/serverless/chunks/674.js:106984:34)
    at Root.load (/Users/richard/Projects/test-grpc/.next/serverless/chunks/674.js:107018:13)
    at Root.loadSync (/Users/richard/Projects/test-grpc/.next/serverless/chunks/674.js:107059:17)
    at Object.loadProtosWithOptionsSync (/Users/richard/Projects/test-grpc/.next/serverless/chunks/674.js:9922:29)
    at loadSync (/Users/richard/Projects/test-grpc/.next/serverless/chunks/674.js:9827:31)
    at module.exports (/Users/richard/Projects/test-grpc/.next/serverless/chunks/674.js:63352:42)
    at /Users/richard/Projects/test-grpc/.next/serverless/chunks/674.js:63429:24
    at Array.reduce (<anonymous>) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/Users/richard/Projects/test-grpc/.next/serverless/grpc/protos/autopilot.proto'
}

[Richard87]

nextjs /netlify/vercell dosen't pick up that the proto files are required when scanning the sourcecode and bundling the serverless-functions. I assume its because of the dynamic loading of files that seems to happen.

The previous error message is if you try to run "netlify build".

This error message is the live one from Netlify's servers:

{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "Error: ENOENT: no such file or directory, open '/var/task/nextPage/grpc/protos/autopilot.proto'",
    "trace": [
        "Runtime.UnhandledPromiseRejection: Error: ENOENT: no such file or directory, open 
        '/var/task/nextPage/grpc/protos/autopilot.proto'",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:314:20)",
        "    at processPromiseRejections (internal/process/promises.js:209:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:98:32)"
    ]
}

[bkiac]

I think the bundler does not pick up the proto files by default but you could try copying them with serverless.yml or Next.js serverless third-party integration

[Richard87]

Thanks, I have already tried to tell Netlify about the file, but the documentation is limited to say the least ;) I'll give next-serverless a try, or copy the ligning grpc code into my src code untill Nextjs can solve it properly :/

[mfbrj]

I'm facing the same issue with the code:

src/app/block/[height]/page.tsx

import lnd from "@/utils/ln";
import { getBlock } from "lightning";
import { notFound } from "next/navigation";

export default async function Post({
  params: { height },
}: {
  params: { height: number };
}) {
  const { block } = await getBlock({ lnd, height: height });

  if (!block) {
    notFound();
  }

  return block;
}

src/utils/ln.tsx

import { authenticatedLndGrpc } from "lightning";

const { lnd } = authenticatedLndGrpc({
  cert: "<base64 cert>",
  macaroon:
    "<base64 macaroon>",
  socket: "lnd-internal:10009",
});

export default lnd;

I can run:

 npm run dev

When I try to access http://localhost:3000/block/800000 I get the error:

 $ npm run dev

> learning-next@0.1.0 dev
> next dev

- ready started server on 0.0.0.0:3000, url: http://localhost:3000
- event compiled client and server successfully in 252 ms (20 modules)
- wait compiling...
- event compiled client and server successfully in 195 ms (20 modules)
- wait compiling /page (client and server)...
- event compiled client and server successfully in 3.8s (483 modules)
- wait compiling...
- event compiled successfully in 152 ms (231 modules)
- wait compiling /block/[height]/page (client and server)...
- event compiled client and server successfully in 3.8s (1228 modules)
- error Error: ENOENT: no such file or directory, open '/home/mfbrj/git/learning-next/.next/server/app/block/grpc/protos/autopilot.proto'
    at Object.openSync (node:fs:603:3)
    at Object.readFileSync (node:fs:471:35)
    at fetch (webpack-internal:///(rsc)/./node_modules/protobufjs/src/root.js:126:34)
    at Root.load (webpack-internal:///(rsc)/./node_modules/protobufjs/src/root.js:152:105)
    at Root.loadSync (webpack-internal:///(rsc)/./node_modules/protobufjs/src/root.js:183:17)
    at loadProtosWithOptionsSync (webpack-internal:///(rsc)/./node_modules/@grpc/proto-loader/build/src/util.js:67:29)
    at loadSync (webpack-internal:///(rsc)/./node_modules/@grpc/proto-loader/build/src/index.js:191:61)
    at module.exports (webpack-internal:///(rsc)/./node_modules/lightning/lnd_grpc/api_for_proto.js:16:44)
    at eval (webpack-internal:///(rsc)/./node_modules/lightning/lnd_grpc/authenticated_lnd_grpc.js:63:30)
    at Array.reduce (<anonymous>) {
  digest: undefined
}
- wait compiling /_error (client and server)...
- event compiled client and server successfully in 2.9s (1361 modules)

I now that the .proto files are all at node_modules/lightning/grpc/protos as show in the image below:

But for some reason, they are expected to be at .next/server/app/block/grpc/protos

Is there any command I must run to fix this? Or any configuration on next.config.js ?

[bkiac]

@mfbrj the problem is that Next.js does not include the proto files in the bundle by default, try this workaround:

1. install copy plugin

npm i copy-webpack-plugin

2. update next.config.js

const CopyPlugin = require('copy-webpack-plugin');

/** @type {import('next').NextConfig} */
const nextConfig = {
    webpack: (config, { isServer }) => {
        // Only run in server mode
        if (isServer) {
          config.plugins.push(
            new CopyPlugin({
              patterns: [
                { 
                  from: 'node_modules/lightning/grpc/protos', 
                  to: 'grpc/protos' 
                },
              ],
            })
          );
        }
        return config;
    },
}

module.exports = nextConfig

let me know if it helps

[mfbrj]

@bkiac I already have this code in nextConfig:

/** @type {import('next').NextConfig} */
const nextConfig = {
    webpack: function (config, options) {
        config.experiments = {
            asyncWebAssembly: true,
            syncWebAssembly: true,
            layers: true
        };
        return config;
    },
}

export default nextConfig

How could I add your code?

[bkiac]

@mfbrj try like this

const CopyPlugin = require("copy-webpack-plugin")

/** @type {import('next').NextConfig} */
const nextConfig = {
    webpack: function (config, options) {
        config.experiments = {
            asyncWebAssembly: true,
            syncWebAssembly: true,
            layers: true,
        }

        if (options.isServer) {
            config.plugins.push(
                new CopyPlugin({
                    patterns: [
                        {
                            from: "node_modules/lightning/grpc/protos",
                            to: "grpc/protos",
                        },
                    ],
                }),
            )
        }

        return config
    },
}

module.exports = nextConfig

export default instead of module.exports does not work for me, but if ESM export works for you feel free to change that, you just need to add the copy plugin in the webpack step

[dylanbathurst]

@bkiac This worked for me but I needed to change the to value to "app/api/grpc/protos" to make it work properly. Thanks!