search

mmomtchev / node-gdal-async

Node.js bindings for GDAL (Geospatial Data Abstraction Library) with full async support
https://mmomtchev.github.io/node-gdal-async/
Apache License 2.0
136 stars 27 forks source link

Challenges Installing and Using Library in Next.js 14 #145

Open agusoler000 opened 8 months ago

agusoler000 commented 8 months ago

I'm facing multiple challenges while attempting to install and utilize a library in my Next.js 14 project. Here's a comprehensive overview of the encountered errors:

Module Resolution Errors: I'm consistently receiving "Module not found" errors for various modules such as npm, nock, and others. These modules are crucial for my project's functionality, but the errors persist despite their installation and presence in the project's dependencies.

HTML Loader Configuration: There seems to be a misconfiguration or absence of proper HTML loader configuration, leading to errors during the webpack compilation process. These errors specifically pertain to the handling of HTML files within the project.

Missing package.json Error: Additionally, there's an error indicating the absence of package.json, which should exist in the expected directory. This missing file might be contributing to the module resolution issues and further complicating the webpack compilation.

Webpack Compilation Errors: During the webpack compilation phase, I'm encountering errors related to missing modules such as npm and nock. These errors halt the compilation process and prevent the successful building of the project.

Despite attempts to reinstall dependencies, review webpack configurations, and ensure proper Next.js setup, these issues persist. I'm seeking guidance and assistance on resolving these errors and effectively integrating the library into my Next.js 14 project.

Any insights or suggestions from the community would be immensely appreciated.

Thank you for your support!

mmomtchev commented 8 months ago

What does this have to do with gdal-async? gdal-async is a Node.js-only native library without browser support?

agusoler000 commented 8 months ago

@mmomtchev I apologize for my poor wording. I'm trying to use gdal-async on the server side (next 14) to process TIFF images, but I'm unable to get it to work due to the errors I mentioned earlier. If you have any suggestions or assistance on why this might be happening, it would be greatly appreciated

mmomtchev commented 8 months ago

This is likely a problem with your configuration and not gdal-async itself. First of all, if you remove gdal-async do the errors disappear?

Post the errors you are getting.

agusoler000 commented 8 months ago

Yes, I encounter errors when I use the library, not when installing it and running the application:


import { NextResponse } from 'next/server';
import { NextApiResponse, NextApiRequest } from 'next';
import  gdal from 'gdal-async'

export  async function POST(req , res: NextApiResponse) {
  try {

  const buffer = await getBufferFromRequestBody(req);
  debugger
  const dataset = await gdal.openAsync(buffer);
  // rest of the code...  

  debugger;

  } catch (error) {
    console.error('Error processing the image:', error);
    // res.status(500).json({ error: 'Error processing the image' });
  }

}

```And the first error it gave me was:

> ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts

___

> ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:43:20
> Module not found: Can't resolve 'mock-aws-s3'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:43:20
> Module not found: Can't resolve 'mock-aws-s3'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:43:20
> Module not found: Can't resolve 'mock-aws-s3'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts

__________

> ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
> <w> [webpack.cache.PackFileCacheStrategy] Restoring pack failed from C:\Users\agust\OneDrive\Escritorio\Proyects\fase_c\avincis_fase_C\inspec-front\inspec\.next\cache\webpack\client-development-fallback.pack.gz: Error: incorrect header check
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts

It seems like it doesn't install all the necessary dependencies.

**I would like to take this opportunity to thank you for the great work you do; thanks to you, my job has been much easier. I also use RLAYERS, and it's an excellent library!**
mmomtchev commented 8 months ago

The problem is node-pre-gyp which must load the native binary. I have done this before with rollup in https://github.com/mmomtchev/xc-db/ but never with webpack. You must find a way to exclude the node-pre-gyp dependencies from your project. node-pre-gyp is the build framework that includes lots of packages that cannot be packed for the web. Once the project is built, it is used only when loading the module to find the binary path:

const binding_path = binary.find(path.resolve(path.join(__dirname, '../package.json')))
const gdal = require(binding_path)

(from lib/gdal.js)

If you hardcode the binary path as ${__dirname}/../lib/binding/node-v111-linux-x64 (check your path), you won't need node-pre-gyp.

Check those links: https://github.com/mapbox/node-pre-gyp/issues/308 https://stackoverflow.com/questions/69775419/webpack-ignore-node-pre-gyp-files

I remember that I had to improve the node-pre-gyp support in rollup for it to work: https://github.com/danielgindi/rollup-plugin-natives/pull/13

mmomtchev commented 8 months ago

@agusoler000 if you can't find a proper solution, maybe the cleanest method would be to replace the @mapbox/node-pre-gyp package with a package that implements only find() and returns the path to the .node file. Then, you will need to copy this .node file and to keep that require that loads it.

ProductOfAmerica commented 6 months ago

@mmomtchev could my error be related to the issue you described above?

⨯ TypeError: binary.find is not a function
   at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/gdal-async@3.8.5/node_modules/gdal-async/lib/gdal.js:5:29)
   at (rsc)/./node_modules/.pnpm/gdal-async@3.8.5/node_modules/gdal-async/lib/gdal.js (...\.next\server\vendor-chunks\gdal-async@3.8.5.js:59:1)
   at __webpack_require__ (...\.next\server\webpack-runtime.js:33:43)
   at eval (webpack-internal:///(rsc)/./app/(public)/api/test/route.ts:7:68)
   at (rsc)/./app/(public)/api/test/route.ts (...\.next\server\app\(public)\api\test\route.js:150:1)
   at __webpack_require__ (...\.next\server\webpack-runtime.js:33:43)
   at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/build/webpack/loaders/next-app-loader.js?name=app%2F(public)%2Fapi%2Ftest%2Froute&page=%2F(publ
ic)%2Fapi%2Ftest%2Froute&appPaths=&pagePath=private-next-app-dir%2F(public)%2Fapi%2Ftest%2Froute.ts&appDir=C%3A%5C%5Capp&pageExtensions=tsx&pageExtensions=ts&pageExtensions=jsx&pageExtensions=js&rootDir=C%3A%5C&isDev=true&tsconfigPath=tsconfig.json&basePath=&assetPrefix=&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!:15:124)
   at (rsc)/./node_modules/.pnpm/next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/build/webpack/loaders/next-app-loader.js?name=app%2F(public)%2Fapi%2Ftest%2Froute&page=%2F(public)%2Fapi%2Ftest%2Froute&a
ppPaths=&pagePath=private-next-app-dir%2F(public)%2Fapi%2Ftest%2Froute.ts&&isDev=true&tsconfigPath=tsconfig.json&basePath=&assetPrefix=&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D! (...\.next\server\app\(public)\api\test\route.js:139:1)
   at __webpack_require__ (...\.next\server\webpack-runtime.js:33:43)
   at __webpack_exec__ (...\.next\server\app\(public)\api\test\route.js:160:39)
   at ...\.next\server\app\(public)\api\test\route.js:161:430
   at __webpack_require__.X (...\.next\server\webpack-runtime.js:121:21)
   at ...\.next\server\app\(public)\api\test\route.js:161:47
   at Object.<anonymous> (...\.next\server\app\(public)\api\test\route.js:164:3)
   at Module._compile (node:internal/modules/cjs/loader:1256:14)
   at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
   at Module.load (node:internal/modules/cjs/loader:1119:32)
   at Module._load (node:internal/modules/cjs/loader:960:12)
   at Module.require (node:internal/modules/cjs/loader:1143:19)
   at mod.require (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\require-hook.js:65:28)
   at require (node:internal/modules/cjs/helpers:119:18)
   at requirePage (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\require.js:109:84)
   at ...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\load-components.js:98:84
   at async loadComponentsImpl (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\load-components.js:98:26)
   at async DevServer.findPageComponentsImpl (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\next-server.js:710:36)
   at async DevServer.findPageComponents (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\dev\next-dev-server.js:577:20)
   at async DevServer.renderPageComponent (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\base-server.js:1910:24)
   at async DevServer.renderToResponseImpl (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\base-server.js:1962:32)
   at async DevServer.pipeImpl (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\base-server.js:920:25)
   at async NextNodeServer.handleCatchallRenderRequest (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\next-server.js:272:17)
   at async DevServer.handleRequestImpl (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\base-server.js:816:17)
   at async ...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\dev\next-dev-server.js:339:20
   at async Span.traceAsyncFn (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\trace\trace.js:154:20)
   at async DevServer.handleRequest (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\dev\next-dev-server.js:336:24)
   at async invokeRender (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\lib\router-server.js:174:21)
   at async handleRequest (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\lib\router-server.js:353:24)
   at async requestHandlerImpl (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\lib\router-server.js:377:13)
   at async Server.requestListener (...\node_modules\.pnpm\next@14.2.3_@babel+core@7.24.5_react-dom@18.3.1_react@18.3.1__react@18.3.1\node_modules\next\dist\server\lib\start-server.js:141:13) {
 page: '/api/test'
}

import path from 'path';
import gdal from 'gdal-async';
import { NextResponse } from 'next/server';

export async function GET({ params }: { params: { userId: string } }): Promise<NextResponse> {
    try {
        const dataset = await gdal.openAsync(
            path.resolve('C:\\NationalCSB_2016-2023_rev23\\CSB1623.gdb'),
        );

        const layer = dataset.layers.get(0);

        console.log('number of features: ' + layer.features.count());
        console.log('fields: ' + layer.fields.getNames());
        console.log('extent: ' + JSON.stringify(layer.extent));
        console.log('srs: ' + (layer.srs ? layer.srs.toWKT() : 'null'));

        return NextResponse.json([], { status: 200 });
    } catch (e) {
        console.error(e);
        return NextResponse.json([], { status: 500 });
    }
}
ProductOfAmerica commented 6 months ago

@mmomtchev made a quick demo for you to clone: https://github.com/ProductOfAmerica/test-gdal "CSB1623.gdb" can be downloaded here I can't even get it working with plain express

mmomtchev commented 6 months ago

You are webpacking gdal, you are not supposed to do this, since gdal does not work in the browser (yet, now that there is SWIG JSE, there might be a future gdal for JavaScript version that works in the browser - but given the current state of affairs between me and the GIS community the project is unlikely in near future).

You have certainly something in your next.confg.js that makes it do this.

Try with this next.config.js, it is going to work much better:

/** @type {import('next').NextConfig} */

export default {
  webpack: (config) => {
    const externals = ['gdal-async'];
    if (config.externals)
      config.externals.push(...externals);
    else
      config.externals = externals;
    return config;
  }
};

Plain express does not webpack, so your problem must be different.

ProductOfAmerica commented 6 months ago

@mmomtchev I guess ignore the express errors. You solved my nextjs problem, so thank you!