Loader error in NextJS 13

[pmcb99]

I'm getting this error with NextJS 13, running Node 20.8:

./node_modules/@anush008/tokenizers-darwin-universal/tokenizers.darwin-universal.node Module parse failed: Unexpected character '�' (1:0) You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders (Source code omitted for this binary file) Tried configuring a loader with

module.exports = { externals: { '@anush008/tokenizers-darwin-universal': 'commonjs @anush008/tokenizers-darwin-universal' }, };

and

module.exports = { module: { rules: [ { test: /\.node$/, use: 'node-loader' } ] }, };

[Anush008]

Hey. The Rust based tokenizers library uses https://github.com/napi-rs/napi-rs for Node bindings. From the error, it appears like Webpack is having issues with bundling it.

Could you try using fastembed in a regular Node application without Webpack? To see if it works and we can narrow the problem down to a Webpack compatibility issue.

[pmcb99]

Hey. The Rust based tokenizers library uses https://github.com/napi-rs/napi-rs for Node bindings. From the error, it appears like Webpack is having issues with bundling it.

Could you try using fastembed in a regular Node application without Webpack? To see if it works and we can narrow the problem down to a Webpack compatibility issue.

Yeah I'm able to get that running so must be a webpack issue

[sorokinvj]

this next config and node-loader solved it for me

/** @type {import('next').NextConfig} */
const nextConfig = {
  webpack: (config) => {
    config.module.rules.push({
      test: /\.node$/,
      use: 'node-loader',
    });
    return config;
  },
};

export default nextConfig;

[Anush008]

Thanks for the insight @sorokinvj