Webpack error thrown using nft.storage in Metaplex Web

[longfeiWan9]

When I am trying to add nft.storage in Metaplex web for PL boomcamp V0 showcase, I ran into a bug which is thrown in nft.storage/encodeBlob/packCar => ipld-car/pack function, and the error message is merge_options__WEBPACK_IMPORTED_MODULE_0__.bind is not a function.

To isolated the issue, I also tried to upload a simple new created metadata.json file via nft.storage in Metaplex web which also did not work. So my assumption is that the libraries that metaplex uses might not be compatible with what is being used in nft.storage or ipld, for example webpack, build script, etc. I have tried many solutions, also got some helps from @yusefnapora. But still have not figure out why. I was recommended to create an issue here so that JS pro from nft.storage team can look into it.

[yusefnapora]

Thanks for opening an issue @longfeiWan9. I'll try investigating some more & see if I can help narrow things down further. I was hoping this was the same issue as https://github.com/nftstorage/nft.storage/issues/876, but it looks like it's still happening when using the pre-built bundle. I'll update this issue if I find anything.

[yusefnapora]

Just updating this thread with some context from my rummaging this morning.

@longfeiWan9 has a branch of the metaplex repo that he's been working on. It is actually using webpack 5, it seems. Here's the output from trying to build the web package:

yarn run build --scope web
yarn run v1.22.17
$ lerna run build --scope web
lerna notice cli v3.22.1
lerna info versioning independent
lerna notice filter including "web"
lerna info filter [ 'web' ]
lerna info Executing command in 1 package: "yarn run build"
lerna ERR! yarn run build exited 1 in 'web'
lerna ERR! yarn run build stdout:
$ next build
info  - Loaded env from /Users/yusef/work/spikes/metaplex-webpack/metaplex/js/packages/web/.env.production
info  - Loaded env from /Users/yusef/work/spikes/metaplex-webpack/metaplex/js/packages/web/.env
info  - Using webpack 5. Reason: Enabled by default https://nextjs.org/docs/messages/webpack5
info  - Checking validity of types...
info  - Creating an optimized production build...
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

lerna ERR! yarn run build stderr:
Failed to compile.

../../node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js
Attempted import error: 'create' is not exported from 'rabin-wasm' (imported as 'create').

> Build error occurred
Error: > Build failed because of webpack errors
    at /Users/yusef/work/spikes/metaplex-webpack/metaplex/js/node_modules/next/dist/build/index.js:15:924
    at async Span.traceAsyncFn (/Users/yusef/work/spikes/metaplex-webpack/metaplex/js/node_modules/next/dist/telemetry/trace/trace.js:6:584)
error Command failed with exit code 1.

lerna ERR! yarn run build exited 1 in 'web'
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The heart of which seems to be:

../../node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js
Attempted import error: 'create' is not exported from 'rabin-wasm' (imported as 'create').

Looking into node_modules/rabin-wasm/dist, it looks like only the UMD bundle has a create method exported:

grep create *

rabin.umd.js:  var module = baseModule ? Object.create(baseModule) : {};
rabin.umd.js:          return Object.create(ctor.prototype, { [THIS]: { value: thisValue, writable: false } });
rabin.umd.js:const create = async (avg, min, max, windowSize, polynomial) => {
rabin.umd.js:    create

So my guess is that webpack is trying to pull in rabin-wasm.js instead of rabin.umd.js. But I don't know how to verify that, or how to trick it into using the UMD version... Or maybe it's possible to add create to the other bundles in the rabin-wasm package?

@hugomrdias or @vasco-santos might know more...

[yusefnapora]

BTW, this seems to be a quite different error than Longfei was seeing, but perhaps that's because I'm trying to run build instead of serve? I'll poke some more and see if I can get the original merge-options error

[yusefnapora]

Okay, I was able to trigger the merge-options error by calling the nft.storage uploader function at runtime

Setting a breakpoint just before it tries to call mergeOptions.bind shows mergeOptions is undefined:

Not at all clear why yet though...

[yusefnapora]

A bit more digging...

The error is happening in ipfs-unixfs-importer options.js, which imports the merge-options package.

When I set my breakpoint, it looks like webpack has somehow translated import mergeOptions from 'merge-options' into

var merge_options__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! @multiformats/murmur3 */
"../../node_modules/ipfs-unixfs-importer/cjs/src/index.js");

Screenshot from paused browser session:

In other words, instead of importing the merge-options package, it's importing the ipfs-unix-importer entry point, and assigning it to the merge_options__WEBPACK_IMPORTED_MODULE_0__ variable.

If you inspect that var in the console, sure enough, it has the importer generator function exported by ipfs-unix-importer, and not what you'd expect if you import merge-options:

Of course...

But we seem to be getting closer to understanding what is happening, if not why...

[yusefnapora]

@alanshaw, I was able to get webpack to spit out some debug information by adding this stats plugin and adding it to the nextjs config:

const { StatsWriterPlugin } = require('webpack-stats-plugin')

// in module.exports:
  webpack: (config) => {
    config.plugins.push(
      new StatsWriterPlugin({
        filename: 'webpack-stats.json',
        stats: {
          assets: true,
          entrypoints: true,
          chunks: true,
          modules: true,
          moduleTrace: true,
        }
      })
    );

    return config;
  }

That makes a giant (~800k lines) json file in the .next directory with details about all the modules.

Grepping through, it looks like webpack is bundling both the CJS and ESM versions of ipfs-unixfs-importer.

Here's the full entry for the cjs version as a gist: https://gist.github.com/yusefnapora/34061ab21064cab30e6bebc050a29e40

And a snippet of the main info:

{
  "type": "module",
  "moduleType": "javascript/auto",
  "layer": null,
  "size": 1346,
  "sizes": {
    "javascript": 1346
  },
  "built": false,
  "codeGenerated": false,
  "buildTimeExecuted": false,
  "cached": true,
  "identifier": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/index.js",
  "name": "../../node_modules/ipfs-unixfs-importer/cjs/src/index.js",
  "nameForCondition": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/index.js",
  "index": 3269,
  "preOrderIndex": 3269,
  "index2": 3295,
  "postOrderIndex": 3295,
  "cacheable": true,
  "optional": false,
  "orphan": false,
  "issuer": "javascript/esm|/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/esm/src/index.js",
  "issuerName": "../../node_modules/ipfs-unixfs-importer/esm/src/index.js",

"... lots of other stuff"
}

I think that the "issuer" is the module that caused this one to be imported, and what's interesting is that the issuer of the CJS version is the ESM version of the same module.

There's also a huge array of "reasons" that includes our friends merge-options and @multiformats/murmur3:

    {
      "moduleIdentifier": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "module": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "moduleName": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "resolvedModuleIdentifier": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "resolvedModule": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "type": "cjs require",
      "active": true,
      "explanation": "",
      "userRequest": "merge-options",
      "loc": "3:19-43",
      "moduleId": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "resolvedModuleId": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js"
    },
    {
      "moduleIdentifier": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "module": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "moduleName": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "resolvedModuleIdentifier": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "resolvedModule": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "type": "cjs require",
      "active": true,
      "explanation": "",
      "userRequest": "@multiformats/murmur3",
      "loc": "5:14-46",
      "moduleId": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js",
      "resolvedModuleId": "../../node_modules/ipfs-unixfs-importer/cjs/src/options.js"
    },
    {
      "moduleIdentifier": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/utils/persist.js",
      "module": "../../node_modules/ipfs-unixfs-importer/cjs/src/utils/persist.js",
      "moduleName": "../../node_modules/ipfs-unixfs-importer/cjs/src/utils/persist.js",
      "resolvedModuleIdentifier": "/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/cjs/src/utils/persist.js",
      "resolvedModule": "../../node_modules/ipfs-unixfs-importer/cjs/src/utils/persist.js",
      "type": "cjs require",
      "active": true,
      "explanation": "",
      "userRequest": "@ipld/dag-pb",
      "loc": "4:12-35",
      "moduleId": "../../node_modules/ipfs-unixfs-importer/cjs/src/utils/persist.js",
      "resolvedModuleId": "../../node_modules/ipfs-unixfs-importer/cjs/src/utils/persist.js"
    },
    {
      "moduleIdentifier": "javascript/esm|/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js",
      "module": "../../node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js",
      "moduleName": "../../node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js",
      "resolvedModuleIdentifier": "javascript/esm|/Users/yusef/work/projects/metaplex/metaplex/js/node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js",
      "resolvedModule": "../../node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js",
      "type": "harmony side effect evaluation",
      "active": true,
      "explanation": "",
      "userRequest": "rabin-wasm",
      "loc": "2:0-36",
      "moduleId": "../../node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js",
      "resolvedModuleId": "../../node_modules/ipfs-unixfs-importer/esm/src/chunker/rabin.js"
    },

It's not at all clear why the CJS module is being pulled in, but it seems like it could be the source of the issue.

[yusefnapora]

@alanshaw I made a super minimal reproduction repo for this: https://github.com/yusefnapora/nft-storage-webpack-repro

It looks like there's nothing metaplex-specific about the error. Here's the minimal dependencies that trigger the issue:

  "dependencies": {
    "next": "^11.1.3",
    "nft.storage": "^5.2.0",
    "react": "17.0.2",
    "react-dom": "17.0.2"
  },

I copied the compilerOptions for the typescript config from the metaplex project. It has target: 'es5', but module: 'esnext', which is a combination I haven't seen a lot, but idk if that's related to the issue or not.

Anyway, at least we have a place to poke at it in isolation now.

[yusefnapora]

I've figured out that the reason nft.storage works with next v12 but not v11 is due to next switching from babel to SWC. If you force next 12 to use babel by creating a .babelrc.json file, it breaks on v12 also 😞 (see the next12-babel branch on the reproduction repo).

[yusefnapora]

Actually, I think I spoke too soon with the above comment. After wiping away node_modules, the next12-babel branch works fine... So it's possible that babel is a red herring.

[alanshaw]

@yusefnapora @longfeiWan9 ok so in https://github.com/longfeiWan9/metaplex/tree/nftStorage there's a dependency on next.js in js/packages/web/package.json but also in js/package.json. When I upgraded both dependencies to v12 I no longer saw the error and was able to upload a file.

My guess is that the issue with next@11 actually exists in a dependency and the updated version is not satisfied by the v11 dependency range but the older dependency is satisfied by v12 - hence we got the older (broken) dependency when both versions are installed in the same project.

Note: I also had to upgrade next-with-less to ^2.0.4 in js/packages/web/package.json to get it to compile.

Considering this resolved.

TL;DR use next@12 and ensure next@11 is not a dependency somewhere else in your project as dependency resolution when installing will undo the fix.