Hot Reload doesn't work with linked modules

[ghost]

• [ check] I HAVE READ THE FAQ AND MY PROBLEM WAS NOT DESCRIBED THERE
• [check ] I WILL GIVE 10$ TO CHARITY IF MY SOLUTION WAS ACTUALLY IN THE README/FAQ

Are you trying to transpile a local package or an npm package? NPM Package

Describe the bug We have multiple modules that are linked to a main project. There are like 11 modules.

A while ago, we used npm install in the main Project and then link the Modules using npm link. Upon saving, the linking worked well. The hot reloading was working and we could see our changes live.

One month ago, after we ran npm install, we encountered an error where the hot reload wouldn't work anymore. When we link the modules, everything seems to be in order. The modules in main project are linked. Upon saving, we don't get any error, the terminal tells us "compiled successfully". Hot reload works only in the main platform now. The interesting part is that when saving in modules, the compilation happens two times and much faster than the one that works. After some days of debugging, we learned that disabling webpack 5 and using webpack 4, would help us a bit. There are still some modules that are not working (4 out of 11).

Our productivity decreased exponentially.

To Reproduce I'm not sure how to tell you. Our Packages are private.

Expected behavior After we install the platform and link all our modules, the hot reload should work no matter where you are saving.

Setup

• Next.js version: v11.1.2
• next-transpile-modules version: 8.0.0
• Node.js version: 14.17.6
• npm/yarn version: npm 6.14.15
• Operating System: macOS Catalina 10.15.7
• Webpack 4 or 5: webpack 4 (webpack 5 is where the big issues come)

Additional context Add any other context about the problem here.

next.config.js.zip

[martpie]

If you are using npm, use file:, not link:.

Also, I highly (highly!) encourage you to switch to NPM workspaces rather than links/files. It makes things much more sane.

[ghost]

Thank you so much for your reply. We will most probably switch to workspaces.

In the meantime, I'm not entirely sure how to use file: instead oflink: Could you help me out on that one? I'm not exactly sure what it means and how to convert our current setup.

[martpie]

This should be of some help :) https://github.com/martpie/next-transpile-modules/blob/1d737ac339e2023c928f30f01f79570e0363944c/src/__tests__/__apps__/npm-basic/package.json#L19-L21

[oalexdoda]

How would that work when deployed to staging / production though @martpie ? Since the shared packages are in NPM?

[martpie]

If you are transpiling packages from npm, why do you need hot reload on them?

If you want to re-use packages locally, you most probably don't want to publish them on npm but just organize a monorepo with workspaces.

Example: https://github.com/martpie/monorepo-typescript-next-the-sane-way

[oalexdoda]

There's a main platform, and 10 shared modules.

The main platform is deployed to a public URL.

The shared modules are used by the main platform, as well as other platforms.

We build both the main platform, the 10 shared (transpiled) modules, and the other platforms relying on the same modules.

Picture building something like Bootstrap, as well as the projects relying on it.

Does that make sense? @martpie

[martpie]

Do you have different repos for each platform? Are you in control of the codebases of the 10 platforms? Are you using yarn or npm?

Because this is typically the perfect usecase for a monorepo using workspaces where you'd had

• website 1
• website 2
• styleguide
• shared-utils

If you start having different repos for every platform, and you absolutely want to link those modules (again, in my experience, this only brings annoyances), you need to make sure the shared modules are symlinked and not copied to your node_modules folder, this is usually the 1st reason why HMR doesn't work.

[martpie]

Also, if you have repos for different applications, having your shared modules in one of the platform does not really make sense. And their development lifecycle should be different than the applications. So in this case, you should not need HMR.

Can you expain your setup more in details?

[oalexdoda]

Yeah, so we're in control of the codebases for the 10 shared modules, as well as the various platforms. But they're different brands, so we need to keep them separate:

• Project 1
• Project 2
• Shared Modules

Also, some of the shared modules will be available open source as well, so we have to keep them separate.

We tried npm link, as well as ln -s on macOS, and HMR still doesn't work. It used to in previous versions, so either Next.js or Next Transpile broke it, just not sure which and when.

To give some extra clarity, Project 1 could have these dependencies:

...
"dependencies": {
        "@shared/blocks": "^1.1.383",
        "@shared/core": "^0.1.222",
        "@shared/database": "^0.0.200",
        "@shared/helpers": "^1.1.163",
        "@shared/interface": "^1.1.194",
}
...

And Project 2 might only use some of them, like:

"dependencies": {
        "@shared/blocks": "^1.1.383",
        "@shared/core": "^0.1.222",
        "@shared/interface": "^1.1.194"
}

Basically a lot of the stuff we build can and should be repurposed for future projects, so we build it as separate modules.

We're only concerned with HMR in Project 1 (the main project) for now as we build up the modules.

[oalexdoda]

Also, we're using NPM.

[oalexdoda]

Could it be something related to the number of files Webpack is trying to watch? Given the number of modules, components, files, etc., there's hundreds of files being watched (if not thousands). I noticed that when reverting to Webpack 4, it throws this error a bunch of times:

Watchpack Error (watcher): Error: EMFILE: too many open files, watch

It does watch, but the error only disappears if you reset the computer.

Running launchctl limit maxfiles in the Terminal says: maxfiles 256 unlimited

[alexander-akait]

maxfiles 256 unlimited

256 files is to small

[oalexdoda]

I found a potential workaround by adding this to next config:

webpack: (config) => {
        if (process.env.NODE_ENV === 'development') {
            config.watchOptions = {
                poll: 10000,
            };
        }

        return config;
    },

Not sure if it's a stable fix or not, but works for now. Perhaps it's a deeper issue with Webpack 5 or Watchpack.

[alexander-akait]

No issues here, your system requires polling, you don't provide more context, so it hard to say why

[oalexdoda]

What additional context would you need? I broke down in detail everything above. I'll be happy to answer any questions that help identify what it is. Also, there was no error whatsoever in the console, so how would one know what pooling is in the first place?

[alexander-akait]

How you webpack? Do you use root? Can you run ulimit -aS?

[Karsens]

@altechzilla you just made my day!! @martpie If you have many projects that all rely on the same shared packages that all need to be transpiled, and you also try to run and share expo projects and next-projects together in the same workspace, it's really hard to make it work, because there are going to be many conflicting packages. At my company we've tried for days but couldn't get it to work, especially because of the invalid hook call error, but there were more things. I think @altechzilla found a really good solution for next projects. Unfortunately this isn't the full solution for me, because react native doesn't support symlinks. That's why I've built a cli that automatically tracks all your node_modules and dependencies and dependants and watches and copies changed things from the source to all destinations: it's called papapackage.

[Karsens]

I put the learnings of this and another issue into a package and published it here

It's very simple (this is the code):

const path = require("path");

const withLinksCreator = (linkableModules) => (nextConfig) => {
  return Object.assign({}, nextConfig, {
    webpack(config, options) {
      if (typeof nextConfig.webpack === "function") {
        config = Object.assign({}, nextConfig.webpack(config, options));
      }

      if (options.isServer) {
        config.externals = ["react", ...config.externals];
      }

      if (process.env.NODE_ENV === "development") {
        config.watchOptions = {
          poll: 2500,
        };
      }

      const aliases = [...linkableModules, "react", "react-dom"].reduce(
        (previous, module) => {
          return {
            ...previous,
            [module]: path.resolve(__dirname, "../..", "node_modules", module),
          };
        },
        {}
      );

      config.resolve.alias = {
        ...config.resolve.alias,
        ...aliases,
      };

      return config;
    },
  });
};

module.exports = withLinksCreator;

... but it makes it easier to also transpile and work with linked modules with fast refresh!

Hope someone finds this useful.