Cannot watch specific dependencies in node_modules

SystemParadox commented 2 years ago

Describe the bug

The documentation for server.watch contains the following example:

server: {
    watch: {
        ignored: ['!**/node_modules/your-package-name/**']
    }
},

This example does not work. It appears that the builtin **/node_modules/** exclude causes chokidar to not even look in node_modules despite the negation in the subdirectory.

It appears this was originally tested (see #5023 and #5239) with ignored: ['!**/node_modules/**']. This does work, but in a real project will almost immediately result in Error: ENOSPC: System limit for number of file watchers reached.

See https://github.com/paulmillr/chokidar/issues/1225. I played with various chokidar options but I couldn't see a way to achieve this.

Reproduction

See chokidar issue.

System Info

System:
    OS: Linux 5.4 Linux Mint 20.3 (Una)
    CPU: (12) x64 AMD Ryzen 5 2600 Six-Core Processor
    Memory: 4.20 GB / 15.56 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.15.1 - /usr/bin/node
    npm: 8.1.1 - ~/npm/bin/npm
  Browsers:
    Chrome: 102.0.5005.61
    Firefox: 101.0
  npmPackages:
    vite: ^2.9.12 => 2.9.12

Used Package Manager

npm

Logs

No response

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

sapphi-red commented 2 years ago

Vite's document seems to be wrong (or there was a behavior change on chokidar side?). See #7850.

bluwy commented 2 years ago

I also wonder if the order matters:

https://github.com/vitejs/vite/blob/d402ad3ae2b724fde7702eb255af502db7ca94d4/packages/vite/src/node/server/index.ts#L284-L288

say if we move the user-specified ones at the top before **/node_modules/** 🤔

SystemParadox commented 2 years ago

@bluwy good thought but alas no, swapping the order doesn't help:

ignored: [
    '!**/node_modules/foo/**',
    '**/node_modules/**',
],

Chokidar still seems to ignore the whole of node_modules and doesn't bother looking inside it.

ryzr commented 2 years ago

@SystemParadox

If it's any help, creating a custom plugin to override the vite-enforced watch options seems to have worked for me

{
    name: 'watch-node-modules',
    configureServer: (server: ViteDevServer) : void => {
        server.watcher.options = {
            ...server.watcher.options,
            ignored: [
                /node_modules\/(?!my-package-name).*/,
                '**/.git/**',
            ]
        }
    }
}

SystemParadox commented 2 years ago

Thanks, that's very helpful as a temporary workaround until chokidar provides an official recommendation of how to fix this properly.

To preempt anyone who tries to close this:

I should not have to add a plugin to include/exclude files, especially since there is an option already for this - it just doesn't work
Negative regex lookaheads are absolutely not acceptable as a long term solution

MadLittleMods commented 2 years ago

Similar to https://github.com/vitejs/vite/issues/6718, it would be nice to exclude locally linked packages from being ignored by default. When I make a change in a sub-dependency, I want the bundle to rebuild.

VanCoding commented 1 year ago

I currently can't use vite because of this issue. I have a monorepo, where I build dependencies separately. but vite doesn't detect changes to them and there seems to be no way of making it detect them. The trick with the plugin by @bluwy didn't work for me either.

MadLittleMods commented 1 year ago

The workaround above using a custom plugin doesn't work for a plain vite.build({ watch: true }) because the configureServer hook never gets called when you're not using the dev server.

I tried using the options universal hook to replace inputOptions.watch.chokidar.ignored as desired but doesn't seem to have any effect on what is actually watched/ignored.

Custom Vite plugin (doesn't work):

{
  name: 'watch-node-modules',
  options(inputOptions) {
    inputOptions.watch.chokidar.ignored = [
      /node_modules\/(?!hydrogen-view-sdk).*/,
      '**/.git/**',
    ];
    return inputOptions;
  }
}

When using Vite, inputOptions.watch.chokidar.ignored is normally:

[
  '**/.git/**',
  '**/node_modules/**',
  '**/test-results/**',
  '/home/eric/Documents/github/element/matrix-public-archive/node_modules/.vite/**'
]

patricknelson commented 1 year ago

Just packaged up @ryzr's awesome little snippet into something reusable where you can also list multiple modules if you want 🚀

import { ViteDevServer } from 'vite';

export function pluginWatchNodeModules(modules) {
    // Merge module into pipe separated string for RegExp() below.
    let pattern = `/node_modules\\/(?!${modules.join('|')}).*/`;
    return {
        name: 'watch-node-modules',
        configureServer: (server: ViteDevServer) : void => {
            server.watcher.options = {
                ...server.watcher.options,
                ignored: [
                    new RegExp(pattern),
                    '**/.git/**',
                ]
            }
        }
    }
}

Then to use it, pass into your plugins array like so:

// Import from the separate file you might store this in, e.g. 'utils'
import { pluginWatchNodeModules } from './utils';

plugins: [
    // ... other plugins...
    pluginWatchNodeModules(['your-plugin', 'another-example']),
],

Edit: p.s. Don't forget to ensure that you exclude these packages from optimizeDeps like so:

optimizeDeps: {
    exclude: [
        'your-plugin',
        'another-example',
    ],
},

TheTedAdams commented 1 year ago

I don't know if something changed in the last month but @patricknelson snippet did not work for me. Been banging my head against this for hours but finally saw this in docs and so tried this:

import type { PluginOption } from 'vite';

export function watchNodeModules(modules: string[]): PluginOption {
  return {
    name: 'watch-node-modules',
    config() {
      return {
        server: {
          watch: {
            ignored: modules.map((m) => `!**/node_modules/${m}/**`),
          },
        },
      };
    },
  };
}

And it worked! The other "gotcha" I wanted to call out is that if you need to include a dep of your excluded dep, you need to include EXACTLY what is in your import line. For me it was react-icons and I wasted a few hours until I realized I had to include react-icons/fi/index.js because that is what was in the import line in my esm package.

quyle92 commented 1 year ago

I don't know if something changed in the last month but @patricknelson snippet did not work for me. Been banging my head against this for hours but finally saw this in docs and so tried this:
import type { PluginOption } from 'vite';

export function watchNodeModules(modules: string[]): PluginOption {
  return {
    name: 'watch-node-modules',
    config() {
      return {
        server: {
          watch: {
            ignored: modules.map((m) => `!**/node_modules/${m}/**`),
          },
        },
      };
    },
  };
}
And it worked! The other "gotcha" I wanted to call out is that if you need to include a dep of your excluded dep, you need to include EXACTLY what is in your import line. For me it was react-icons and I wasted a few hours until I realized I had to include react-icons/fi/index.js because that is what was in the import line in my esm package.

Many thanks. It works, but I need to Ctrl + S my vite.config.js (I am using react in my laravel codebase) to make my app load again to see changes in the excluded package. Is there any way that make vite automatically reload without manually saving vite.config.js again?

TheTedAdams commented 1 year ago

@quyle92 you may be running into cache stuff. Are you also listing your package in optimizeDeps.exclude? This is the final version of plugin I've been using:

import type { PluginOption } from 'vite';

export function watchNodeModules(modules: string[]): PluginOption {
  return {
    name: 'watch-node-modules',
    config() {
      return {
        server: {
          watch: {
            ignored: modules.map((m) => `!**/node_modules/${m}/**`),
          },
        },
        optimizeDeps: {
          exclude: modules,
        },
      };
    },
  };
}

quyle92 commented 1 year ago

Hi @TheTedAdams , Thanks for your reply. If I add optimizeDeps: { exclude: modules, }, vite failed to build my app and throw error at node_modules/uncontrollable/lib/esm/utils.js with message being Uncaught SyntaxError: ambiguous indirect export: default.

jfirebaugh commented 1 year ago

The issue with using optimizeDeps.exclude for this is that this also excludes deps of that package from vite's esm/cjs interop magic. So you then have to run through and include a bunch of your deps-of-deps in optimizeDeps.include to re-opt them in to esm/cjs interop. (@quyle92 that's probably the issue you're running into.)

domoritz commented 1 year ago

I added the plugin in https://github.com/vitejs/vite/issues/8619#issuecomment-1707700396 and vite updates my linked package when I save the vite config but it still doesn't update when I change the source code. I have resolve.preserveSymlinks: true enabled. Any suggestions for how I can watch a linked package?

TY-LIU commented 12 months ago

English is not my native language, and there may be grammar errors in the following content. Please understand.

I tried this method, but not work.(https://github.com/antfu/vite-plugin-restart/issues/10) This is my method. Create a new .env file for the root path.

# .env
VITE_CHANGE_KEY=anything

and use node to change this file

// generateId.js
import { readFile, writeFile } from 'fs';

readFile('./.env', 'utf-8', (err, contents) => {
  if (err) {
    console.error(err);
    process.exit(1);
    return;
  }
  contents = `VITE_CHANGE_KEY=${Math.random()}`;
  writeFile('./.env', contents, 'utf-8', (err) => {
    if (err) {
      console.log(err);
    } else {
      console.log(contents);
    }
  });
});

Because Vite will observe changes in the. env file and then re-run. (https://vitejs.dev/guide/dep-pre-bundling.html#caching) So, when your specific dependencies have changed, you can run node generateId.js. As for how to know if the dependency has changed, you can use nodemon

// nodemon.json
{
    "exec": "npm run *** && npm run generate-id", // generate-id just scripts command => 'node generateId.js'
}

It's stupid, but it works.

AttackXiaoJinJin commented 11 months ago

@quyle92 you can read package's dependencies and put them to vite optimizeDeps. include：

// vite.config.js

import path from 'path'
import {readFileSync} from 'fs'

export function watchNodeModules(modules) {
  return {
    name: 'watch-node-modules',
    config() {
      return {
        server: {
          watch: {
            ignored: modules.map((m) => `!**/node_modules/${m}/**`),
          },
        },
        optimizeDeps: {
          exclude: modules,
          include:modules.reduce((totalIncludes,m)=>{
            const url=path.join(process.cwd(), `node_modules/${m}/package.json`)
            const source = readFileSync(url, 'utf-8');
            const pkg = JSON.parse(source);
            const dependencies=pkg.dependencies
            // https://vitejs.dev/config/dep-optimization-options.html#optimizedeps-exclude
            const include=Object.keys(dependencies).reduce((includes,d)=>{
              // remove types package
              if(d.includes('@types')){
                return includes
              }
              includes.push(`${m} > ${d}`)
              return includes
            },[])
            totalIncludes=[...new Set([...totalIncludes,...include])]
            return totalIncludes
          },[]),
        },
      };
    },
  };
}

sabarnix commented 10 months ago

@quyle92 you can read package's dependencies and put them to vite optimizeDeps. include：

// vite.config.js

import path from 'path'
import {readFileSync} from 'fs'

export function watchNodeModules(modules) {
  return {
    name: 'watch-node-modules',
    config() {
      return {
        server: {
          watch: {
            ignored: modules.map((m) => `!**/node_modules/${m}/**`),
          },
        },
        optimizeDeps: {
          exclude: modules,
          include:modules.reduce((totalIncludes,m)=>{
            const url=path.join(process.cwd(), `node_modules/${m}/package.json`)
            const source = readFileSync(url, 'utf-8');
            const pkg = JSON.parse(source);
            const dependencies=pkg.dependencies
            // https://vitejs.dev/config/dep-optimization-options.html#optimizedeps-exclude
            const include=Object.keys(dependencies).reduce((includes,d)=>{
              // remove types package
              if(d.includes('@types')){
                return includes
              }
              includes.push(`${m} > ${d}`)
              return includes
            },[])
            totalIncludes=[...new Set([...totalIncludes,...include])]
            return totalIncludes
          },[]),
        },
      };
    },
  };
}

This throws error does not provide an export named 'default'

molaux commented 10 months ago

Just packaged up @ryzr's awesome little snippet into something reusable where you can also list multiple modules if you want 🚀

import { ViteDevServer } from 'vite';

export function pluginWatchNodeModules(modules) {
  // Merge module into pipe separated string for RegExp() below.
  let pattern = `/node_modules\\/(?!${modules.join('|')}).*/`;
  return {
      name: 'watch-node-modules',
      configureServer: (server: ViteDevServer) : void => {
          server.watcher.options = {
              ...server.watcher.options,
              ignored: [
                  new RegExp(pattern),
                  '**/.git/**',
              ]
          }
      }
  }
}

Then to use it, pass into your plugins array like so:

// Import from the separate file you might store this in, e.g. 'utils'
import { pluginWatchNodeModules } from './utils';

plugins: [
  // ... other plugins...
  pluginWatchNodeModules(['your-plugin', 'another-example']),
],

Edit: p.s. Don't forget to ensure that you exclude these packages from optimizeDeps like so:

optimizeDeps: {
  exclude: [
      'your-plugin',
      'another-example',
  ],
},

Was unable to get it working until I added server.watcher._userIgnored = undefined picked from chokidar source (when configureServer is called, server.watcher is already instanciated).

export function pluginWatchNodeModules (modules) {
  return {
    name: 'watch-node-modules',
    configureServer: (server) => {
      const regexp = `/node_modules\\/(?!${modules.join('|')}).*/`
        server.watcher.options = {
          ...server.watcher.options,
          ignored: [
            '**/.git/**',
            '**/test-results/**',
            new RegExp(regexp)
          ]
        }
        server.watcher._userIgnored = undefined
    },
    config () {
      return {
        optimizeDeps: {
          exclude: modules
       }
      }
    }
  }
}

Kolobok12309 commented 8 months ago

Mb anyone worked with nuxt 3 Fix of @AttackXiaoJinJin work with small changes, but SSR updated only one time, to fix this we add to nuxt.config.ts

{
...
  nitro: {
    devServer: {
      watch: [...absoluteResolvedPackagePaths]
    }
  }
...
}

acupofspirt commented 8 months ago

Did work for me:

function watchPackages(packageNames) {
  let isWatching = false;

  return {
    name: 'vite-plugin-watch-packages',

    buildStart() {
      if (!isWatching) {
        packageNames.forEach((packageName) => {
          const absPackagePath = path.resolve('node_modules', packageName);
          const realPackagePath = fs.realpathSync(absPackagePath);

          this.addWatchFile(realPackagePath);
        });

        isWatching = true;
      }
    },
  };
}

// in vite.config.js
{
  plugins: [watchPackages(['dayjs', 'foo/bar', '@some-scoped-package/utils'])]
}

But it works only for build --watch mode. Linked packages (npm link/npm i <../../your/fs/path>) will work too.

domoritz commented 8 months ago

Thanks for sharing @acupofspirt! This still doesn't seem to work for me in https://github.com/vega/editor if I link https://github.com/vega/vega-lite. I run yarn watch in Vega-Lite and make a change in the code but the editor does not refresh. Any idea what might be wrong?

acupofspirt commented 8 months ago

@domoritz It will work only for build --watch. I've updated my comment. Sorry for the confusion 😔

domoritz commented 8 months ago

Ah bummer, I really need it with vite serve.

HugoMcPhee commented 7 months ago

If it helps, it works with npx vite without any custom scripts when re-saving the vite config file while force is true

in the config: optimizeDeps: { force: true }, or if the command is started with: --force

yzy415 commented 6 months ago

6718

Hi @acupofspirt,

I tried using this plugin. It's great and worked well for the files inside the src or the root but not in the module graph.

But it did not work for the files inside the node_modules.

I did use the build --watch mode. I also tried to print the path the files it added, and the path to the files were valid.

May I know your vite version?

Thanks!

yzy415 commented 6 months ago

6718

Hi @acupofspirt,

I tried using this plugin. It's great and worked well for the files inside the src or the root but not in the module graph.

But it did not work for the files inside the node_modules.

I did use the build --watch mode. I also tried to print the path the files it added, and the path to the files were valid.

May I know your vite version?

Thanks!

Any file of other projects inside the node_modules can't be watched expectedly either. It's weird.

JiaJiaJiang commented 6 months ago

Hi, is there anyone still care about it? ALL workarounds above not work for me, that's quite irritating... I am spending the whole night for this thing that shouldn't happen at all.

domoritz commented 6 months ago

Hi, is there anyone still care about it?

✋ me for sure. It's been bugging my team for a while.

maylow22 commented 6 months ago

I dont use command "vite", which does not reflect changes in node_modules correctly. Instead I use these scripts:

 "preview": "vite preview",
 "build-watch": "vite build --watch",
 "dev": "concurrently \"npm:build-watch\" \"npm:preview\""

This checks for all changes (even in node_modules) and restarts the dev server. It is triggered by changes in my linked package without problem.

domoritz commented 6 months ago

Oh nice. How fast is that?

I switched to parcel which seems to work with watching deps if I set --watch-dir ...

domoritz commented 6 months ago

Hmm, I tried your suggestion @maylow22 but vite doesn't seem to rebuild when I change the sources in my linked dependency. Do you have an example repo I can try or a specific config/plugin?

maylow22 commented 6 months ago

It does not listen for changes in your source code, but rather in the built package. Therefore, I have a watcher script in a linked package that rebuilds the linked package (shared library) whenever a change in the source code occurs:

 "watch": "vite build --watch"

After this watcher finishes its build, another watcher in the main app triggers a build in the main app. I don't claim it's a perfect solution, but it works. It is slower than restarting a development server, and since it performs a production build, it might not be ideal.

wangrongding commented 5 months ago

Waiting for a best practice.🤖

SystemParadox commented 4 months ago

Chokidar v4 is going to drop support for globs altogether (see https://github.com/paulmillr/chokidar/issues/1225). So this is now firmly in vite's arena. Vite now has the opportunity to show how this can be done right and hopefully other tools will follow vite's example.

As far as I can work out, for any kind of include/exclude system to avoid this problem you need 3 options:

include
exclude
forceInclude

This works pretty much as it does now, exclude would default to node_modules/**, and include defaults to **. But the user can also specify forceInclude to override things that are excluded.

Different naming or other combinations of 3 options may be possible (I considered defaultExclude, include and exclude but that doesn't work so well), but you need minimum 3 or you just end up with this problem where you can't include things within node_modules sanely.

Obviously there are more complicated options which would also work, such as an ordered list of entries with type: include/exclude against each of them.

dominikg commented 4 months ago

The workaround with patching watcher options in configureServer can be improved a little by replacing the existing node_modules ignore, so that other custom ignores are preserved:

function pluginWatchNodeModules(modules) {
    let pattern = `/node_modules\\/(?!${modules.join('|')}).*/`;
    return {
        name: 'watch-node-modules',
        configureServer: (server: ViteDevServer) : void => {
            const node_modules_i = server.watcher.options.ignored.indexOf('**/node_modules/**');
            server.watcher.options.ignored.splice(node_modules_i,1,new RegExp(pattern));
            server.watcher._userIgnored=undefined;
        },
    }
}

This does give you file change events for files in the modules listed and triggers vite hmr, however plugins handleHotUpdate hook can get called with an incomplete context if you are using pnpm due to a mismatch between the symlink and real path of the module inside node_modules.

For that to work, vite needs to update https://github.com/vitejs/vite/blob/b55c32f7e36ee7cc3754a5d667785d066dece10a/packages/vite/src/node/server/hmr.ts#L120 to use the real path of file before finding affected modules.

lizyChy0329 commented 3 months ago

If you are creating with UI in vite, maybe you want vite build -w --mode lib to the lib directory and debug from vite CLI

Then you will frustrated to find that the page doesn't find the latest lib! But you can try it as follows:

add --force flag for scripts-dev in package.json
add optimizeDeps.exclude = [<your-package>] in vite.config.ts
add server.watch.ignored= ['!**/dist/**'] in vite.config.ts

It's work on my case

example

zhangHongEn commented 3 months ago

@SystemParadox

If it's any help, creating a custom plugin to override the vite-enforced watch options seems to have worked for me

{
    name: 'watch-node-modules',
    configureServer: (server: ViteDevServer) : void => {
        server.watcher.options = {
            ...server.watcher.options,
            ignored: [
                /node_modules\/(?!my-package-name).*/,
                '**/.git/**',
            ]
        }
    }
}

MAC is successful, Windows is invalid Does vite accept pull requests to fix this issue first?

xenobytezero commented 1 month ago

Tried everything on this page and could not get a dependency (workspace: or portal: in Yarn) to correctly HMR, or even refresh the cache to take changes in the dependency. Going to have to move away from the dev server to a much worse dummy Express server and do a vite build --watch. Hopefully we can get a feature added for this soon.

Has anyone got this working with a scoped package (@myscope/package)?

vitejs / vite