Exclude node_modules folders from Spotlight indexing

[as-com]

Do you want to request a feature or report a bug? Feature

What is the current behavior? node_modules folders are indexed by Spotlight on macOS

What is the expected behavior? node_modules folders are (possibly optionally) excluded from indexing

This can be done by putting a .metadata_never_index in node_modules whenever the folder is created.

~~This may not work (it didn't work for me?) Another way is to create node_modules.noindex, and symlink node_modules to it. Alternatively, figure out a way to programmatically add to the list of Spotlight exclusions.~~ Looks like I was holding it wrong.

[burkeshartsis]

It seems like this only just became an issue when updating macOS to Mojave 🤷‍♂️

[as-com]

@burkeshartsis I thought node_modules polluting spotlight search results and high CPU usage from indexing always been an issue.

[tcrossland]

See also https://github.com/npm/npm/issues/15346

[jrz]

15346 says the "true" solution is add folder names to the index. This is pure nonsense. It should be possible to add the metadata_never_index, OR to set a file attribute. It's either this, or having a global repository and better module loading (and only have spotlight index the files once).

[chrisbull]

UPDATE .metadata_never_index doesn't work on Majave or Catalina

Found this quick script to add .metadat_never_index to your node_modules folders.

find . -type d -name "node_modules" -exec touch "{}/.metadata_never_index" \;

I added it as a bash alias, so I can use apply quickly.

alias fix-spotlight='find . -type d -name "node_modules" -exec touch "{}/.metadata_never_index" \;'

Just thought I'd share here, for anyone else that is looking. :)

[kiddyfurby]

I am sorry but will yarn consider this feature?

[dicksonhk]

Based on @chrisbull 's idea, this script does the same thing faster by not going inside any node_modules directories found (coz the whole node_modules directory had already been ignored)

find . -type d  -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \;

For myself, I run it on the home directory, so I exclude hidden directories and Library directory to speed things up

find . -type d -path './.*' -prune -o -path './Pictures*' -prune -o -path './Library*' -prune -o -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \; -print

I also use the -print option so I know which folder is affected

[justin808]

For those using @dicksonhk's answer (thanks!) on MacOS, you have to have a dot or other directory after the command "find".

Maybe the same for other linux.

I agree that yarn should automatically add this file, although this is MacOS specific. Maybe some hook is better?

[dicksonhk]

For those using @dicksonhk's answer (thanks!) on MacOS, you have to have a dot or other directory after the command "find".

Maybe the same for other linux.

I agree that yarn should automatically add this file, although this is MacOS specific. Maybe some hook is better?

@justin808 You're right! I edited my answer to add . -type d and it should work fine! Also, if you are running the command in the home directory like me you should take a look at the second command which excludes unnecessary directories :)

[joaomnb]

My Spotlight seems to ignore these .metadata_never_index files, every node_modules folder is still indexed... I'm running macOS Mojave. Anyone had the same problem?

[rolandleth]

Until this gets added — if it will — another approach would be to add a postinstall script:

"postinstall": "touch ./node_modules/.metadata_never_index"

Also, I didn't manage to make @dicksonhk's script to work (maybe because I'm using fish, dunno), but I did find another handy little script: https://github.com/Willian-Zhang/Ignore-node_modules-in-Spotlight.

[0x80]

@rolandleth I think it makes more sense to use "preinstall", so spotlight doesn't start picking up the files while the install process is ongoing.

But now I tried it is seems that postinstall and preinstall both don't add the file to my node_modules. Even if I delete the folder and run install again 🤔

[rolandleth]

There is no node_modules folder at preinstall, though, if you come from a clean slate.

Be sure to show hidden files when checking if it's there or not:

[furudean]

@rolandleth

There is no node_modules folder at preinstall, though, if you come from a clean slate.

"preinstall": "mkdir -p node_modules; touch ./node_modules/.metadata_never_index"

[rolandleth]

Ha, nice call. Sadly, the file gets deleted after yarn finishes, no idea why 🤷🏻‍♂️

[MariuzM]

Guys does adding .metadata_never_index really work? Because it's still indexing node_modules and subfolders. I've ran sudo find . -type d -path './.*' -prune -o -path './Pictures*' -prune -o -path './Library*' -prune -o -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \; -print And then I cleared my index by using

sudo mdutil -i off /
sudo rm -rf /.Spotlight*
sudo mdutil -i on /
sudo mdutil -E /

And noticed that it still indexing node folders/subfolders

[shanedg]

I didn't see any change with .metadata_never_index (caveat that I don't think I cleared my index after adding) but adding my projects folders to to the Spotlight privacy list does seem to have helped. I haven't noticed high resource usage by Spotlight processes since (Mojave and Catalina).

[joaomnb]

Same. As I had said before, .metadata_never_index does not work on my system. It didn't with macOS Mojave, and still doesn't now with Catalina.

[tsujp]

I've read that .noindex might be an option for Spotlight? Do you see any results with that @joaomnb ?

[joaomnb]

No, doesn't work for me either. Spotlight still indexes everything...

[joaomnb]

I did some research on that .noindex thing and found this.

This works in a different way. If you want the folder test/ to be ignored by Spotlight, you can rename it to test.noindex/. While this does work, renaming folders is obviously not an acceptable solution...

[nemosmithasf]

.metadata_never_index doesn't work for me either on Catalina\

Potential workaround is to put all projects in a single folder, black-list that folder via spotlight privacy settings and then create a VScode workspace outside that folder which points to the project folder internally.

[Strajk]

I've glued together solutions found here and around the internets and automated it:

• Get current Spotlight Exclusions
• Find all node_modules dirs in a specified directory
• Add new ones to Spotlight Exclusions
• Restart system service to apply changes

It's definitely not a code that I would be proud of, but it works :)

https://github.com/Strajk/setup/blob/master/programs/prevent-spotlight-from-indexing-node-modules.js

[wiegell]

I've glued together solutions found here and around the internets and automated it:

• Get current Spotlight Exclusions
• Find all node_modules dirs in a specified directory
• Add new ones to Spotlight Exclusions
• Restart system service to apply changes

It's definitely not a code that I would be proud of, but it works :)

https://github.com/Strajk/setup/blob/master/programs/prevent-spotlight-from-indexing-node-modules.js

Is this actually working for anyone?? Find candidates are not evaluated??

[Strajk]

@wiegell still working for me :) can you clarify what do you mean by not being evaluated?

I've written the script intentionally in a very simple, step-by-step style so it's easy to walk-through and debug. I recommend running it in debug mode (I use WebStorm) and checking what each line does :)

[cblims]

You are missing execSync on line 21:

const candidates = cleanArray((`find ${DIR} -type d -name 'node_modules' -prune\`))

This works for me:

const candidates = cleanArray(execSync(`find ${DIR} -type d -name 'node_modules' -prune`))

[kuncevic]

@Strajk just tried to give it to go and got an error

Candidates: 11, Existing: 0, To add: 11
child_process.js:669
    throw err;
    ^

Error: Command failed: printf "MYPWD" | sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

I am on the latest Big Sur 11.2.1

UPD:

If I just run this command sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist getting an error Error Reading File: /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

Tried to get manually to this folder, seems like no permissions to get in to folder /System/Volumes/Data/.Spotlight-V100/

[jasonbarry]

Sorry, it's not clear to me and it hasn't been brought up yet – why not just put all your JS projects into ~/Sites, and drag the Sites directory into the Spotlight privacy pane? Just exclude all your web projects from being Spotlight indexed in general since you can ⌘P in your IDE to find what you want. Then npm/yarn installs won't trigger an mds_stores reindex.

[kuncevic]

@jasonbarry ✅ done

However I wonder how to fix the error above 🤪

[jeffhuys]

Sorry, it's not clear to me and it hasn't been brought up yet – why not just put all your JS projects into ~/Sites, and drag the Sites directory into the Spotlight privacy pane? Just exclude all your web projects from being Spotlight indexed in general since you can ⌘P in your IDE to find what you want. Then npm/yarn installs won't trigger an mds_stores reindex.

You don't know how many times I've been saved by spotlight searching a snippet of code that I KNEW I had somewhere, but didn't know where. Also, webstorm is much more of an energy hog than spotlight is (after indexing, that is).

[v-kiniv]

@Strajk just tried to give it to go and got an error

Candidates: 11, Existing: 0, To add: 11
child_process.js:669
    throw err;
    ^

Error: Command failed: printf "MYPWD" | sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

I am on the latest Big Sur 11.2.1

UPD:

If I just run this command sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist getting an error Error Reading File: /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

Tried to get manually to this folder, seems like no permissions to get in to folder /System/Volumes/Data/.Spotlight-V100/

I had the same issue and resolved it by granting Full Disk Access permission to Terminal.app(or any other app you're using to run the script):

[JamesLIAOHJ]

Base on @chrisbull and @dicksonhk ， those find command is still create too mouch file on my macOS Big Sur 11.6. so I modified the command

find -E /Volumes/MacData/workspace -type d -name 'node_modules' -not -regex ".+/node_modules/.+/node_modules" -exec touch '{}/.metadata_never_index' \; -print

this command affect folders that have no /node_modules/ in the middle but /node_modules at the end.

[iloveitaly]

@JamesLIAOHJ I thought apple removed support for metadata_never_index (ref). Are you finding that this magic file works for you?

[fengerzh]

No, the only way to stop it is to add your whole project folder into "Privacy of spotlight". Evil Apple.