pcwatch is very slow when a node_modules folder is present

[mathiassoeholm]

Hi there 👋

pcwatch seems like a fantastic tool, but it is currently too slow to be of very much use for me. With verbose mode turned on, I noticed that it was traversing every single file in node_modules, and rightfully concluding that it should not apply that file to the remote.

It seems a bit weird that it has to check each file instead of just ignoring the folder completely. I have pcconfig.json with PLAYCANVAS_BAD_FOLDER_REG set to "node_modules".

If I delete my node_modules folder, it becomes significantly faster. It still seems to be traversing a bunch of files in the hidden .git directory, but it's at least usable.

Have I just misconfigured something, or is there any chance that an optimization can be made, such that it doesn't need to iterate over every file in the node_modules folder? :-)

[yaustar]

It seems a bit weird that it has to check each file instead of just ignoring the folder completely. I have pcconfig.json with PLAYCANVAS_TARGET_DIR set to "node_modules".

This sounds odd, you want to ignore the node_modules folder but you have it set to be the target directory to be watched? 😅

Can you share you pcconfig.json file and also a screenshot of the folder that you are syncing please?

[yaustar]

Oh, I'm wondering if it's traversing all the files and checking the regex filter on a per file basis 🤔 . It probably has to do that to apply the regex properly.

I guess a way around it is to set the PLAYCANVAS_TARGET_SUBDIR param to be a folder in the repo root folder so it doesn't have to look at node_modules and .git

This is what I've done with https://github.com/playcanvas/playcanvas-editor-ts-template where it builds to a sub directory and that's what playcanvas-sync watches

[mathiassoeholm]

Thanks for the fast reply! That is an odd configuration indeed, I meant PLAYCANVAS_BAD_FOLDER_REG not PLAYCANVAS_TARGET_DIR oops.

My pcconfig.json looks like this:

{
  "PLAYCANVAS_BRANCH_ID": "85605eed-1576-4bd7-a2dd-fe90cd6b0d9c",
  "PLAYCANVAS_PROJECT_ID": 955349,
  "PLAYCANVAS_BAD_FILE_REG": "^\\.|~$|package.json|jsconfig.json|tsconfig.json|.ts$|pcignore.txt|pcconfig.json",
  "PLAYCANVAS_BAD_FOLDER_REG": "node_modules"
}

I also have a global .pcconfig in my home directory like this:

{
  "PLAYCANVAS_USE_CWD_AS_TARGET": 1,
  "PLAYCANVAS_BAD_FILE_REG": "^\\.|~$",
  "PLAYCANVAS_BAD_FOLDER_REG": "\\."
}

Here's my directory structure:

Given that it's called "bad folder reg", it should be able to skip node_modules completely shouldn't it? The sub dir property sounds like an acceptable workaround, I think I'll give it a try :-)

[yaustar]

Given that it's called "bad folder reg", it should be able to skip node_modules completely shouldn't it?

I forgot about bad folder reg 🤔 @zachpeterpaul, do you have any thoughts on this?

[mathiassoeholm]

It turns out that I can greatly simplify my configuration if I use the subdir option, now all I need for my pcconfig.json is:

{
  "PLAYCANVAS_BRANCH_ID": "85605eed-1576-4bd7-a2dd-fe90cd6b0d9c",
  "PLAYCANVAS_PROJECT_ID": 955349,
  "PLAYCANVAS_TARGET_SUBDIR": "build"
}

There's no longer any bad files or folders to take into account. It's also really fast, because there's no node_modules in my build folder obviously.

You can consider the matter closed for my sake now since I'm no longer blocked, but I still think there's a possible optimization here.

Thanks for pointing me towards your TS template, it gave me the idea 👍