Unexpected invalidation of all paths hindering pytest runs due to excessive retries by filesystem changes

[achimnol]

Describe the bug When I run pants test :: -- -m 'not integration' on a checkout of https://github.com/lablup/backend.ai on macOS, pants detects filesystem changes too frequently and restarts many pytest processes in the middle, causing resource occupation issues like overlapping of TCP port numbers calculated from execution slots, file-based locks, and etc.

You could try by yourself by just cloning the repo and run the above command. No long installation is required because the test suite spawns their own database containers with fixtures locally using Docker and Docker Compose. Just ensure that pyenv install 3.11.3 && mkdir .tmp and have Docker Compose v2 (either Docker Desktop or OrbStack).

Pants version 2.16.0rc4 (installed via scie-pants)

OS It happens on macOS Ventura 13.4 only. I'm not sure about the previous macOS versions, but potentially some internal filesystem changes in recent updates like this 107550080 issue might be related somehow (this particular issue has blocked double-click opening of most Korean-named files in 13.3's Finder!).

I have an Ubuntu 22.04 aarch64 VM on this Mac, but there is no symptoms like this with the same repository setup.

Additional info An excerpt from .pants.d/pants.log:

20:25:48.16 [INFO] handling request: `--python-repos-find-links=-['file:///Users/joongi/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/find_links/2.16.0rc4/920d0b88/index.html']
 test :: -- -m not integration`
20:26:03.15 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:03.21 [INFO] notify invalidation: cleared 7766 and dirtied 27565 nodes for all paths
20:26:03.34 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:03.36 [INFO] notify invalidation: cleared 4 and dirtied 27565 nodes for all paths
20:26:03.36 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:03.38 [INFO] notify invalidation: cleared 4 and dirtied 27565 nodes for all paths
20:26:08.07 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:08.08 [INFO] notify invalidation: cleared 7698 and dirtied 27565 nodes for all paths
20:26:08.16 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:08.16 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:08.25 [INFO] notify invalidation: cleared 4 and dirtied 27565 nodes for all paths
20:26:09.00 [INFO] Extending leases
20:26:09.07 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.08 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.26 [INFO] notify invalidation: cleared 7698 and dirtied 27565 nodes for all paths
20:26:09.26 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.34 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.34 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.36 [INFO] Done extending leases
20:26:09.39 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.40 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths

[stuhood]

If the .tmp directory is moved outside of the workspace, does the issue still reproduce?

Pants' application of file ignoring happens after the kernel has sent us events: if your containers are creating large numbers of events inside of .tmp, or otherwise triggering a kernel bug that way, then the .gitignore won't be able to help with that.

[achimnol]

It seems to improve the situation! I'll test a little bit more.

The reason to use %(buildroot)s/.tmp was to avoid private tmp issue with Snap-based Docker (as I've reported in the troubleshooting guide).

I think it would be better to update the docs to use something like %(home)s/.cache/<project-name>.

[benjyw]

Sounds like this is no longer an issue, so closing.