Symlinked source directories are wonky

[zjturner]

If I have a folder symlinked in my source directory, it doesn’t seem to pick up changes underneath it correctly.

for example, if I change a buck file inside the symlinked folder it doesn’t always detect it. If i retarget the symlink it doesn’t always detect that either.

The way to fix it is to restart the daemon, but often when doing that it triggers a full rebuild, like the internal state got messed up and it wasn’t correctly serializing whatever it uses to keep track of what’s up to date and what’s not.

tested on macOS, not sure if that makes a difference

[ndmitchell]

Is the destination of the symlink still in the Buck2 project? Are you using Watchman or INotify as your file watcher? (The default in open source is INotify). Originally we didn't have great support for symlinks, but it has improved over recent months, but I'm not sure what we expect it to be able to do (and if we expect that on all platforms or not)

[zjturner]

Using the default. How do you change to watchman? Since watchman isn’t the default Im guessing there’s some limitations with it?

[ndmitchell]

The limitation with watchman is that you have to install it on your machine. Inside Meta we use Watchman exclusively - it's much better, but requires setup, so we default to no-Watchman to give a better out of the box experience. Set[buck2] file_watcher=watchman in the .buckconfig to use it (and restart your Buck2 daemon).

[zjturner]

There's actually an even worse problem here. Symlinked source directories are not traversed by buck2 targets. They also aren't traversed by bxl when you call ctx.configured_targets() or ctx.unconfigured_targets(). I don't think this would be fixed by watchman, as the first thing you see on the watchman home page is that it doesn’t follow symlinks, and there’s a github issue that’s been open for around 8 years requesting support.

Symlinks are a pretty important requirement for us because they're how we manage all of our 3rd party dependencies. We run a command before the build that syncs all 3rd party dependencies into a package cache, and we link from the source tree into the package cache. It's going to be a very big performance penalty for us if we have to copy the package cache directory into each source directory.

Also, forgot to answer your earlier question. The link target is outside of the source tree.

im pretty sure inotify at least has the capability to follow symlinks, but maybe it just isn’t right now. It sounds like fixing that in the runtime might be our best option, which also means we won’t be able to use watchman.