Robustly watching a single file is HIGHLY nontrivial, best practices highly desired! 🙏

[egnor]

This crosses operating system versions.

A very common (perhaps the most common) use pattern for fsnotify is to watch a file (or files) and perform some update whenever a change occurs (e.g. file is saved in an editor). This seems like it should be very simple. However, there are many gotchas in fsnotify (and its interactions with underlying systems) that make this quite error prone.

1) Events need to be watched in a separate goroutine, otherwise deadlock is possible (https://github.com/howeyc/fsnotify/issues/7) (at least this one is in the FAQ).

2) It's not clear which or all of Write, Rename or Create need to be monitored (https://github.com/fsnotify/fsnotify/issues/17, https://github.com/fsnotify/fsnotify/issues/80, https://github.com/fsnotify/fsnotify/issues/255, https://github.com/fsnotify/fsnotify/issues/282, https://github.com/howeyc/fsnotify/issues/91). Probably all three, to cover various platforms and editors.

3) If reading after a Write notification, there's significant danger of reading a half-written file. Platforms often have mitigations for this, e.g. with inotify one could look for IN_CLOSE_WRITE, but this isn't exposed with fsnotify, so you just have to take care and maybe delay a bit (ugh).

4) Once a file is removed (perhaps during a remove-rename atomic update cycle), fsnotify will stop watching it, and the watch needs to be replaced (https://github.com/fsnotify/fsnotify/issues/214, https://github.com/fsnotify/fsnotify/issues/220, https://github.com/fsnotify/fsnotify/issues/238, https://github.com/fsnotify/fsnotify/issues/254) if the file may appear again. Due to the gap between removal and watch replacement, the file should also be directly checked when this happens.

5) Depending on the system, the watch may only apply to the "inode" (underlying file object), not the name, which means that parent directory renames may get missed. Some people recommend watching the entire stack of parent directories (https://github.com/fsnotify/fsnotify/issues/214#issuecomment-371818544).

6) When watching a file over time to trigger a potentially long refresh (e.g. recompile of some input), it's important to "coalesce" multiple events to avoid unnecessary updates, both because many changes will generate several events, and also because there's no need to update multiple times if the file was written multiple times during a previous update. This requires a loop select that sets a flag for the event and then triggers the recompile in the default: clause, which very few people seem to do.

7) If the file could be a symlink, behavior is uncertain and definitely undocumented (https://github.com/fsnotify/fsnotify/issues/199, https://github.com/fsnotify/fsnotify/issues/210, https://github.com/fsnotify/fsnotify/issues/227) and it may not be possible to notice if the symlink itself changes targets.

Other projects struggling with these issues: https://github.com/gohugoio/hugo/issues/3850. https://github.com/gohugoio/hugo/issues/4701, https://github.com/gohugoio/hugo/issues/5205, https://github.com/nathany/looper/issues/14, https://github.com/fatedier/frp/issues/1176, https://github.com/robgonnella/ardi/issues/55

Unfortunately, given the state of affairs, fsnotify is difficult to use even for the simplest cases. It allows platform details to leak through (different semantics of various events) but also does not offer full platform-specific affordances (precisely documented semantics, and niceties like IN_CLOSE_WRITE).

Obviously, resolving this is a lot of work given platform subtleties! But, better documentation with a robust example would be less work and go a long way to making this library usable without a near-100% probability of spawning bugs for literally any project that uses it. If nothing else, maybe we can at least catalog the pitfalls?

[nathany]

Do you have thoughts on where we could begin documenting these things?

One possibility is the website https://github.com/fsnotify/fsnotify.org -- but I would kind've like to move the hosting to fsnotify.github.io.

[egnor]

I'd say that's up to you!

I would have expected all this guidance to be built into the godoc comments (which show up at https://pkg.go.dev/github.com/fsnotify/fsnotify) in appropriate places, clarifying the specific semantics of types and functions, and giving helpful tips where appropriate. That's certainly where I landed looking for API documentation.

[nathany]

That does sound ideal.

The thing I like about the blog is that (in theory) anyone could write a post with a challenge encountered, a workaround, or a tip. It's more adhoc, but it's also easier to do.

From there we can hopefully improve the API documentation and the library itself, but it does seem like a more difficult task that should be exhaustive and vetted more carefully.

[egnor]

Well, to start with, does the list above seem right-ish to you? Any specific clarifications on the items, or other gotchas you're aware of?

[nathany]

Sorry, I haven't had a chance to go through each point in detail yet. I really appreciate having a comprehensive list like this -- but also trying to rifle through notifications for many pull requests and issues.

[lqhandsome]

In this case, you must listen to directories, not files, too bad!!!

[lqhandsome]

在这种情况下，你必须听目录，而不是文件，太糟糕了！！！

The file is via k8s ConfigMap

[horahoradev]

This list looks good to me. I definitely think we should add a disclaimer on Linux's read/write consistency/atomicitiy guarantees (or lack thereof) for 3.

[sswanv]

In this case, you must listen to directories, not files, too bad!!!

Hello, how did you solve this problem in the end?

[lqhandsome]

Forgotten

发自我的iPhone

------------------ Original ------------------ From: sswanv @.> Date: Sat,Jan 6,2024 4:51 PM To: fsnotify/fsnotify @.> Cc: lqhandsome @.>, Comment @.> Subject: Re: [fsnotify/fsnotify] Robustly watching a single file is HIGHLY nontrivial, best practices highly desired! �� (#372)

In this case, you must listen to directories, not files, too bad!!!

Hello, how did you solve this problem in the end?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[Judimax]

Greetings @arp242 I saw an interesting quote from your thread https://github.com/fsnotify/fsnotify/issues/324#issuecomment-1200151103 is there an option for the watcher to be configured to consider text editors sort of a duplicatingStrategy to handle text editors in general that does this or vscode as well