Asynchronously adding watches blocks at `notifier.watch`

[tovrstra]

I tested out notifykit as it seems to promising alternative to watchdog (which has all sorts of issues).

Alas, I run into an issue in one of my first experiments:

#!/usr/bin/env python
import asyncio
import os

from notifykit import Notifier

async def manage_watches(notifier):
    for i in range(10):
        path = f"./subdir{i:04d}"
        if not os.path.isdir(path):
            os.mkdir(path)
        print("BEFORE", path)
        notifier.watch([path])
        print("AFTER", path)
        await asyncio.sleep(0.01)

async def watch() -> None:
    notifier = Notifier(debounce_ms=200, debug=True)
    task = asyncio.create_task(manage_watches(notifier))
    async for event in notifier:
        print(event)
    await task

if __name__ == "__main__":
    asyncio.run(watch())

This example simulates the use case of a more complex asynchrounous program, where one task is responsible for adding (and removing watches) while another task is responsible for acting upon events.

When running this example, I get the following output before the script blocks:

BEFORE ./subdir0000
watcher: INotifyWatcher { channel: Sender { .. }, waker: Waker { inner: Waker { waker: WakerInternal { fd: File { fd: 8, path: "anon_inode:[eventfd]", read: true, write: true } } } } }
AFTER ./subdir0000
BEFORE ./subdir0001

It seems that adding watches while iterating over the notifier is not allowed. Adding the watches before is not a problem, but that is not my use case. (The recursive option is also not useful for the case on which this example is based, because it has a more complicated set of rules to decide which directories to (un)watch.)

[roma-glushko]

@tovrstra hey Toon 👋

The issue makes sense to me and it would be nice to fix it in notifykit. Let me investigate what it takes to get to work.

Adding the watches before is not a problem

You are right that I have been using it by watch()ing some directories recursively so far.

[roma-glushko]

@tovrstra apart from the blocking on watch/unwatch, there might be another unpleasant thing. The notify-rs, the underlying library notifykit is based on, stops the whole watching process on registering a new watching path, add a new path to the watch list and start another watching thread:

So it's a bit intrusive I would say 😌 Not sure how helpful it is in this state for your use case, but let me know.

[tovrstra]

Thanks for explaining all this! The watch_inner in notify-rs a bit unfortunate because it seems that some events may be missed, depending on how often watches are added or removed. If I understand correctly, the watchfiles library will have the same constraint, because it is also based on notify-rs?

I'm a bit puzzled by this design choice in notify-rs. It seems plausible to me that a program that reacts to file changes can also have "watch this other directory" as a response to a file change. It would seem appropriate to have good support for such use cases.