force_polling and ignore_permission_denied are not working together

[bloodbee]

Description

Hi there,

I'm using watchfiles in production code, and I've found some strange behaviour depending on whether or not the force_polling and ignore_permission_denied options are used.

I will describe all the scenarios, changing those two options when needed.

1) force_polling = False, ignore_permission_denied = False

If force_polling is not activated, and a directory containing files is copied into the watched folder, only the top directory is notified. Even with recursive option activated (by default).

When the watcher is running, in an other terminal:

➜  /tmp mkdir /TEST
➜  /tmp touch TEST/file.txt
➜  /tmp cp -r TEST watcher/

Result:

watcher: INotifyWatcher { channel: Sender { .. }, waker: Waker { inner: Waker { waker: WakerInternal { fd: File { fd: 8, path: "anon_inode:[eventfd]", read: true, write: true } } } } }
raw-event=Event { kind: Create(Folder), paths: ["/tmp/watcher/TEST"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
Change.added /tmp/watcher/TEST

➜  /tmp ls watcher/*/*                
➜  /tmp watcher/TEST/file.txt

2) force_polling = True, ignore_permission_denied = False

The scenario 1) with force_polling activated: Result:

watcher: PollWatcher { watches: Mutex { data: {"/tmp/watcher": WatchData { root: "/tmp/watcher", is_recursive: true, all_path_data: {"/tmp/watcher": PathData { mtime: 1723044824, hash: None, last_check: Instant { tv_sec: 172960, tv_nsec: 959087850 } }} }}, poisoned: false, .. }, data_builder: Mutex { data: DataBuilder { build_hasher: None, now: Instant { tv_sec: 172960, tv_nsec: 959087850 } }, poisoned: false, .. }, want_to_stop: false, message_channel: Sender { .. }, delay: Some(300ms) }
raw-event=Event { kind: Modify(Metadata(WriteTime)), paths: ["/tmp/watcher"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=2
raw-event=Event { kind: Create(Any), paths: ["/tmp/watcher/TEST"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
raw-event=Event { kind: Create(Any), paths: ["/tmp/watcher/TEST/file.txt"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
Change.added /tmp/watcher/TEST/file.txt
Change.modified /tmp/watcher
Change.added /tmp/watcher/TEST

➜  /tmp ls watcher/*/*                
➜  /tmp watcher/TEST/file.txt

First question, how to achieve this result with force_polling deactivated?

3) force_polling = True, ignore_permission_denied = True

Now the fun part, let's say that we want to copy a directory with some bad permissions into the watcher, when force_polling = True AND ignore_permission_denied = True. I copy the directory using sudo, as our users are on Windows/MAC, and they copy the directories/files via direct transfer - so they don't care about permissions:

➜  /tmp sudo chmod u-rwx TEST2   
➜  /tmp ls -la | grep TEST2        
d---rwxr-x  2 user user    4096 août   7 17:36 TEST2
➜  /tmp sudo cp -r TEST2 watcher/

The permission error is not ignored at all:

watcher: PollWatcher { watches: Mutex { data: {"/tmp/watcher": WatchData { root: "/tmp/watcher", is_recursive: true, all_path_data: {"/tmp/watcher": PathData { mtime: 1723044853, hash: None, last_check: Instant { tv_sec: 173190, tv_nsec: 820411807 } }, "/tmp/watcher/TEST": PathData { mtime: 1723044853, hash: None, last_check: Instant { tv_sec: 173190, tv_nsec: 820411807 } }, "/tmp/watcher/TEST/file.txt": PathData { mtime: 1723044853, hash: None, last_check: Instant { tv_sec: 173190, tv_nsec: 820411807 } }} }}, poisoned: false, .. }, data_builder: Mutex { data: DataBuilder { build_hasher: None, now: Instant { tv_sec: 173190, tv_nsec: 820411807 } }, poisoned: false, .. }, want_to_stop: false, message_channel: Sender { .. }, delay: Some(300ms) }
raw-event=Event { kind: Modify(Metadata(WriteTime)), paths: ["/tmp/watcher"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=2
raw-event=Event { kind: Create(Any), paths: ["/tmp/watcher/TEST2"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
Traceback (most recent call last):
  File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/watchfiles/main.py", line 253, in awatch
    raw_changes = await anyio.to_thread.run_sync(watcher.watch, debounce, step, timeout, stop_event_)
  File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
_rust_notify.WatchfilesRustInternalError: error in underlying watcher: IO error for operation on /tmp/watcher/TEST2: Permission denied (os error 13)

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/mathieu/PerfectMemory/Pmscs/nascar/src/pmsc/bin/test_watcher", line 32, in <module>
  |     asyncio.run(main())
  |   File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
  |     return loop.run_until_complete(main)
  |   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
  |     return future.result()
  |   File "/home/mathieu/PerfectMemory/Pmscs/nascar/src/pmsc/bin/test_watcher", line 27, in main
  |     await asyncio.gather(task_watcher)
  |   File "/home/mathieu/PerfectMemory/Pmscs/nascar/src/pmsc/bin/test_watcher", line 11, in run
  |     async for changes in awatch(
  |   File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/watchfiles/main.py", line 251, in awatch
  |     async with anyio.create_task_group() as tg:
  |   File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/watchfiles/main.py", line 253, in awatch
    |     raw_changes = await anyio.to_thread.run_sync(watcher.watch, debounce, step, timeout, stop_event_)
    |   File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    |     return await get_async_backend().run_sync_in_worker_thread(
    |   File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    |     return await future
    |   File "/home/mathieu/.cache/pypoetry/virtualenvs/nascar-services-DFvlMQM0-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    |     result = context.run(func, *args)
    | _rust_notify.WatchfilesRustInternalError: error in underlying watcher: IO error for operation on /tmp/watcher/TEST2: Permission denied (os error 13)
    +------------------------------------

I know this is a specific use case, but this is what we got on our end with final users... Of course, when force_polling is False, the permission error is completely ignored.

My final questions:

How can I achieve the result of having a force_polling + ignore_permission_denied that actually works? If not possible, is there a way to catch the exception, without crashing the watcher?

Example Code

import asyncio
from watchfiles import awatch

DIR_TO_WATCH = '/tmp/watcher'

async def run(stop_event, *args):

    async for changes in awatch(
        DIR_TO_WATCH,
        debug=True,
        force_polling=False,  # will change depending on explained scenarios
        ignore_permission_denied=False,  # will change depending on explained scenarios
        stop_event=stop_event
    ):
        for change in changes:
            print(change[0], change[1])

async def main():
    stop_event = asyncio.Event()

    task_watcher = asyncio.create_task(run(stop_event))
    await asyncio.gather(task_watcher)

    stop_event.set()

if __name__ == "__main__":
    asyncio.run(main())

Watchfiles Output

No response

Operating System & Architecture

Linux-5.15.0-56-generic-x86_64-with-glibc2.35

62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022

Environment

No response

Python & Watchfiles Version

python: 3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0], watchfiles: 0.22.0

Rust & Cargo Version

No response

[samuelcolvin]

should be relatively simple to fix in, PR welcome.

[bloodbee]

Downgrading to watchfiles 0.21.0 fixed the issue. Thank you.