On MacOS, actions happening shortly after starting watcher are not reported

[jpek42]

Working with MacOS Monterey (12.6.1) on an M1 MacBook Pro, using efsw 1.3.0.

We notice that on MacOS, unlike on Windows, file system activity that happens shortly after calling addWatch are not reported. If we wait some time -- e.g., 5 seconds -- before the activity happens, then these events are reported.

Is there something we need/must do to have these reported correctly?

[SpartanJ]

Hi, that doesn't sound right. It's happening in the efsw-test too? Did you call watch after calling addWatch? Otherwise, can you provide a minimal example?

[jpek42]

Thanks for the comment. I haven't tried efsw-test yet, but will do that, and try to put together a simple sample. We're calling watch before addWatch.

[jpek42]

I just looked at the test program. It seems to expect that the actually tracked activity will happen by the user outside of the test program. What we are doing is actually starting the watcher, and then triggering the activity within the test program itself, which (it seems) creates an opportunity for things to happen "too quickly."

[SpartanJ]

Oh, I remember why this is not working as intended, it's very simple and it's my fault because I was too lazy to catch your use case. There's an actual 100 ms sleep if you start watching without any watch since it needs to wait until a watch is added to start processing events. I'll try to fix it soon, but if you want to avoid the issue right now you can just call watch only after calling addWatch at least once.

[SpartanJ]

I fix for at least the issue I was referring to was pushed, that should be enough to solve your issue.

[SpartanJ]

Just to clarify, you'll need to build efsw with EFSW_USE_CXX11 defined (it's defined by default if you build with the project build tool, I'll deprecate older C++ versions soon).

[jpek42]

Thanks for that. In addition to the delay issue, I'm seeing a lot of odd behavior when running on MacOS. I get some replicated events, and sometimes a different set of events from run to run.

I've attached an updated test program -- which can be built in place of efsw-test -- which is a unit test of the watcher functionality, using the catch2 testing framework (which I've included in the zip).

This sets up a test area in a temp directory, and then makes a variety of changes, with a listener hanging around, and then tries to verify that the expected events happen.

It would be great if you could run it, and see if it behaves as you'd expect. I'm not quite clear on exactly what types of events should be received for various types of activity. I'm sure this is system-dependent, which is why there is some logic to adjust expectations based on the OS and/or the watcher being used (we are comparing against boost::asio::dir_monitor.

Currently cases 5 & 6 are failing. Still trying to work out the appropriate things to watch for.

This might be useful to you. Feel free to use as you see fit. This uses std::filesystem, FWIW.

efsw-test-catch2.zip

[SpartanJ]

Hi, thanks. I'll take a look at the tests, I'm guesssing that your problems are system-dependent, since FSEvents has some issues clarified in the README of the project: FSEvents for OS X Lion and beyond in some cases will generate more actions than in reality ocurred, since fine-grained implementation of FSEvents doesn't give the order of the actions retrieved. In some cases I need to guess/approximate the order of them. FSEvents has several weird behaviors, don't spect to have the same events in all OSes, that it's not possible because every watcher implementation relies on the OS features to emmit those events, and they behave very differently.

[jpek42]

Yep, I saw those. Just trying to figure out the best way to have a unit test for what happens. I'll keep tinkering. Are you going to tag a new "release" at some point? Are you happy with the state of master as it exists now?

[SpartanJ]

Are you going to tag a new "release" at some point?

Yes, I just pushed 1.3.1 tag.

I think you'll have to unit test on each platform separately, at least for the Move event, since it will be really hard to get right because it behaves really differently on each platform. I know it sucks but it shouldn't break any use case. But that's the reason I never ended up creating a unit test. I might take your unit test as a base to start working on it.

I'll close this issue and continue looking into the unit tests. Thanks!

[jpek42]

Thank you.

[jpek42]

@SpartanJ - You mentioned about duplicate events. Do you also see cases of just different behavior? In the unit test I sent, Test 5 makes a change to a single file. Usually, this results in just one modify event for the file. But sometimes (~20%) it results in extra notifications, e.g., an add of that file and a modification of its directory.

Is that something that you've experienced?

[SpartanJ]

Yes, that's totally normal. For example, a modified event depending on the OS will generate several modified events if the file is being copied and takes some time (for big files there's no a Moved "ended" event in any OS afaik), but how many events you'll get it will totally undefined. There are many situations like that. To give another example: watch dir / recursively and moving a file from /a/1 to /b/2 will generate totally different events depending on the OS. In Linux, you should expect a Remove event in /a and an Add event in /b, and probably a modified event. But in Windows, you'll probably get a single Move event. The only way to get consistent events between OSes is by using the generic file watcher, but its performance is miles away from the "native" implementations. If you find some library that is consistent between OSes, is because they implemented a generic solution, that might be good for some situations. efsw prioritize performance over consistency. In conclusion: several event notifications behavior are undefined. It doesn't mean that usually you will get what you expect, but there are many "corner cases" that each OS implementation has.

[jpek42]

Yep, that's understood. What I'm describing is all on MacOS.

[SpartanJ]

I'll take a look later at what you're describing, I never saw that particular behavior but it doesn't surprise me.

[jpek42]

FYI, as a test, I installed FSMonitor, and had it run in parallel with the test program. When this extra activity was noted in the test, it did not show up in FSMonitor. For whatever that's worth.

[SpartanJ]

Hi, @jpek42. I was able to reproduce your issue with the test you provide me. I made a little change in the macOS implementation that at least for me fixes the inconsistency (I found a way to force more granularity in the events received by the OS). I'm currently using a modified version of your test to test the efsw, if I find a way to make these tests consistent in every OS I might add them as part of the CI if you don't mind. Thanks

[jpek42]

Hi @SpartanJ - I've gotten back around to testing with the latest code, and I'm still seeing extra "add" notifications on MacOS, just before the expected modify notifications, when making a change to a file (the "Test 5" in my test program). I see this, too, when running efsw-test, but not consistently.

This is probably OK; as you've said, it doesn't surprise you. Just wanted to let you know. Thanks, and happy new year!

[SpartanJ]

Thanks! Happy new year!