Watcher

Quick Start

#include "wtr/watcher.hpp"
#include <iostream>
#include <string>

using namespace std;
using namespace wtr;

// The event type, and every field within it, has
// string conversions and stream operators. All
// kinds of strings -- Narrow, wide and weird ones.
// If we don't want particular formatting, we can
// json-serialize and show the event like this:
//   some_stream << event
// Here, we'll apply our own formatting.
auto show(event e) {
  cout << to<string>(e.effect_type) + ' '
        + to<string>(e.path_type)   + ' '
        + to<string>(e.path_name)
        + (e.associated ? " -> " + to<string>(e.associated->path_name) : "")
       << endl;
}

auto main() -> int {
  // Watch the current directory asynchronously,
  // calling the provided function on each event.
  auto watcher = watch(".", show);

  // Do some work. (We'll just wait for a newline.)
  getchar();

  // The watcher would close itself around here,
  // though we can check and close it ourselves.
  return watcher.close() ? 0 : 1;
}

# Sigh
PLATFORM_EXTRAS=$(test "$(uname)" = Darwin && echo '-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -framework CoreFoundation -framework CoreServices')
# Build
eval c++ -std=c++17 -Iinclude src/wtr/tiny_watcher/main.cpp -o watcher $PLATFORM_EXTRAS
# Run
./watcher

modify file /home/e-dant/dev/watcher/.git/refs/heads/next.lock
rename file /home/e-dant/dev/watcher/.git/refs/heads/next.lock -> /home/e-dant/dev/watcher/.git/refs/heads/next
create file /home/e-dant/dev/watcher/.git/HEAD.lock

Enjoy!

Tell Me More

A filesystem event watcher which is

1. Friendly

   I try to keep the 1623 lines that make up the runtime of Watcher relatively simple and the API practical:

   auto w = watch(path, [](event ev) { cout << ev; });

   wtr.watcher ~

2. Modular

   Watcher may be used as a library, a program, or both. If you aren't looking to create something with the library, no worries. Just use ours and you've got yourself a filesystem watcher which prints filesystem events as JSON. Neat. Here's how:

   # The main branch is the (latest) release branch.
   git clone https://github.com/e-dant/watcher.git && cd watcher
   # Via Nix
   nix run | grep -oE 'cmake-is-tough'
   # With the build script
   tool/build --no-build-test --no-run && cd out/this/Release # Build the release version for the host platform.
   ./wtr.watcher | grep -oE 'needle-in-a-haystack/.+"' # Use it, pipe it, whatever. (This is an .exe on Windows.)

3. Efficient

   You can watch an entire filesystem with this project. In almost all cases, we use a near-zero amount of resources and make efficient use of the cache. We regularly test that the overhead of detecting and sending an event to the user is an order of magnitude less than the filesystem operations being measured.

4. Well Tested

   We run this project through unit tests against all available sanitiziers. This code tries hard to be thread, memory, bounds, type and resource-safe. What we lack from the language, we try to make up for with testing. For some practical definition of safety, this project probably fits.

5. Dependency Minimal

   Watcher depends on the C++ Standard Library. For efficiency, we use System APIs when possible on Linux, Darwin and Windows. For testing and debugging, we use Snitch and Sanitizers.

6. Portable

   Watcher is runnable almost anywhere. The only requirement is a filesystem.

Usage

Project Content

The important pieces are the (header-only) library and the (optional) CLI program.

• Library: include/wtr/watcher.hpp. Include this to use Watcher in your project.
• Program: src/wtr/watcher/main.cpp. Build this to use Watcher from the command line.

A directory tree is in the notes below.

The Library

Copy the include directory into your project. Include watcher like this:

#include "wtr/watcher.hpp"

The event and watch headers are short and approachable. (You only ever need to include wtr/watcher.hpp.)

There are two things the user needs:

• The watch function
• The event object

watch takes a path, which is a string-like thing, and a callback, with is a function-like thing. Passing watch a character array and a lambda would work well.

Typical use looks like this:

auto watcher = watch(path, [](event ev) { cout << ev; });

watch will happily continue watching until you stop it or it hits an unrecoverable error.

The event object is used to pass information about filesystem events to the (user-supplied) callback given to watch.

The event object will contain:

• associated, another event, associated with this one, such as a renamed-to path. (This is a recursive structure.)
• path_name, which is an absolute path to the event.
• path_type, the type of path. One of:
  • dir
  • file
  • hard_link
  • sym_link
  • watcher
  • other
• effect_type, "what happened". One of:
  • rename
  • modify
  • create
  • destroy
  • owner
  • other
• effect_time, the time of the event in nanoseconds since epoch.

The watcher type is special.

Events with this type will include messages from the watcher. You may recieve error messages or important status updates, such as when it first becomes alive and when it dies.

The last event will always be a destroy event from the watcher. You can parse it like this:

bool is_last = ev.path_type == path_type::watcher
            && ev.effect_type == effect_type::destroy;

Happy hacking.

Your Project

This project tries to make it easy for you to work with filesystem events. I think good tools are easy to use. If this project is not ergonomic, file an issue.

Here is a snapshot of the output taken while preparing this commit, right before writing this paragraph.

{
  "1666393024210001000": {
    "path_name": "./watcher/.git/logs/HEAD",
    "effect_type": "modify",
    "path_type": "file"
  },
  "1666393024210026000": {
    "path_name": "./watcher/.git/logs/refs/heads/next",
    "effect_type": "modify",
    "path_type": "file"
  },
  "1666393024210032000": {
    "path_name": "./watcher/.git/refs/heads/next.lock",
    "effect_type": "create",
    "path_type": "other"
  }
}

Which is pretty cool.

A capable program is here.

Consume

This project is accessible through:

• Conan: Includes the header
• Nix: Provides isolation, determinism, includes header, cli, test and benchmark targets
• Bazel: Provides isolation, include header and cli targets
• tool/build: Includes header, cli, test and benchmark targets
• CMake: Includes header, cli, test and benchmark targets
• Just copying the header file

Conan

See the package here.

Nix

nix build # To just build
nix run # Build the default target, then run without arguments
nix run . -- / | jq # Build and run, watch the root directory, pipe it to jq
nix develop # Enter an isolated development shell with everything needed to explore this project

Bazel

bazel build cli # Build, but don't run, the cli
bazel build hdr # Ditto, for the single-header
bazel run cli # Run the cli program without arguments

tool/build

tool/build
cd out/this/Release

# watches the current directory forever
./wtr.watcher
# watches some path for 10 seconds
./wtr.watcher 'your/favorite/path' -s 10

This will take care of some platform-specifics, building the release, debug, and sanitizer variants, and running some tests.

CMake

cmake -S . -B out
cmake --build out --config Release
cd out

# watches the current directory forever
./wtr.watcher
# watches some path for 10 seconds
./wtr.watcher 'your/favorite/path' -s 10

Bugs & Limitations

Watchers on all platforms intentionally ignore modification events which only change the acess time on a file or directory.

The utility of those events was questionable.

It seemed more harmful than good. Other watchers, like Microsoft's C# watcher, ignore them by default. Some user applications rely on modification events to know when themselves to reload a file.

Better, more complete solutions exist, and these defaults might again change.

Providing a way to ignore events from a process-id, a shorthand from "this" process, and a way to specify which kinds of event sources we are interested in are good candidates for more complete solutions.

OS APIs Used

Linux

• inotify
• fanotify
• epoll
• eventfd

Darwin

• FSEvents
• dispatch

Windows

• ReadDirectoryChangesW
• IoCompletionPort

Minimum C++ Version

For the header-only library and the tiny-watcher, C++17 and up should be fine.

We might use C++20 coroutines someday.

Cache Efficiency

$ tool/gen-event/dir &
$ tool/gen-event/file &
$ valgrind --tool=cachegrind wtr.watcher ~ -s 30

I   refs:      797,368,564
I1  misses:          6,807
LLi misses:          2,799
I1  miss rate:        0.00%
LLi miss rate:        0.00%

D   refs:      338,544,669  (224,680,988 rd   + 113,863,681 wr)
D1  misses:         35,331  (     24,823 rd   +      10,508 wr)
LLd misses:         11,884  (      8,121 rd   +       3,763 wr)
D1  miss rate:         0.0% (        0.0%     +         0.0%  )

LLd miss rate:         0.0% (        0.0%     +         0.0%  )
LL refs:            42,138  (     31,630 rd   +      10,508 wr)
LL misses:          14,683  (     10,920 rd   +       3,763 wr)
LL miss rate:          0.0% (        0.0%     +         0.0%  )

Namespaces and the Directory Tree

Namespaces and symbols closely follow the directories in the devel/include folder. Inline namespaces are in directories with the - affix.

For example, wtr::watch is inside the file devel/include/wtr/watcher-/watch.hpp. The namespace watcher in wtr::watcher::watch is anonymous by this convention.

More in depth: the function ::detail::wtr::watcher::adapter::watch() is defined inside one (and only one!) of the files devel/include/detail/wtr/watcher/adapter/*/watch.hpp, where * is decided at compile-time (depending on the host's operating system).

All of the headers in devel/include are amalgamated into include/wtr/watcher.hpp and an include guard is added to the top. The include guard doesn't change with the release version. In the future, it might.

watcher
├── src
│  └── wtr
│     ├── watcher
│     │  └── main.cpp
│     └── tiny_watcher
│        └── main.cpp
├── out
├── include
│  └── wtr
│     └── watcher.hpp
└── devel
   ├── src
   │  └── wtr
   └── include
      ├── wtr
      │  ├── watcher.hpp
      │  └── watcher-
      │     ├── watch.hpp
      │     └── event.hpp
      └── detail
         └── wtr
            └── watcher
               ├── semabin.hpp
               └── adapter
                  ├── windows
                  │  └── watch.hpp
                  ├── warthog
                  │  └── watch.hpp
                  ├── linux
                  │  ├── watch.hpp
                  │  ├── sysres.hpp
                  │  ├── inotify
                  │  │  └── watch.hpp
                  │  └── fanotify
                  │     └── watch.hpp
                  └── darwin
                     └── watch.hpp

You can run tool/tree to view this tree locally.