Watch specific paths

[trenton42]

Change

Allow paths to be watched to be specified.

Rational

While working on a project, I realized that I needed to watch directories outside the current path. Trying to exclude all other paths would be impractical, so I wrote this patch to allow only specific directories to be watched.

This change is backwards compatible with the current reflex and adds a command line switch to specify the paths that will be included.

New usage:

reflex -p ../path_to_lib -p path_to_files echo "file changed is {}"

If a path is not specified, the the current directory will be used (as with current reflex).

[cespare]

You could run reflex in the parent directory and use -r or -g to specify just the directories you want.

[trenton42]

Hmm, maybe I misunderstood the error messages I was getting when I tried that today. I have one project that is structured like so:

api
    api
        - some files to watch
    tests
        - lots of files to ignore

When I run reflex -g 'api/*.py' echo '{}' I get the error message: [info] Error while watching new path tests/: too many open files

I didn't look too closely, but is that something I can ignore?

(this is just running reflex within the project)

[trenton42]

Also, the comment in the readme:

Run reflex in the most specific directory possible.

Made me think that it might be helpful to make directory selection as specific as possible.

[cespare]

I didn't look too closely, but is that something I can ignore?

No, reflex won't pick up some of your changes if it's unable to open some files because it runs into system limits. https://github.com/cespare/reflex#open-file-limits lists some of the workarounds.

[...] Made me think that it might be helpful to make directory selection as specific as possible.

Yes, but from the perspective of how reflex works today, if you want to watch paths in the parent directory, you're being too specific :P

I'd sort of prefer not to use the solution you're proposing for a couple of reasons:

• The combination of multiple paths + globs + regexes gets very confusing for the user. (For example: what if one path contains another path? I think you'll see changes twice, under two different names. And there are lots of similarly confusing situations.)
• It's essentially adding a workaround in the tool interface for an underlying issue which could be solved in other ways
• Fewer options is better than more options

However, the fundamental idea of making more specific watch directories is definitely a good one that could help out with the 'too many files' issue. Maybe we could do this automatically. For example, if we notice that the globs/regexes all have some (mutually exclusive) file prefixes, we could make watches for those specific directories instead of watching ..

Let me think about this more.

[bradobro]

@cespare Thanks for a cool project. Thanks for open-sourcing it!

I can appreciate your desire for simplicity. I think about how much rsync includes and excludes make my brain hurt--or at least take a lot of fiddling sometimes to do what I intend.

OTOH, my hunch is that most users of this will be people who used to rely on nodemon or fswatch but switched because reflex works better.

I think @trenton42's use case will be common among backend developers when they are integration testing an API or full stack app that needs to run in a different process from the tests even though the tests work in the same tree.

FWIW, the only way I've seen for filtering file and directory inclusions that feels intuitive and reliable is .gitignore (and things that copy it). It gives up some brevity, but it's easy to parse, easy to reason about (because it only handles what's NOT watched), and gets you all the way home most of the time with just globs and slashes.

I'm going to play a bit more with stock reflex to see how far I can push it. Hope you'll keep thinking about a clean and simple.

[cespare]

I've given this quite a bit of thought, and I still think that the proposed solution adds too much user-facing complexity and new corner cases to outweigh its benefits.

I think there's a way that reflex can watch more specific paths automatically. For example, if the user passes

-g api/*.py -g tests/foo/bar/*.py

then reflex can automatically rewrite this as two separate watches:

-g *.py         in api/
-g *.py         in tests/foo/bar

The same thing can be done with -r arguments (note to self: r.LiteralPrefix helps). There's a lot of ways this code could go wrong. For example, if the user gave

-g api/*.py -g api/foo/*.py

then we need to be careful not to double-watch api/foo/*.py changes.

All that said, I'm not inclined to try to write this "smart paths" code right now. Doing it in a robust, well-tested way will require a good amount of work/refactoring to fit into reflex as it exists today, and when we're done it still might end up not being worth the implementation complexity.

---

I think that this general problem doesn't come up all that much for users, and the most common cause is that open file limits are simply too small. If your OS only allows you, say, 256 open files (as is the case on older OS X versions, I believe), you're going to run into this problem regardless.

After you've made sure that your open file limit is reasonable, if you still have the problem, the next thing to do is see if there's a giant subdirectory you can ignore. For example, if tests/ has some subdirectory with 100k files in it, you can ignore that using -R '^tests/bigdir.

A last-ditch workaround is to run two copies of reflex, one each in the directories you're interested in (perhaps using a wrapper script).

Thanks for your contribution, and please do let me know whether you're able to get this working using the techniques I've suggested.

[trenton42]

Thanks for the consideration and thought you have put into this! In the end, I was not able to get it to work without including many, many arguments. The main problem is that I have two directories that I want to watch, but one of them is in the parent directory (which also includes lots of other directories). Running reflex there requires adding a whole lot of -R flags which became far too complex.