New Watchdog Reloader Is Not Detecting Changes

[jquacinella]

Hello all,

I just upgraded to Werkzeug 0.10, after which I install watchdog to get the new inotify reloader working to help reduce CPU usage on my laptop. However, the reloader no longer works (or at am minimum is random in what it detects).

To start debugging, I tried to run the sample watchdog code on their homepage (https://pypi.python.org/pypi/watchdog) in an iPython shell and initially got an error. It related to not being able to allocate inotify watches due to a system constraint. After upping the limit (echo "131072" >> /proc/sys/fs/inotify/max_user_watches), I was able to run the code in the shell.

However, after that, the reloader still does not work. I am not sure what else I can do to help debug this, as watchdog seems to import correctly and my app does not throw any errors. Some diagnostic details:

% lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.1 LTS Release: 14.04 Codename: trusty

Versions: Werkzeug==0.10.1 watchdog==0.8.3 Flask==0.10.1

Please let me know if you need further information, I would be glad to debug.

[joostdevries]

@jquacinella are you using Vagrant or Virtualbox?

[jquacinella]

Ah, should have mentioned that I am using a virtualenv for the application (but this is running directly on a linux laptop, no VM). I rebuilt the virtualenv from my requirements file, and still the same issue of not detecting changes.

[jperras]

@jquacinella Was watchdog working before you upgraded to 0.10? It's not clear from your initial comment if you had it installed before you upgraded.

[jquacinella]

I did not have it installed prior. I installed watchdog when searching for why Flask takes up so much CPU. This was evidently due to the stat reloader, and it was advised that as of 0.10, that I should install watchdog. I installed it, only to find what I reported above, that the reloader essentially does not work.

[jquacinella]

Any update on this? The reloader was super conveinent for developing, and the stat reloader kills my laptop. Any ideas on what I can do to help debug the issue?

[untitaker]

Please see https://github.com/mitsuhiko/werkzeug/issues/682 regarding the performance problems of the stat-reloader.

Regarding watchdog not working: Which filesystem are you using? Does the inotifywait work correctly for you? (it's not required by Werkzeug, just a useful utility for testing fs events)

[jquacinella]

After installing the right packages on ubuntu, yes, inotifywait does work:

$ inotifywait ./ Setting up watches. Watches established. ./ OPEN,ISDIR

So it seems like its able to get at least one event. However, using inotify reloader still does not work:

-> % ./run-local.sh

• Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
• Restarting with inotify reloader

The script will stay there and never reload, despite making changes to files in the directory. Filesystem is EXT4.

[Humphreybas]

Same issue here (system with ubuntu 14.04, same flask/werkzeug/watchdog versions as the topic starter).

The only change I did see detection on is the following:

  * Detected change in '/usr/local/lib/python3.4/dist-packages/flask_bower/__init__.py', reloading
  * Restarting with inotify reloader

My app.py however does not trigger a restart.

[untitaker]

@Humphreybas @jquacinella Could you try https://github.com/mitsuhiko/werkzeug/pull/721:

pip install git+https://github.com/untitaker/werkzeug.git@frozen-reloader

[jquacinella]

I used pip to install your version of werkzeug, and can confirm from a shell that the verion comes up as:

'version': '0.11.dev0',

Restarting the Flask application still doesn't reload files while in inotify mode.

[Humphreybas]

Sorry, my contribution is quite useless, I can no longer reproduce the problem. However I cannot pinpoint what has changed. I restarted my laptop in the meantime, I did some regular ubuntu updates, but I cannot tell you why it works now..

[untitaker]

@jquacinella How does it not reload? Do you still get the "restarting with"-message or does it not react at all (important distinction)?

[untitaker]

@Humphreybas Perhaps because a new Werkzeug version has been released? Although it shoudn't have affected the inotify/watchdog reloader.

[jquacinella]

This is my output:

-> % ./run-local.sh

• Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
• Restarting with inotify reloader

If I make any changes to files, I see no more output at all.

[untitaker]

To clarify, you never see a "detected change" line?

[jquacinella]

Correct

[ThiefMaster]

There seems to be an issue with how some editors/IDEs handle writing files. When I edit a monitored file using vim the reloader picks up the change fine, but when I do so via PyCharm (which accesses the file via SMB), it doesn't.

A simple pyinotify-based script to dump events shows these events when modifying a file using vim (I hope I didn't remove any relevant events from the log):

<Event cookie=3331293 dir=False mask=0x40 maskname=IN_MOVED_FROM name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event cookie=3331293 dir=False mask=0x80 maskname=IN_MOVED_TO name=example.py~ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py~ src_pathname=example/example.py wd=1 >
<Event dir=False mask=0x100 maskname=IN_CREATE name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x20 maskname=IN_OPEN name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x2 maskname=IN_MODIFY name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x8 maskname=IN_CLOSE_WRITE name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x4 maskname=IN_ATTRIB name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x200 maskname=IN_DELETE name=example.py~ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py~ wd=1 >
<Event dir=False mask=0x20 maskname=IN_OPEN name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x1 maskname=IN_ACCESS name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x10 maskname=IN_CLOSE_NOWRITE name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x20 maskname=IN_OPEN name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x1 maskname=IN_ACCESS name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event dir=False mask=0x10 maskname=IN_CLOSE_NOWRITE name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >

When saving the same file in PyCharm I get this log:

<Event dir=False mask=0x100 maskname=IN_CREATE name=example.py___jb_bak___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_bak___ wd=1 >
<Event dir=False mask=0x20 maskname=IN_OPEN name=example.py___jb_bak___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_bak___ wd=1 >
<Event dir=False mask=0x2 maskname=IN_MODIFY name=example.py___jb_bak___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_bak___ wd=1 >
<Event dir=False mask=0x8 maskname=IN_CLOSE_WRITE name=example.py___jb_bak___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_bak___ wd=1 >
<Event dir=False mask=0x4 maskname=IN_ATTRIB name=example.py___jb_bak___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_bak___ wd=1 >
<Event dir=False mask=0x10 maskname=IN_CLOSE_NOWRITE name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event cookie=3331342 dir=False mask=0x40 maskname=IN_MOVED_FROM name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py wd=1 >
<Event cookie=3331342 dir=False mask=0x80 maskname=IN_MOVED_TO name=example.py___jb_old___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_old___ src_pathname=example/example.py wd=1 >
<Event cookie=3331343 dir=False mask=0x40 maskname=IN_MOVED_FROM name=example.py___jb_bak___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_bak___ wd=1 >
<Event cookie=3331343 dir=False mask=0x80 maskname=IN_MOVED_TO name=example.py path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py src_pathname=example/example.py___jb_bak___ wd=1 >
<Event dir=False mask=0x200 maskname=IN_DELETE name=example.py___jb_old___ path=example pathname=/home/adrian/dev/flask-multiauth/example/example.py___jb_old___ wd=1 >

[jquacinella]

Yes, I can confirm this is the issue. The same thing happens with vim versus Sublime:

With sublime:

-> % watchmedo log --patterns=".py;.txt" --ignore-directories --recursive . on_moved(self=<watchdog.tricks.LoggerTrick object at 0x2c50050>, event=<FileMovedEvent: src_path='./application/.subl757.tmp', dest_path='./application/core.py'>)

With vim:

on_moved(self=<watchdog.tricks.LoggerTrick object at 0x150f050>, event=<FileMovedEvent: src_path='./application/core.py', dest_path='./application/core.py~'>) on_created(self=<watchdog.tricks.LoggerTrick object at 0x150f050>, event=) on_modified(self=<watchdog.tricks.LoggerTrick object at 0x150f050>, event=)

I have attempted a fix (along with a better error message) and created a pull request.

[rastikerdar]

On Geany, unchecking use_gio_unsafe_file_saving (Edit->Preferences->Various) solved the problem.

[jquacinella]

Hopefully, in the next release, you will not have to do that.

[eventuallyc0nsistent]

thanks @jquacinella ! I never thought it would be an issue with Sublime.

[davidism]

@JasonThomasData Note that the middle part of that answer and the comments are irrelevant. The Werkzeug reloader does not use Python's reload mechanism, no matter how it's run. Also, that's not related to what's being discussed here, which is how the reloader didn't pick up files that were saved atomically by certain editors.

[bersace]

Same here with Doom Emacs, flask reloader does not detect saves.

[bersace]

Is there a way to configure inotify watcher reloader to detect atomic saves ?

[bersace]

I found a workaround user Pylons hupper:

$ FLASK_APP=myapp hupper -m flask
Starting monitor for PID 9877.
Usage: python -m flask [OPTIONS] COMMAND [ARGS]...

...