search

merofeev / docker-windows-volume-watcher

A tool to notify Docker contianers about changes in mounts on Windows.
MIT License
188 stars 30 forks source link

Errors when change files from JetBrains IDE #15

Open mrFleshka opened 5 years ago

mrFleshka commented 5 years ago

Hi, thx for your script, it really helps)

But i have some trouble. I think is not your problem, but I can offer a solution.

History

In work i use JetBrains IDE. And it has specific file save algorithm. On file save IDE create tmp file, update it, then move real file to old backup, then replace real with tmp.

In my test tool it looks like:

INFO:root:EVENT created: ./test\index.js___jb_tmp___ - ./test\index.js___jb_tmp___
INFO:root:EVENT modified: ./test\index.js___jb_tmp___ - ./test\index.js___jb_tmp___
INFO:root:EVENT moved: ./test\index.js - ./test\index.js___jb_old___
INFO:root:EVENT moved: ./test\index.js___jb_tmp___ - ./test\index.js
INFO:root:EVENT deleted: ./test\index.js___jb_old___ - ./test\index.js___jb_old___

You can see two suffixes: ___jb_tmp___ and ___jb_old___

Of course i know about ignore_patterns and add following suffixes to this list. But it not filter move event. With ignores patterns ignore_patterns=["*___jb_tmp___", "*___jb_old___"] it looks like:

INFO:root:EVENT moved: ./test\index.js - ./test\index.js___jb_old___
INFO:root:EVENT moved: ./test\index.js___jb_tmp___ - ./test\index.js

Where second path is event.dest_path (what you use as host_path in __change_handler).

Example

If i run your script as:

docker-volume-watcher --verbose -e *.git* *build* *node_modules* *.docker* *.idea* *___jb_tmp___ *___jb_old___ --debounce 0.4

then i get following output (i have 3 containers with one volume):

INFO:root:Notifying container nginx-test about change in /var/www/app/assets/scss
INFO:root:Notifying container php-fpm-test about change in /var/www/app/assets/scss
INFO:root:Notifying container node-test about change in /var/www/app/assets/scss
INFO:root:
INFO:root:Notifying container nginx-test about change in /var/www/app/assets/scss/app.scss___jb_old___
INFO:root:
INFO:root:Notifying container php-fpm-test about change in /var/www/app/assets/scss/app.scss___jb_old___
INFO:root:
INFO:root:Notifying container node-test about change in /var/www/app/assets/scss/app.scss___jb_old___
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container php-fpm-test about change in /var/www/app/assets/scss/app.scss
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container nginx-test about change in /var/www/app/assets/scss/app.scss
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container node-test about change in /var/www/app/assets/scss/app.scss
INFO:root:
INFO:root:
INFO:root:

As you can see, container try to interract with app.scss___jb_old___ that really does not exists at that moment (already moved to app.scss).

Possible solution

You can add replace in processing relative_host_path for remove ___jb_tmp___ and ___jb_old___ substrings from path. In reasult we take 2 events of app.scss that can easly be fixed by debouncer.

What do you think?

I can create PR, but I don't know Python well =)

merofeev commented 5 years ago

Hi Dmitriy! Thanks for your warm feedback. Also thank you for taking time to report such detailed and interesting issue.

What if we manually filter all events by dest_path in __change_handler? In this case your example should look like this:

INFO:root:EVENT created: ./test\index.js___jb_tmp___ - ./test\index.js___jb_tmp___ [IGNORED by watchdog]
INFO:root:EVENT modified: ./test\index.js___jb_tmp___ - ./test\index.js___jb_tmp___ [IGNORED by watchdog]
INFO:root:EVENT moved: ./test\index.js - ./test\index.js___jb_old___ [IGNORED by handler]
INFO:root:EVENT moved: ./test\index.js___jb_tmp___ - ./test\index.js [PASS]
INFO:root:EVENT deleted: ./test\index.js___jb_old___ - ./test\index.js___jb_old___ [IGNORED by watchdog]

What do you think? (Maybe I'm missing something important).

I can implement this in new branch, but I will have to ask you to test it with number of different IDEs (since I have completely moved from Windows to Linux and can't test it)

mrFleshka commented 5 years ago

Yes, your option is exactly what I proposed, but from a slightly different angle. This should work, because the destination path is used as a file from the event. Put it in the new branch and I will definitely check the correctness of the work of the new edits in the JetBranes IDE and the Eclipse. Please add an additional log on verbous-level for filtered events.

merofeev commented 5 years ago

Hi Dmitriy! I have implemented what we have discussed in the new branch. You should be able to install this experimental version by typing

pip3 install git+https://github.com/merofeev/docker-windows-volume-watcher.git@strict_exclude_patterns --upgrade

Please let me know if this fixes issues with JetBrains IDE and doesn't break anything else.

mrFleshka commented 5 years ago

Thx for your work! I'll check it at weekend)

mrFleshka commented 5 years ago

Hi, i am redy for test results)

PatternMatchingEventHandler check with empty ignore_patterns:

JetBrains:

INFO:root:EVENT created: ./test\index.css___jb_tmp___ - ./test\index.css___jb_tmp___
INFO:root:EVENT modified: ./test\index.css___jb_tmp___ - ./test\index.css___jb_tmp___
INFO:root:EVENT moved: ./test\index.css - ./test\index.css___jb_old___
INFO:root:EVENT moved: ./test\index.css___jb_tmp___ - ./test\index.css
INFO:root:EVENT deleted: ./test\index.css___jb_old___ - ./test\index.css___jb_old___

Notepad++:

INFO:root:EVENT modified: ./test\index.css - ./test\index.css
INFO:root:EVENT modified: ./test\index.css - ./test\index.css

Eclipse:

INFO:root:EVENT modified: ./test\index.css - ./test\index.css
INFO:root:EVENT modified: ./test\index.css - ./test\index.css

PatternMatchingEventHandler check with ignorepatterns=["*_\_jb_tmp", "*_\_jbold"]:

JetBrains:

INFO:root:EVENT moved: ./test\index.css - ./test\index.css___jb_old___
INFO:root:EVENT moved: ./test\index.css___jb_tmp___ - ./test\index.css

Notepad++:

INFO:root:EVENT modified: ./test\index.css - ./test\index.css
INFO:root:EVENT modified: ./test\index.css - ./test\index.css

Eclipse:

INFO:root:EVENT modified: ./test\index.css - ./test\index.css
INFO:root:EVENT modified: ./test\index.css - ./test\index.css

Debug with docker-windows-volume-watcher, mount to one container, default debounce (no ignore patterns):

JetBrains:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css___jb_tmp___
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css___jb_tmp___
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css___jb_old___
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:

Notepad++:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:

Eclipse:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:

Debug with docker-windows-volume-watcher, mount to one container, default debounce (-e ___jb_tmp___ ___jb_old___):

JetBrains:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:
INFO:root:Ignoring event in c:\_Work/Development/DockerWebpackTest\test\index.css___jb_old___ since it matches one of exclude_patterns.
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:

Notepad++:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:

Eclipse:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:

Debug with docker-windows-volume-watcher, mount to one container, (-e ___jb_tmp___ ___jb_old___ --debounce 0.3), best options:

JetBrains:

INFO:root:Ignoring event in c:\_Work/Development/DockerWebpackTest\test\index.css___jb_old___ since it matches one of exclude_patterns.
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test
INFO:root:
INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:

Notepad++:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:

Eclipse:

INFO:root:Notifying container dockerwebpacktest_node_run_478fc7b3c10e about change in /var/www/app/test/index.css
INFO:root:

As a result: With custom ignore_patterns check it works great. Notepad and Eclipse double file update fixed by debounce 0.3 option. For JetBrains additionally added event for dyrectory (cuz we have events for file create and remove). It's normal.

I think all work great now and ready to production)

mrFleshka commented 5 years ago

Hi, do you implement this in main thread?