Python client & remote directory monitoring

[mhauskn]

I've been a fan of this repo since I started playing with Torch. However, most of my work is in Python so I decided to write a python client. The client is a little sparse at the moment and will just visualize images (not charts/graphs/videos etc).

In additional to visualizing data/images on the local machine I wanted to use this repo to visualize images automatically generated by models being trained on remote computers. So rather than loading files from .gfx.js/static/data I've made this path configurable. Starting a server using the python client looks like this:

gfx-start [dir] [port]

Where [dir] is the directory to monitor for images and port is the server port.

Often the directories I want to monitor are remote directories mounted via nfs/sshfs. Node's fs.watch doesn't correctly monitor file changes on remote directories, so I replaced it with the more reliable chokidar (https://github.com/paulmillr/chokidar) for notifications.

If you guys ever want to visualize models that aren't being trained on the local machine perhaps these modifications will be helpful :)

[clementfarabet]

Hey, thanks a lot for that big patch. Glad that repo is helpful to you!

I'm especially interested in the part that gets rid of fs.watch, which is clearly unstable (even locally on some OSes). Just to confirm: did you get rid of it entirely?

Will test it soon and then merge.

Thanks, C

[clementfarabet]

Ok, I just gave it a try, and the watcher doesn't work anymore for me. That's on OSX 10.9.

I'll look more later tonight.

C.

[soumith]

OK chokidar does exactly what we were discussing earlier. It uses fs.watch with a fallback to the slower but portable fs.watchfile on unsupported platforms. Haven't tried this PR yet, but will soon give it a go.

[mhauskn]

Good point! I hadn't tested chokidar when trying to monitor the default directory (~/.gfx.js/static/data). I had to modify a couple things to get it working:

1) changed static path to server.js to include the data directory 2) changed chokidar's options to not ignore files starting with '.' (This was preventing it from seeing changes happening in the ~/.gfx.js/static/data directory)

These changes are now reflected in the pull request.

However, for some reason, when using torch/lua to draw images, the html file associated with the first image is not detected as being added. After the first image draw, subsequent images are rendered without problem.

I've tested by drawing lena on linux and osx.

[clementfarabet]

Cool, getting there: now I'm seeing another issue. When you refresh your browser, it renders everything that's available in the cache directory (/static/data). Since we never delete that history, it's prob not a good idea!

On Tue, May 20, 2014 at 5:02 PM, Matthew Hausknecht < notifications@github.com> wrote:

Good point! I hadn't tested chokidar when trying to monitor the default directory (~/.gfx.js/static/data). I had to modify a couple things to get it working:

1) changed static path to server.js to include the data directory 2) changed chokidar's options to not ignore files starting with '.' (This was preventing it from seeing changes happening in the ~/.gfx.js/static/data directory)

These changes are now reflected in the pull request.

— Reply to this email directly or view it on GitHubhttps://github.com/clementfarabet/gfx.js/pull/9#issuecomment-43682694 .

[mhauskn]

Fixed typo in chokidar options to prevent it from adding existing files. Try again.

[clementfarabet]

OK, one last issue: it seems that the first graphics rendered after the window was just open/visited doesn't show up. Have you seen that?

[mhauskn]

Yes I have. I tried to debug it without success. When the first graphic is rendered chokidar detects the .png file but does not detect the .html file. All further graphics work just fine. I have no idea why it behaves this way. Perhaps one solution would be to ask the server to render some dummy graphic when starting up.

[soumith]

It might be stemming from https://github.com/paulmillr/chokidar/blob/master/index.js#L236 , if there's no prev instance then the first event might be missed. A quick way to check would be to use the same branch on a Linux machine where fs.watch will be used instead and see if the problem persists.

[soumith]

But I don't fully understand yet how the events are triggered

[clementfarabet]

Yeah let's try to figure it out before we merge. It looks good otherwise...

Clément

On May 21, 2014, at 1:49 AM, Soumith Chintala notifications@github.com wrote:

But I don't fully understand yet how the events are triggered

— Reply to this email directly or view it on GitHubhttps://github.com/clementfarabet/gfx.js/pull/9#issuecomment-43714599 .

[mhauskn]

I can confirm that the problem persists on a linux machine as well. Another interesting thing is that if I copy the (html,png) files into the static/data directory, it detects them on the first go. Also python plotting (as in the example script below) results in detection the first time. It seems that only the first html file written by lua doesn't get detected...

import matplotlib
matplotlib.use('Agg') # Don't display
import matplotlib.pyplot as plt
import gfx
import numpy as np

plt.plot(np.random.rand(5))
fname = '/Users/user/.gfx.js/static/data/plot.png'
plt.savefig(fname)
gfx.render(fname)
plt.close()

[mhauskn]

After extensive debugging I've come to the conclusion that there's something weird going on in the graphicsmagick save code run by the lua client. As evidence, if you replace the gm.save call in line 79 of js.lua with a touch:

   -- dump image:
   local filename = win .. '.png'
   -- gm.save(js.static .. filename, img)
   os.execute('touch '..js.static .. filename)

then both the png and html files are detected. (Of course they aren't displayed correctly since the image is empty.)

I've tried creating files via touch/copy/python/lua and the only time I've seen chokidar not detect them is when they are created just after the graphicsmagick save call. I'm not sure whats going on in the graphicsmagick save but it seems to be temporarily masking chokidar from seeing the creation of other files.

[mhauskn]

I've changed chokidar to not use polling by default. This results correct image detection. Added polling as a parameter in the server.js file. Python client disables polling in order to detect images on remote filesystems.

[soumith]

okay, didn't know it used polling by default (which means fs.watchFile). it gets tricky with polling, as I mentioned in that line number earlier, if you have the first two files saved within a shorter window than the polling interval, it misses the event.

[clementfarabet]

arg, but so for OSes with no support for fs.watch, then people will actually have this problem right?

On Fri, May 23, 2014 at 5:28 PM, Soumith Chintala notifications@github.comwrote:

okay, didn't know it used polling by default (which means fs.watchFile). it gets tricky with polling, as I mentioned in that line number earlier, if you have the first two files saved within a shorter window than the polling interval, it misses the event.

— Reply to this email directly or view it on GitHubhttps://github.com/clementfarabet/gfx.js/pull/9#issuecomment-44062777 .

[soumith]

i think so, i cant test it right now am traveling for a couple of weeks, but from a look at the source-code, that's all i could find that might be the issue.

[mhauskn]

Yes, I think so too.

[mhauskn]

I realized that it's much easier to run a server on the remote machine and tunnel localhost:8000 through ssh than it is to visualize images being produced on a remote mounted directory. So I've reverted out the changes to all the library/server files. The PR, should you choose to accept it, contains only a python client.

[soumith]

ok now this PR is just the python client, I guess it can be merged. I'll fix the fs.watch/watchFile in a couple of weeks anyways, just need to get to a stable computer.

[soumith]

@mhauskn I've integrated clean, working chokidar based watching. It would be good to have your python client now, as a separate PR.