watch-fs on nfs

[xeor]

I think a common usecase for watch-fs would be having the tiddlers stored on an nfs share where the tiddly-server is running, then mounting the same share on a client. In cases like this inotify events are lost.

https://github.com/linonetwo/tiddlywiki-plugins/blob/master/plugins/linonetwo/watch-fs/watch.js#L372 uses nodes fs.watch (https://nodejs.org/api/fs.html#fs_fs_watch_filename_options_listener), which does not support nfs very good.

Alternative would be to have an option to say "use polling", then use node fs.watchFile (https://nodejs.org/api/fs.html#fs_fs_watchfile_filename_options_listener) instead, which looks like is slower, but supports nfs somewhat.

A second way would be to have a little helper daemon running with the server that does a simple touch on the file based on mtime. This triggers the fs.watch event, even inside a container, inside kubernetes using nfs.

I'll do some more thinking and fiddling with this to see if I can come up with a good solution :)

[linonetwo]

Actually the fs library part is copied from node-watch, it is ok to replace it with other impl.

I choosed it because it is dependency free, so easy to cpoy into tw. If you find any other good watcher library, feel free to tell me, or try replace node-watch yourself.

I wanted to use chokidar actually, but it havr too many dependencies, make it uneasy to copy into tw as a plugin's dependency.

[xeor]

Yea, dependencies are a pain in this case.. I'll dig around a little and see if I can find a good solution, or maybe an option to fallback on native poll-based.

[xeor]

It seams like everywhere I look, chokidar is the solution. It is down to 7 dependencies and they have taken away a ton of deps since v2 it seams.. It looks solid, and watch-fs needs a solid watch'er. I don't think it is that bad using it even tho it adds some deps.

Maybe rollup.js can be used to package it, so it will be a single file?

Alternative is to use fs.watchFile nativly..?

What do you think? I can try to make something around chokidar and rollup if you want. Or if you are totally against it, I will try to do watchFile instead :)

[linonetwo]

Hi, if you can get rollup and chokdir work, that would be great!

---

I tried to use rollup to build this, and to use chokdar, but I couldn't get it right, so I decided to finish the logic first.

[xeor]

I've tested this some at https://github.com/xeor/tiddlywiki-plugins/commit/fc4bca72141fd6075381915e3e0fcd10f43a9436, and you can now do

npm install
npm run bundle

and copy the built 3rds.js as a tiddler, then it chokdir will be available using:

const ext = require('./3rds');
ext.chokidar.watch('/tiddlywiki/data/tiddlers').on('all', (event, path) => {
  console.log(event, path);
});

it looks like it might work, however, some things are crashing now. Trying to iron out :)

3rds.js ended up taking around 171K, it is not minified, but I'm not sure it should.. 171K is not that much..

[xeor]

Ok, I think it is ready for some testing.. It works, but I'm not sure if anything is broken. I'll do some more testing as well..

To test

npm install
npm run bundle

• copy in the generated 3rds.js and it's .meta
• Use the modified FileSystemMonitor.js

The complete diff is at https://github.com/linonetwo/tiddlywiki-plugins/compare/master...xeor:master

I changed the isNotNonTiddlerFiles to be a list of glob's instead, more readable and chokidar had a lot of things to watch out for when doing filtering using functions..

[linonetwo]

LGTM, I Will try use this in my wiki today to test it, and you cam make a PR here, I will merge it. Thanks for making this!

[linonetwo]

Oh, to be used as a tw plugin, maybe you have to do some cleanup, or place things into a sub-folder, I don't know which one is better.

And maybe we don't need to ignore 3rds.js, otherwise this file won't be able to commit into github in github based wiki.

[xeor]

I'll do some cleaning:) removing old code, writing some notes about pollbased watching, and not ignoring 3rds.js. will submit a pr later today..

I'm not sure how those github wikis work.. is it enough to just put everything in a folder called 3rds, then keep the generated file in the root?

[linonetwo]

I think after bundle, you can copy everything we need to plugins/linonetwo/watch-fs/watch-fs. So there will be a copy of most of the files. And in Readme we can inform the user this folder is generated...

I know this sounds kind of "dirty", but that will enable a user to quickly copy a folder to their plugins folder. What do you think? And this can be confusing to some new TW users, so I will write a good Readme to get them to know how to get this plugin working.

[linonetwo]

And I recommend SyncThing, it is an opensource software to sync folder p2p, I'm using it to sync my game saves, maybe I will try to use it to sync TW tiddlers, replacing git sync one day.

Is SyncThing work the similar way as nfs?

[xeor]

I think syncthing has the files locally, doesnt it? In that case, the nfs problem won't hit.

I use nfs as a shared volume between some kubernetes workers, and I want tiddlywiki to float around and live on one of them.. Then mounting the same nfs share on my main desktop so I can edit the files. Syncing would be a way, but it feels flaky :p

Maybe the folder can be named plugins/linonetwo/watch-fs/dist, dist stands for distributable, and is normally used.

Are you thinking about moving everything from where it is today over to this folder? If so, I can bundle that logic into the package.json file so that npm run build will do it. Is it every file that has an corresponding .meta

[xeor]

I just did some basic tests for how long it takes for an vscode edit to show up on web. vscode via nfs node running on another server mounting the same nfs share and around 900 files for it to watch (and CHOKIDAR_INTERVAL=200, default 100).

• vscode to web: everything from 5 to 20 seconds (sometimes even 40 seconds). Always got there tho..
• web to vscode: instant if you have the note in focus in vscode
• vscode to web, then change note before it edited the web: web wins, file overwritten with web-info

I don't think the goal for this sort of filesync to be fast, but reliable. I think most usecases here are single-users that just want their notes in vscode, nfs and mobile.

[linonetwo]

vscode to web: everything from 5 to 20 seconds (sometimes even 40 seconds). Always got there tho..

Notice there is a "canSync"

https://github.com/linonetwo/tiddlywiki-plugins/blob/1083f2141613d9ccb49984abb8ddedc493f289aa/plugins/linonetwo/watch-fs/FileSystemMonitor.js#L126-L137

And sync client in the web https://github.com/linonetwo/tiddlywiki-plugins/blob/master/plugins/linonetwo/watch-fs/trigger-sync.html check this.

This is because change may be frequent, and sometimes frequent change caused race condition... So I wait till disk get stable.

[xeor]

yea, but I don't think this was waiting for code. I think it was the poll logic that was slow. But it really doesn't matter :) I know poll is far from optimal regarding file-watching.

Agree on the dist folder and an npm job that copy/generate and puts everything in it. All files that got the corresponding .meta?

[linonetwo]

Yes, every file with a .meta, or you can try advanced json meta like the way flibbles does https://github.com/flibbles/tw5-relink/blob/master/plugins/relink/js/tiddlywiki.files

[xeor]

Is https://github.com/xeor/tiddlywiki-plugins/tree/master/plugins/linonetwo/watch-fs/watch-fs what you ment? You want to call it watch-fs even tho it is already inside a folder like that? Or did I misunderstand something?

Everything is generated and put in place in that folder now by npm run make see https://github.com/xeor/tiddlywiki-plugins/blob/master/plugins/linonetwo/watch-fs/package.json for info..

Any other things left to fix? I did some minor updates to the readme as well :)

[linonetwo]

I think this makes it easier for a user to drag&drop it right into their wiki's plugins folder, without the need to rename it. Just need to be clear in the readme to prevent confusion.

You are making it great! Now please send a PR and I will merge it.