have node.js dynamically monitor the wiki file system to pick up changes made by outside agencies

[rcrath]

problem: when node.js is used on a wiki stored in dropbox and changed in differnt instances, the node.js server has to be killed and restarted and the browser refreshed for the updated changes to show. expected behavior: Tiddlywiki updates to latest version in dropbox folder by reloading the browser window, as in TWC.

Not sure if this is related, but after server restart, I am getting the following error:

syncer-browser - 16:19:28 25 2 2015
Error retrieving skinny tiddler list: XMLHttpRequest error code: 0

steps to reproduce:

1. have two always-on computers synced via dropbox
2. make changes on computer 1, dropbox syncs to cloud, files on computer 2 are updated.
3. On computer 2, refresh browser or close and reopen browser. new tiddlers are not present. Not sure if updated preexisting tiddlers will show updated content, but I assume because of the "skinny" fetching they will, since they are fetching the contents of the updated .tid files. Have not tested this though. but new tiddlers smiply do not appear.

workaround: as per this post and this one on google groups:
A. write tiddlers using http API (but unable to find out how to do that). OR B. restart node.js server instance (as outlined in second post) and refresh browser before starting on a different computer. Works for single user scenario, which is what I am doing. Jeremy says we do plan to fix this, so this is just making it a GitHub issue. I hope it is not a dupe, as I looked for it and could not find it in open or closed issues.

[mardukbp]

This has been discussed in #970. My workaround for this issue is in #1069. I hope you find it useful.

[Jermolene]

Thanks @mardukbp

The plan is indeed to support dynamic (re)loading of tiddlers from the file system. At one point we did have an implementation that was merged:

https://github.com/Jermolene/TiddlyWiki5/pull/176

But it was subsequently disabled because there were problems with the implementation. One of the issues at the time was Node.js's poor support for fs.watch(). I believe that has now matured significantly, so it would certainly be worth revisiting the issue.

[felixhayashi]

+1 Would be a step towards a multi user CMS. At the moment only one person can work on wiki content because of sync issues.

[Jermolene]

At the moment only one person can work on wiki content because of sync issues.

Which sync issues are you thinking of? The multi-user issues that I am aware of are more around the lack of separation between the state of different users.

[rcrath]

Thanks @mardukbp. I did not understand the issue in #970 and did not see a workaround by you in #1069. Is that to restart the server? Not a coder so maybe I missed the point.

@Jermolene should 176 be reopened and this closed as duplicate? or leave this open as new issue? Also would like to know what "One workaround in the meantime is to use the HTTP API to create tiddlers; they will appear with immediate effect." as I was unable to find anything on how to do this. Maybe beyond my skill set anyway?

@felixhayashi THe scenario I laid out is a one person access situation where writing is not taking place simultaneously on multiple instances, but where the multiple servers remain running on always on machines with the user on one or the other at any one time. I am guessing that the multiple user scenario would add another level to the issue.

[mardukbp]

Sorry, the workaround does not solve your problem but allows inserting tiddlers without having to restart the server. It consists of inserting tiddlers programatically using JavaScript. The code is at the bottom of #1069 and can be used in a bookmarklet or the JavaScript console of your browser.

[felixhayashi]

@rcrath

ah, ok then I misunderstood this.

Which sync issues are you thinking of? The multi-user issues that I am aware of are more around the lack of separation between the state of different users.

I will have to test it again to say what was the particular problem. Generally, I am worried about "lost updates" and "dirty reads" because tiddlers could be accessed by more than one party and no lock is put. This prohibits working in a team on the same wiki. But I guess it is another issue...

-Felix

[sukima]

If this feature is reimplemented check out the chokidar project. It smooths over a lot of the rough edges of fs.watch.

[whacked]

Using chokidar per @sukima's suggestion I'm running a fork that picks up on tiddlers edited externally, and updated tiddlers appear on browser refresh.

I'm not familiar with the core code and it's a bit simpler than #176 so I don't know if it's actually a comparable change or if it's PR-ready. If anyone else familiar with the filesystemadaptor.js code can do a look over,

https://github.com/Jermolene/TiddlyWiki5/compare/master...whacked:2015-12_addtiddler-with-chokidar-monitor

[Jermolene]

Hi @whacked great that you're experimenting in this area. I don't think your code is quite ready for the core, but it's pretty good that you've got something that's working for you.

The things that jumped out are the special treatment for drafts and lack of support for non .tid files. TiddlyWiki currently doesn't have any npm dependencies, which is intended to keep it simple. For cases like this, I'd prefer to keep chokidar optional, and only enable the functionality if the require succeeds.

[rcrath]

@whacked and everyone else, thanks for working on this (yells "go team" from the sidelines)

[whacked]

@Jermolene I understand and respect the rationale, thanks for the feedback.

@rcrath as a result mine is likely a niche fork for the time being.

Some other problems with the fork: 1. files created from the web interface save fine, but files created externally (e.g. "test.tid") save, and get an additional copy ("test 1.tid") created. 2. sometimes the server crashes after edits, probably due to the file system watcher loop causing a conflict somewhere. I haven't played with deletions.

Problems aside, the fork notifies the browser to changes via websocket, so edits from external editors are auto-updated. It's an obvious extension, but really speeds up interop.

[n8sabes]

Is there any additional thought or work on this?

It seems this type of solution enables Dropbox or similar service to be used for storage which enables a small team to keep their shared wiki synced without a server.

More importantly, this type of solution enables an offline wiki when traveling without Internet access. In the event there are simultaneous tiddler updates, Dropbox manages the conflict for manual review and takes the latest update in the interim.

I have not fully thought through the implications of this type of syncing, but never the less I would like to see live updates from outside changes to the file system as I use Dropbox for my wikis across systems. I workaround by restarting the TiddlyWiki service with a bash script, which is less than convenient.

[Jermolene]

Hi @n8sabes there's not been any change, I'm afraid.

[otakup0pe]

How tied are we to using fs.watch for this?

[Jermolene]

Hi @otakup0pe happy to consider any alternatives. Bear in mind that there are actually two issues to be resolved:

• The mechanism for learning of changes to the filing system. There are many complex issues here. For example, we store some tiddlers with a separate .meta file. How do we ensure that we wait for both files to be updated before re-reading the tiddler?
• Figuring out the semantics of reloading tiddlers. There's a lot of complexity if we want to handle reloading modules or plugins

Personally, I came away from the experiments we did in 2014 thinking that the benefits weren't worth the work involved, but I'm happy to support others exploring the problems.

(Just for the record, yet another alternative approach would be to focus on supporting WebDAV in the TW Node.js server. Then one could mount a wiki as a file system, and interact with individual tiddlers that way).

[otakup0pe]

While WebDAV would open up some neat possibilities, it wouldn't solve the problem of having an external system synchronize the underlying tiddlers. For this, we really need to ensure that state is properly updated when the underlying tiddler changes.

I'll have to dig into the TW5 server to get a feel for it. With only the barest context to the actual code, it seems like we could simply track the MAC times of files (including supplemental files that are presumably linked internally) and reload internal cache as needed. Is the concern around reloading modules and plugins with regards to reloading the client as well? I'd be happy with the server actually being reloaded, with some element of conflict detection on tiddlers being written to.

[rcrath]

rather than auto updating, perhaps an easier method would be to just notify if any of the files in the underlying local node.js hierarchy change (e.g., I leave TW open at home, change something at work, dropbox syncs, I go home and work on the home computer, which has a notice saying the underlying files have changed) and let the user decide what to do. I have a little bash script I run that restarts the node.js TW server which forces the TW to reload...just having a notice that there are new files would be useful to me. Of course it might be nicer if the tiddlers had individual notices only if updated, but with the "save layout" function, reloading the whole wiki is fine too. I am working in a single user scenario with multiple computers, only one of which is in use at a time, all synced by dropbox, so for me timing is not really an issue...I don't need both at once, just always the most recent with there normally being discrete sessions at each instance/computer with time between where nothing is happening. There would not be simultaneous writes under normal circumstances. Just to be clear, this version of the problem has a different set of parameters than a multiuser collaboratively written wiki, which is different enough to be a separate issue -- I'd like that too, but am happy with TW as a personal wiki!

[pmust]

Has there been any progress on this? It would be wonderful to be able to have local Nodejs servers on computers (without exposing anything to the internet) where each could make changes to the underlying TiddlyWiki (in cloud) and have all of them dynamically update.

[Jermolene]

Hi @DemoniWaari there has been a development -- @inmysocks has published Bob, a plugin that significantly extends the client-server configuration of TW5. One of the features is "Two-way real-time syncing between the browser and file system". See https://github.com/OokTech/TW5-Bob for more details.

[pmust]

Hi @DemoniWaari there has been a development -- @inmysocks has published Bob, a plugin that significantly extends the client-server configuration of TW5. One of the features is "Two-way real-time syncing between the browser and file system". See https://github.com/OokTech/TW5-Bob for more details.

Ooh how did I miss this. Thank you, this sounds exactly what I am after!