tiddlywiki by node.js can't write tiddlers to server

[turtlekey]

Tiddlywiki5 is awesome👍👍👍, and I have used it for a long time🌹. But today is terrible, because I found some of my tiddlers have lost after rebooting tiddlywiki, really lost💔. Yes, now I have figured out that the reason is that some tiddlers are only in server's cache rather than write to the server, to the disk permanently, to the tiddlers/ directory, and eventually they lost after rebooting. Also what I have found is below:

• I have lost some tiddlers, not all, some older tiddlers are in the directory tiddlers/.
• I try deleting some tiddlers in the directory tiddlers/, and then writing tiddler from browser page, eventually the tiddler is writen to the tiddlers/ directory successfully. So maybe there is size limit for the directory tiddlers/?
• the available size of the server disk is enough (30G).
• the authority mode of the directory tiddlers/ is 775.
• the log file has 8.5 million rows, most of them are the error: syncer-server-filesystem: Dispatching 'save' task: ××, Sync error while processing save of '××': Error: EACCES: permission denied, open '/home/myhome/turtlewiki/tiddlers/××.md, it can be seen that the tiddlywiki tries to write the tiddler to the directory tiddlers/ all the time, but fails all the time🤷‍♀️.

TiddlyWiki Configuration (please complete the following information):

• Version: v5.1.22
• Saving mechanism: Node.js
• Plugins installed: Highlight, Markdown, Sync changes to TW5 server or TiddlyWeb server, Synchronise changes to the local filesystem, TiddlyWiki5 core

Desktop (please complete the following information):

• OS: centos7(server), window10(client)
• Browser: chrome
• Version: 92.0.4515.131

Please help me to resolve this problem. I am looking forward your reply👀. Thanks very much❤❤❤!

[linonetwo]

Yes, this guy lost half a year of tiddlers.

He is using the official nodejs setup, not TiddlyGit. But my TiddlyGit is just a wrapper of nodejs, so TiddlyGit also suffers from this. One of the TiddlyGit users also reports that Windows sometimes prevent write into the filesystem, and TW just silently writes into the memory cache, while Typora will still somehow rise a Windows prompt to ask for permission to write to fs. https://github.com/tiddly-gittly/TiddlyGit-Desktop/issues/96

So force creates the file, or inform the user that fs is unavailable, is very important for nodejs setup, I think.

[saqimtiaz]

There was a similar issue reported in #4742 and some measures were introduced to guard against data loss in 5.1.23. There is definitely more work to be done in this area to inform the user when there are issues writing to disk and some of the ideas discussed there need further consideration.

I am also curious @turtlekey , have you at any point the changed the user which owns the tiddlers directory or the way in which you run the node process (which might have changed the user for the node process). I realize that mode 775 makes this an unlikely source of errors but it would be good to confirm.

Could you also please share the number of files in the tiddler directory and the size of the tiddler directory?

[turtlekey]

@linonetwo thanks for cuing🚗. @saqimtiaz thanks for your reply and let me answer your question firstly.

• I haven't change the owner of the tiddlers directory, because I always connect to my server via the only username, besides root, and the way to find the tiddlywiki process is ps aux|grep username, so these can make sure that the owner is the same one. And the way I run the node tiddlywiki process is nohup tiddlywiki mytiddlywiki --listen host=0.0.0.0 port=**** username=** password=** &, I think there is no problem.
• the number of files in the tiddler directory is 122.
• the size of the tiddler directory is 616k.

By the way, I have read the issue #4742, and I think the two issues are the same. I have found that there is a Draft tiddler in my tiddler directory, and tiddlywiki tries to save it all the time, but fails, the newer tiddlers can't write to the tiddler directory because the Draft tiddler blocks the way. The date confirms it, the Draft tiddler was created on Dec 13, 2020, and since then, my all new tiddlers lost, this is a huge loss to my life and work, really.

So the question is why the Draft tiddler can't save to the tiddler directory? The problem is from server IO or tiddlywiki? And in other hand, the front-end and back-end behavior inconsistently, because I have clicked the save button, and the cloud icon go from red to normal, I think everything is ok, but my tiddlers will lose indeed, I hope we can resolve this problem as soon as possible😎.

[saqimtiaz]

@turtlekey thank you for the thorough answers, it is really helpful in eliminating possible causes of the issue.

By the way, I have read the issue #4742, and I think the two issues are the same. I have found that there is a Draft tiddler in my tiddler directory, and tiddlywiki tries to save it all the time, but fails, the newer tiddlers can't write to the tiddler directory because the Draft tiddler blocks the way.

Could you please share:

• the full filename and path of this Draft tiddler
• the Draft tiddler .tid file as well as the file for StoryList and DefaultTiddlers

I am wondering if there are characters in the filename or the tiddler itself that are somehow causing a write error. Oh, if you can also check ownership of this Draft file that would be great too.

After you have done that, it would be worth checking if the improvements in 5.1.23 would help with this issue. So if you are comfortable, backup your wiki folder and then try upgrading to 5.1.23.

[turtlekey]

@saqimtiaz thank you for the further reply. you have reminded me where is the problem from.

Oh, if you can also check ownership of this Draft file that would be great too.

Yes, some tiddlers are owned by root, not me, including the Draft tiddler, so it is the reason for writing failure. Now I wonder why some tiddlers are owned by root, I create tiddlers from the browser side all the time, not server side, Maybe sometimes I run the process by the user root, instead of the general user, especially when testing and debugging. Hope we can inform users when errors occur like writing failure via front-side, and avoid this kind of block. Good luck for TW5❤

[pmario]

Yes, some tiddlers are owned by root, not me, including the Draft tiddler, so it is the reason for writing failure. Now I wonder why some tiddlers are owned by root, I create tiddlers from the browser side all the time, not server side, Maybe sometimes I run the process by the user root, instead of the general user, especially when testing and debugging. Hope we can inform users when errors occur like writing failure via front-side, and avoid this kind of block. Good luck for TW5:heart:

Ouch. ... I'm sorry, that you lost data. .... IMO we need a "Best Practice" description somewhere linked to GettingStarted node.js, where we explain how to start a server with its own user, that has no other write rights than to the wiki tiddlers and files folder.

This will also be more secure, since most systems are set up in a way, that the "standard" user is also member at the group "sudoers". The server should definitely be not in this group. ...

[pmario]

It seems, the log file did contain some hints about the problem. So this info should also be part of the "best practice" docs