search

OokTech / TW5-Bob

A plugin that makes tiddlywiki a multi-user wiki on node
BSD 3-Clause "New" or "Revised" License
216 stars 28 forks source link

Content randomly not syncing, without error #109

Open jeromebaum opened 4 years ago

jeromebaum commented 4 years ago

I'm hitting a weird issue where, seemingly randomly, my content isn't being saved. The save icon in the sidebar is not red, and there is no error message - as far as I can tell, there's no difference between cases where my content is saved, and cases where it's not.

Is there anything I can do to try to track this down? It's pretty hard to reproduce this issue so I haven't been able to catch anything in the javascript console so far.

I am running (check any that apply, put an x inside the [ ] to check a box, like this: [x]):

and using

(The server is behind a caddy reverse proxy that adds basic auth)

Before posting I read issue guidelines and:

jeromebaum commented 4 years ago

I suspect it's related to the websocket connection getting broken - I just had the issue occur again and checked the console, and the only error I saw was related to the websocket connection. I'll keep updating as I learn more.

creinig commented 4 years ago

I have perhaps the same problem and have identified some peculiarity that could explain it at least partially.

My setup:

The wiki was populated by copying over the *.tid files from an existing nodejs TW5. Before that I tried adding the existing wiki as subwiki, but due to different paths on the two systems that didn't work.

In my case, the tiddlers are automatically saved, but for every tiddler I have two files on disk - TIDDLERNAME.tid and TIDDLERNAME_1.tid - and TW / Bob seems to randomly choose which of these to serve and into which to save.

From the outside this looks as if the files are sometimes not saved at all.

inmysocks commented 4 years ago

@jeromebaum

I have recently run into the websockets not giving any error but not saving changes as well. I have not found any consistent way to make this happen and sometimes it will just start working again and save all the changes.

I suspect it has something to do with some bottleneck with saving to the file system. I will continue to look into this, if you find a

@creinig

The problem with your setup is having two instances of bob and foldersync all modifying the same wiki files at the same time. That causes race conditions where depending on the state of the three processes it gives the behaviour you describe. The federated wiki syncing features I am currently working on will give a way to solve this problem, but they aren't documented and polished enough to be too useful.

The only solution until that is available is to only have one process affecting the files at a time, so in your case the foldersync should only run when Bob isn't running on either computer.

inmysocks commented 4 years ago

After a bit more testing it looks like the problem that I am seeing isn't that it isn't trying to sync, it is that there is sometimes a very long delay before files get saved. I just made some edits that took about a minute and a half to save to the file system. And for some reason when the tiddler saving is delayed like this it also doesn't sync the new tiddlers with other open browser tabs.

Hopefully this is the same problem you are having because I think I know where the problem is.

inmysocks commented 4 years ago

The updates in the newest release (1.4.0-beta) should address these problems. I am going to try it for a bit longer before saying this is solved. If you run into these problems using the newest version please let me know.

PARC6502 commented 4 years ago

I believe I am running into the same issue on Bob 1.5.2 (I believe I am up to date with the master branch).

I am running the node version locally, on a linux system, using firefox. Recently I am getting this mainly while editing journal entries, which is only relevant because this will also cause issues with the default new journal text ($:/config/NewJournal/Text).

Currently I am seeing this behaviour every single time I try to create my journal entry for 31st may. It's not a very long or complicated tiddler so I have no idea why it would doing this, but I have yet to succeed in creating it.

I am seeing the following behaviour:

  1. At some point I will finish editing a tiddler and it will have a padlock icon on it, I can no longer make changes to it
  2. Updated tiddler is not showing up in the terminal where I ran the server command
  3. From this point on, regardless of any changes I make, they do not show up in the server terminal
  4. On restarting the server I have lost all changes, they are also not in any backups
  5. I see the folloiwng in the browser console: 
    services.settings: InvalidSignatureError: "Invalid content signature (main/normandy-recipes-capabilities)"
    InvalidSignatureError resource://services-settings/RemoteSettingsClient.jsm:123
    _validateCollectionSignature resource://services-settings/RemoteSettingsClient.jsm:732
remote-settings.js:322
    BroadcastService: receivedBroadcastMessage: handler for remote-settings/monitor_changes threw error: InvalidSignatureError: "Invalid content signature (main/normandy-recipes-capabilities)"
InvalidSignatureError resource://services-settings/RemoteSettingsClient.jsm:123
    _validateCollectionSignature resource://services-settings/RemoteSettingsClient.jsm:732
  6. Save icon is red, and remains red
  7. Opening wiki in different browser allows me to create new tiddlers that appear to be saving but the locked tiddler remains locked
  8. Tried editing the problem tiddler in chromium, after restarting the server, and got the following error 
    WebSocket error:  TypeError: Cannot read property 'fields' of undefined

I am also not really sure how to react when this happens, other than downloading all changed tiddlers, restarting, and reuploading them.

inmysocks commented 4 years ago

@PARC6502 I think I know what caused the problem you are describing, it should be fixed in the nexe branch. I will get around to merging that with dev at some point but for now you could try that branch.

xeor commented 4 years ago

I'm hitting the same problem as well.. Already lost a couple of new notes :( Do you know when 1.4.0 is out so I can start trusting my notes? :)

xeor commented 4 years ago

I looked some more at my case, and I am able to reproduce this using the commander plugin and change tag on ~120 files at the same time.

For me, it seams to be stuck in a loop. Where it tries to do the whole update, but again and again..

Saving a new note when this happens shows no errors, but the save probably ends up faaar behind.

I can test 1.4.0 beta, but I also think there should be an indicator to see how this job-queue is doing, as well as a fix :) I'll come back with more info if I have

xeor commented 4 years ago

I understand that this is a side-project. But can I make some donation to free up some time for anyone to work on the stability of the sync? At least an indicator when a new note is not synced (don't like loosing notes).

joshuafontany commented 4 years ago

Hi. I belive @inmysocks and I have tracked down one of the possible root race-condition bugs behind this failure.

See https://github.com/OokTech/TW5-Bob/pull/134

It requires a patch/PR to TW that has not been accepted into the core yet.

xeor commented 4 years ago

Hi. I belive @inmysocks and I have tracked down one of the possible root race-condition bugs behind this failure.

See #134

It requires a patch/PR to TW that has not been accepted into the core yet.

Thanks for the info. I saw that issue a couple of days ago and was very happy! Great work! I'll see if I can but together a build with that pr for my testing. I'll report back how it went in that issue

inmysocks commented 4 years ago

There between me and @joshuafontany changes in 1.6.0 should help or fix the problems here. It will be released this weekend or Monday.

tejasvi commented 4 years ago

I am having similar issues. Though I do get the (red) not saved indicator. I haved tracked it down to following piece of code:

    $tw.Bob.Reconnect = function (sync) {
      if($tw.connections[0].socket.readyState !== 1) {
        $tw.Bob.setup();
        if(sync) {
          $tw.Bob.syncToServer();
        }
      }
    }
    $tw.Bob.syncToServer = function () {
      // Use a timeout to ensure that the websocket is ready
      if($tw.connections[0].socket.readyState !== 1) {
        setTimeout($tw.Bob.syncToServer, 100)
        console.log('waiting')
      } else {

On console, $tw.connections[0].socket.readyState is not equal to 1 but no console.log('waiting') logs. It seems $tw.Bob.syncToServer is not being called. Either way $tw.Bob.syncToServer() is of no use here (infinite loop), since it effectively needs a $tw.Bob.Reconnect(). I was able to resume sync after it. So, where exactly the call to $tw.Bob.syncToServer() is made? Apart from that $tw.Bob.syncToServer should call $tw.Bob.syncToServer() after certain recursion depth due to $tw.connections[0].socket.readyState !== 1.

inmysocks commented 4 years ago

No, it is of no use here because this has nothing to do with the problem described.

This is for when the browser gets disconnected from the server while the server is still running. In that case there is a reconnect button that appears and uses this function, unless you have clicked that button this code will never run and you shouldn't see the 'waiting' message. The function does what it is meant to do.

If the not saved indicator is red then the most likely situation is that everything is working, but the core syncer puts a delay between each message sent to the server so it can take a long time to save everything if you modify or add a lot of tiddlers at the same time.

If the problem is the connection to the server a 'not connected to the server' banner will appear, if your problem isn't just slow syncing the problem would be something with the syncer module, not this.

tejasvi commented 3 years ago

You are right. I am using Mono theme by JD and the banner was hidden under the top bar. I fixed it by replacing top: with bottom: in WebSockets/BrowserMessageHandlers.js.

dtenbrinck commented 2 years ago

Is there any progress on this issue? I am using TW5-Bob Version 1.7.3 BETA Intense Kitten and I also randomly lose content when creating or editing tiddlers. There is no clear indication that the connection to the server is broken (no red banner, no red check mark), but there seems to by no synchronization between the local Wiki and the server.

I am trying to convince friends and colleagues to use TW5-Bob, but losing content randomly leads people to doubt. Is there anything I can do to help pinning this bug down? Any suggestions from the devs how I can see log messages in detail and try to debug?

Thanks for your engagement in this project. I really appreciate this software a lot.