Hidden file synchronization 'Fetch' overwrites hidden files in CouchDB

nickshanks347 commented 3 months ago

Abstract

When using 'Fetch' on hidden file synchronization, the hidden files in the DB will be overwritten. They should only be overwritten when clicking 'Overwrite'

Expected behaviour

Hidden files should be fetched and overwritten on the local device with hidden files from the DB

Actually happened

Hidden files are uploaded from the device and overwritten in CouchDB

Reproducing procedure

On first device setup livesync and overwrite hidden files. Everything works
On second device, setup livesync with the setup URI
When asked about hidden file sync, click 'keep them disabled'
When local DB has been set up, observe that Livesync works between both devices
On second device, enable hidden file sync with 'Fetch'
On first device, you can see that hidden files have been reloaded, which should not have been the case

On step 6, if we are fetching the hidden files then they should be applied/reloaded on second device but actually it seems like they're being uploaded from second device and therefore causing the first device to apply them.

Other

Don't think any reports are necessary as I tested with my desktop as first device and macbook & iphone as second device. Let me know if any reports are needed

GottZ commented 3 months ago

I can confirm. I've added a new snippet to .obsidian/snippets and it got lost after hitting fetch on another device.

nickshanks347 commented 3 months ago

Don't want to digress but I moved to customization sync because of this issue. Customization sync has a slight issue where a couple of my plugins 'main.js' files aren't synced.

GottZ commented 3 months ago

tbh.. just judging from the behavior I constantly notice when updating hidden files, I think the sync solution has to be completely reviewed especially for use on mobile.

nickshanks347 commented 3 months ago

tbh.. just judging from the behavior I constantly notice when updating hidden files, I think the sync solution has to be completely reviewed especially for use on mobile.

I agree. Customization sync is working well for me though to sync settings and plugins. It's just sometimes the plugins runtime files aren't synced correctly but otherwise it works well

vrtmrz commented 3 months ago

Thank you for opening the issue! I still trying to reproduce this but have not done yet.

Fetching on the second device could make merged documents. Hence it possibly transferred to the first device. However, the information of @GottZ could indicate more serious. I will address it.

@nickshanks347 Oh, main.js is supposed to be handled by MAIN, but did something go wrong?

GottZ commented 3 months ago

Thank you for opening the issue! I still trying to reproduce this but have not done yet.

Fetching on the second device could make merged documents. Hence it possibly transferred to the first device. However, the information of @GottZ could indicate more serious. I will address it.

@nickshanks347 Oh, main.js is supposed to be handled by MAIN, but did something go wrong?

I've noticed a couple things. when I'm updating my client on windows and create new notes too and let it all sync with livesync and then open the android app, the app will just report hidden files have been scanned without any modifications (ye the client side) but for some strange reason, it does not fetch new data from the couchdb instance. some android log for context:

23.3.2024, 15:22:46->Cache initialized 300 / 250000000000
23.3.2024, 15:22:46->loading plugin
23.3.2024, 15:22:46->Self-hosted LiveSync v0.22.16 0.22.16 
23.3.2024, 15:22:46->xxhash for plugin initialised
23.3.2024, 15:22:46->Waiting for ready...
23.3.2024, 15:22:46->Opening Database...
23.3.2024, 15:22:47->Database is now ready.
23.3.2024, 15:22:47->Log window opened
23.3.2024, 15:22:48->Initialize and checking database files
23.3.2024, 15:22:48->Checking deleted files
23.3.2024, 15:22:48->Collecting local files on the DB: 25
23.3.2024, 15:22:48->Collecting local files on the DB: 50
23.3.2024, 15:22:48->Collecting local files on the DB: 75
23.3.2024, 15:22:48->Collecting local files on the DB: 100
23.3.2024, 15:22:48->Collecting local files on the DB: 125
23.3.2024, 15:22:48->Collecting local files on the DB: 150
23.3.2024, 15:22:48->Collecting local files on the DB: 175
23.3.2024, 15:22:48->Collecting local files on the DB: 200
23.3.2024, 15:22:48->Collecting local files on the DB: 225
23.3.2024, 15:22:48->Collecting local files on the DB: 250
23.3.2024, 15:22:48->Collecting local files on the DB: 275
23.3.2024, 15:22:48->Collecting local files on the DB: 300
23.3.2024, 15:22:48->Collecting local files on the DB: 325
23.3.2024, 15:22:48->Collecting local files on the DB: 350
23.3.2024, 15:22:48->Collecting local files on the DB: 375
23.3.2024, 15:22:48->Collecting local files on the DB: 400
23.3.2024, 15:22:48->Collecting local files on the DB: 425
23.3.2024, 15:22:48->Collecting local files on the DB: 450
23.3.2024, 15:22:48->Collecting local files on the DB: 475
23.3.2024, 15:22:48->Collecting local files on the DB: 500
23.3.2024, 15:22:48->Collecting local files on the DB: 525
23.3.2024, 15:22:48->Collecting local files on the DB: 550
23.3.2024, 15:22:48->Collecting local files on the DB: 575
23.3.2024, 15:22:48->Collecting local files on the DB: 600
23.3.2024, 15:22:48->Updating database by new files
23.3.2024, 15:22:48->UPDATE DATABASE: Nothing to do
23.3.2024, 15:22:48->UPDATE STORAGE
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:10, FAILED:0, LAST:350
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:20, FAILED:0, LAST:341
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:30, FAILED:0, LAST:331
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:40, FAILED:0, LAST:324
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:50, FAILED:0, LAST:312
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:60, FAILED:0, LAST:300
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:70, FAILED:0, LAST:290
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:80, FAILED:0, LAST:283
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:90, FAILED:0, LAST:273
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:100, FAILED:0, LAST:259
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:110, FAILED:0, LAST:252
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:120, FAILED:0, LAST:242
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:130, FAILED:0, LAST:229
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:140, FAILED:0, LAST:221
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:150, FAILED:0, LAST:212
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:160, FAILED:0, LAST:202
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:170, FAILED:0, LAST:192
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:180, FAILED:0, LAST:180
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:190, FAILED:0, LAST:173
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:200, FAILED:0, LAST:163
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:210, FAILED:0, LAST:150
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:220, FAILED:0, LAST:140
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:230, FAILED:0, LAST:130
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:240, FAILED:0, LAST:120
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:250, FAILED:0, LAST:110
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:260, FAILED:0, LAST:100
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:270, FAILED:0, LAST:92
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:280, FAILED:0, LAST:82
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:290, FAILED:0, LAST:70
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:300, FAILED:0, LAST:60
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:310, FAILED:0, LAST:50
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:320, FAILED:0, LAST:40
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:330, FAILED:0, LAST:30
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:340, FAILED:0, LAST:19
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:350, FAILED:0, LAST:9
23.3.2024, 15:22:49->UPDATE STORAGE: DONE:360, FAILED:0, LAST:0
23.3.2024, 15:22:49->UPDATE STORAGE All done: DONE:368, FAILED:0
23.3.2024, 15:22:49->Initialized, NOW TRACKING!
23.3.2024, 15:22:49->Synchronizing hidden files...
23.3.2024, 15:22:49->Scanning hidden files.
23.3.2024, 15:22:50->Hidden file: 100/163
23.3.2024, 15:22:50->Hidden files scanned: 6 files had been modified
23.3.2024, 15:22:50->Synchronizing hidden files done
23.3.2024, 15:22:50->Scanning hidden files.
23.3.2024, 15:22:51->Hidden file: 100/163
23.3.2024, 15:22:51->Hidden files scanned: 0 files had been modified
23.3.2024, 15:23:51->Scanning hidden files.
23.3.2024, 15:23:52->Hidden file: 100/163
23.3.2024, 15:23:52->Hidden files scanned: 0 files had been modified
23.3.2024, 15:24:51->Scanning hidden files.
23.3.2024, 15:24:52->Hidden file: 100/163
23.3.2024, 15:24:52->Hidden files scanned: 0 files had been modified
23.3.2024, 15:25:51->Scanning hidden files.
23.3.2024, 15:25:52->Hidden file: 100/163
23.3.2024, 15:25:52->Hidden files scanned: 0 files had been modified
23.3.2024, 15:26:31->OneShot Sync begin... (sync)
23.3.2024, 15:26:31->Looking for the point last synchronized point.
23.3.2024, 15:26:31->The request may have failed. The reason sent by the server: 404: 
23.3.2024, 15:26:31->Replication activated
23.3.2024, 15:26:31->The request may have failed. The reason sent by the server: 404: 
23.3.2024, 15:26:33->The request may have failed. The reason sent by the server: 404: 
23.3.2024, 15:26:39->↑0 ↓24 (261)
23.3.2024, 15:26:39->↑0 ↓38 (2)
23.3.2024, 15:26:40->Hidden file conflicted:i:.obsidian/community-plugins.json
23.3.2024, 15:26:40->Hidden file conflicted:i:.obsidian/plugins/obsidian-list-callouts/main.js
23.3.2024, 15:26:40->Hidden file conflicted:i:.obsidian/plugins/obsidian-list-callouts/manifest.json
23.3.2024, 15:26:40->Hidden file conflicted:i:.obsidian/plugins/obsidian-list-callouts/styles.css
23.3.2024, 15:26:40->Hidden file conflicted:i:.obsidian/plugins/pdf-plus/main.js
23.3.2024, 15:26:40->Older one has been deleted:i:.obsidian/plugins/obsidian-list-callouts/main.js
23.3.2024, 15:26:40->Older one has been deleted:i:.obsidian/plugins/obsidian-list-callouts/styles.css
23.3.2024, 15:26:40->Older one has been deleted:i:.obsidian/plugins/pdf-plus/main.js
23.3.2024, 15:26:40->Hidden file conflicted:i:.obsidian/plugins/pdf-plus/manifest.json
23.3.2024, 15:26:40->Object merge:i:.obsidian/plugins/obsidian-list-callouts/manifest.json
23.3.2024, 15:26:40->Chunks saved: doc: .obsidian/plugins/obsidian-list-callouts/manifest.json ,chunks: 1 (new:1, recycled:0, cached:0)
23.3.2024, 15:26:40->STORAGE --> DB:.obsidian/plugins/obsidian-list-callouts/manifest.json: (hidden) Done
23.3.2024, 15:26:40->Hidden file .obsidian/plugins/obsidian-list-callouts/manifest.json has conflicted revisions, to keep in safe, writing to storage has been prevented
23.3.2024, 15:26:40->Processing 0 Index/Untitled.md (0 Index/ :6-55c) : Done
23.3.2024, 15:26:40->STORAGE <- DB (modify,plain) 0 Index/0 Index.md
23.3.2024, 15:26:40->Processing 0 Index/0 Index.md (0 Index/ :153-4) : Done
23.3.2024, 15:26:41->Processing 0 Index/Bolognese.md (0 Index/ :53-8a) : Done
23.3.2024, 15:26:41->STORAGE <- DB (create,plain) Data/Rezepte/Die gepfuschte Bolognese.md
23.3.2024, 15:26:41->Processing Data/Rezepte/Die gepfuschte Bolognese.md (Data/Rez :12-cc) : Done
23.3.2024, 15:26:41->Processing Data/Rezepte/Bolognese.md (Data/Rez :11-0f) : Done
23.3.2024, 15:26:41->Skipped merging: .obsidian/community-plugins.json
23.3.2024, 15:26:51->Hidden file conflicted:i:.obsidian/community-plugins.json
23.3.2024, 15:26:51->Hidden file conflicted:i:.obsidian/plugins/pdf-plus/manifest.json
23.3.2024, 15:26:53->Entry removed:i:.obsidian/plugins/pdf-plus/manifest.json (i:.obsid-3-3b3423f98ad0634c7f66c6bfffa3c57f)
23.3.2024, 15:26:53->Conflicted revision has been deleted: .obsidian/plugins/pdf-plus/manifest.json
23.3.2024, 15:26:53->STORAGE <-- DB:.obsidian/plugins/pdf-plus/manifest.json: written (hidden, overwrite)
23.3.2024, 15:26:53->STORAGE --> DB:.obsidian/plugins/pdf-plus/manifest.json: extracted (hidden,merged)
23.3.2024, 15:27:01->Entry removed:i:.obsidian/community-plugins.json (i:.obsid-138-ddcd6d14b30b8ec59b01b7fedf6de3b3)
23.3.2024, 15:27:01->Conflicted revision has been deleted: .obsidian/community-plugins.json
23.3.2024, 15:27:01->Skipped merging: .obsidian/community-plugins.json
23.3.2024, 15:27:01->STORAGE <-- DB:.obsidian/community-plugins.json: written (hidden, overwrite)
23.3.2024, 15:27:01->STORAGE --> DB:.obsidian/community-plugins.json: extracted (hidden,merged)
23.3.2024, 15:27:05->Could not merge conflicted json
23.3.2024, 15:27:05->Scanning hidden files.
23.3.2024, 15:27:05->Scanning hidden files.
23.3.2024, 15:27:06->Hidden files scanned: 1 files had been modified
23.3.2024, 15:27:06->Scanning hidden files.
23.3.2024, 15:27:07->Hidden file: 100/166
23.3.2024, 15:27:07->Chunks saved: doc: .obsidian/appearance.json ,chunks: 1 (new:0, recycled:1, cached:0)
23.3.2024, 15:27:07->STORAGE --> DB:.obsidian/appearance.json: (hidden) Done
23.3.2024, 15:27:07->STORAGE <-- DB:.obsidian/plugins/dataview/main.js: written (hidden, overwrite)
23.3.2024, 15:27:07->STORAGE <-- DB:.obsidian/plugins/dataview/manifest.json: written (hidden, overwrite)
23.3.2024, 15:27:07->STORAGE <-- DB:.obsidian/plugins/obsidian-tasks-plugin/styles.css: written (hidden, overwrite)
23.3.2024, 15:27:07->STORAGE <-- DB:.obsidian/plugins/obsidian-projects/styles.css: written (hidden, overwrite)
23.3.2024, 15:27:07->Chunks saved: doc: .obsidian/plugins/notion-like-tables/styles.css ,chunks: 1 (new:0, recycled:1, cached:0)
23.3.2024, 15:27:07->STORAGE --> DB:.obsidian/plugins/notion-like-tables/styles.css: (hidden) Done
23.3.2024, 15:27:07->Chunks saved: doc: .obsidian/plugins/homepage/styles.css ,chunks: 1 (new:0, recycled:1, cached:0)
23.3.2024, 15:27:07->STORAGE --> DB:.obsidian/plugins/homepage/styles.css: (hidden) Done
23.3.2024, 15:27:07->STORAGE <-- DB:.obsidian/plugins/obsidian-projects/main.js: written (hidden, overwrite)
23.3.2024, 15:27:07->STORAGE <-- DB:.obsidian/plugins/obsidian-day-planner/styles.css: written (hidden, overwrite)
23.3.2024, 15:27:07->STORAGE <-- DB:.obsidian/plugins/obsidian-projects/manifest.json: written (hidden, overwrite)
23.3.2024, 15:27:07->Chunks saved: doc: .obsidian/plugins/customjs/styles.css ,chunks: 1 (new:0, recycled:1, cached:0)
23.3.2024, 15:27:07->Chunks saved: doc: .obsidian/plugins/code-emitter/main.js ,chunks: 1 (new:0, recycled:1, cached:0)
23.3.2024, 15:27:07->Chunks saved: doc: .obsidian/plugins/code-emitter/styles.css ,chunks: 1 (new:0, recycled:1, cached:0)
23.3.2024, 15:27:08->STORAGE --> DB:.obsidian/plugins/customjs/styles.css: (hidden) Done
23.3.2024, 15:27:08->STORAGE --> DB:.obsidian/plugins/code-emitter/main.js: (hidden) Done
23.3.2024, 15:27:08->STORAGE --> DB:.obsidian/plugins/code-emitter/styles.css: (hidden) Done
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/obsidian-task-collector/main.js: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/obsidian-task-collector/manifest.json: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/obsidian-task-collector/data.json: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/obsidian-task-collector/styles.css: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/excalibrain/manifest.json: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/excalibrain/main.js: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/excalibrain/styles.css: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE -x> DB:.obsidian/plugins/local-backup/data.json: (hidden) Done
23.3.2024, 15:27:08->STORAGE -x> DB:.obsidian/plugins/local-backup/main.js: (hidden) Done
23.3.2024, 15:27:08->STORAGE -x> DB:.obsidian/plugins/local-backup/manifest.json: (hidden) Done
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/pdf-plus/data.json: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/pdf-plus/main.js: written (hidden, overwrite)
23.3.2024, 15:27:08->STORAGE <-- DB:.obsidian/plugins/pdf-plus/styles.css: written (hidden, overwrite)
23.3.2024, 15:27:08->Hidden files scanned: 29 files had been modified
23.3.2024, 15:27:10->Hidden files scanned: 22 files had been modified
23.3.2024, 15:27:10->Scanning hidden files.
23.3.2024, 15:27:11->Hidden files scanned: 1 files had been modified
23.3.2024, 15:27:34->↑1 (3735) ↓38 (2)
23.3.2024, 15:27:41->↑2 (830) ↓38 (2)
23.3.2024, 15:27:43->↑21 (LIVE) ↓38 (2)
23.3.2024, 15:27:43->↑32 (LIVE) ↓38 (2)
23.3.2024, 15:27:43->Replication completed
23.3.2024, 15:27:51->Scanning hidden files.
23.3.2024, 15:27:52->Hidden file: 100/163
23.3.2024, 15:27:52->Hidden files scanned: 0 files had been modified

I've hit the replicate action on android to initiate the sync.

the bad part: the data that was synced from db to android did not persist.

on phone:

on my windows machine:

as you can see, some of the plugins have different filesizes on android.

vrtmrz commented 2 months ago

Thank you for the detailed information!

It seems that some files are conflicted. And indeed, it could be happened at the initial sync in the current algorithm. Possibly, we have to delete all of the existing hidden files on the local database, and then receive them from the remote. I will check it.

as you can see, some of the plugins have different filesizes on android.

This is a concern. Is there any difference in the actual file being saved? I would appreciate it if you could try to pull them with adb and compare them.

GottZ commented 2 months ago

I can only write down from memory:

I've resolved conflicts a couple times before by just completely re-instanciating the whole sync plugin from scratch.

I've been using this ignore pattern on all devices:

\/node_modules\/,
\/\.git\/,
\/obsidian-livesync\/,
\.obsidian\/plugins\/recent-files-obsidian\/data\.json,
\.obsidian\/workspace\.json,
\.obsidian\/workspace-mobile\.json,
\.stignore,
workspace$

all devices had live-sync and dotfiles enabled.

I've noticed moving folders in my vault re-appearing at their previous location, dublicate files, overwritten changes and files gone missing throughout using this plugin. I've noticed most recent changes only appearing minutes after starting the vault on mobile. after waiting, new changes appeared instant as they should.

I'd recommend adapting sync behavior a little.

if I open a note, it should always be nr 1 priority, to check if this file was changed upstream. if so, sync changes to the current note downwards immediately.
check for .obsidian/plugins changes, to ensure plugins are up2date.
1. check if obsidian-livesync was updated. in that case, update.
2. prompt for reloading in case updates were applied
check the remaining vault for changes

out of various personal stability requirements, I've moved off to syncthing successfully for now.

vrtmrz commented 1 month ago

Sorry for being late! I have found some code for handling internal files that lead to suspicious behaviour. These have been fixed in v0.23.8.

It is a little difficult to say for sure, as the identity is still unclear, but it is possible that the latest version is a little better.

I'd recommend adapting sync behavior a little.

Self-hosted LiveSync synchronises all files with conflicts, leaving conflict resolution to the client. Therefore, it might be caused by other reasons. -- i.e., conflict resolution failure or some other things. Because, while they were conflicted, they should not be written into the storage in the default setting. The behaviour you described could have been achieved through a set of settings.

I regret the delay in responding to you, and, lost the chance to solve your problem.

However, I am so relieved that your environment has stabilised even if with other tools. I hope that things will be somewhat better for you if you return!

I close this issue with some regrets. Please feel free to open this again.

vrtmrz / obsidian-livesync