Daily note doesnt sync to primary device, deletes previously written content and syncs deletion back to secondary device

Thank you for taking the time to report this issue! To improve the process, I would like to ask you to let me know the information in advance.

All instructions and examples, and empty entries can be deleted. Just for your information, a filled example is also written.

Abstract

Livesync eats my daily note when it has been created from phone first

Expected behaviour

Sync begins on Desktop after note has been modified on mobile
Desktop creates daily note as well from template while sync begins
Sync ends and merges the text from mobile into template
Note: this has worked fine for quite some time

Actually happened

Desktop creates file from template
Sync error (see log)
Empty daily note from template is synced back from desktop to mobile, deleting all my work

Reproducing procedure

Note: If you do not catch the reproducing procedure, please let me know the frequency and signs. -> Happens whenever I begin writing a daily note from phone and desktop gets it later

Report from the LiveSync

For more information, please refer to Making the report.

Logs

``` 10/13/2023, 8:18:28 PM->loading plugin 10/13/2023, 8:18:28 PM->Self-hosted LiveSync v0.20.2 0.20.2 10/13/2023, 8:18:28 PM->Self-hosted LiveSync has undergone a major upgrade. Please open the setting dialog, and check the information pane. 10/13/2023, 8:18:28 PM->Waiting for ready... 10/13/2023, 8:18:29 PM->Opening Database... 10/13/2023, 8:18:29 PM->Database is now ready. 10/13/2023, 8:18:31 PM->Initialize and checking database files 10/13/2023, 8:18:31 PM->Checking deleted files 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 25 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 50 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 75 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 100 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 125 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 150 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 175 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 200 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 225 10/13/2023, 8:18:32 PM->Collecting local files on the DB: 250 10/13/2023, 8:18:32 PM->Updating database by new files 10/13/2023, 8:18:32 PM->UPDATE DATABASE 10/13/2023, 8:18:32 PM->UPDATE DATABASE: PASS:0, FAILED:0 10/13/2023, 8:18:32 PM->UPDATE STORAGE 10/13/2023, 8:18:32 PM->UPDATE STORAGE: PASS:98, FAILED:0 10/13/2023, 8:18:32 PM->CHECK FILE STATUS:2/2 10/13/2023, 8:18:32 PM->CHECK FILE STATUS:2/2: PASS:71, FAILED:0 10/13/2023, 8:18:33 PM->CHECK FILE STATUS:1/2 10/13/2023, 8:18:33 PM->CHECK FILE STATUS:1/2: PASS:100, FAILED:0 10/13/2023, 8:18:33 PM->Initialized, NOW TRACKING! 10/13/2023, 8:18:33 PM->Before LiveSync, start OneShot once... 10/13/2023, 8:18:33 PM->OneShot Sync begin... (pullOnly) 10/13/2023, 8:18:33 PM->Replication activated 10/13/2023, 8:18:34 PM->Replication completed 10/13/2023, 8:18:34 PM->LiveSync begin... 10/13/2023, 8:18:34 PM->Replication activated 10/13/2023, 8:18:34 PM->DB -> STORAGE (modify,plain) Daily Notes/2023/10/2023-10-12-Thursday.md 10/13/2023, 8:18:34 PM->Applied Daily Notes/2023/10/2023-10-12-Thursday.md (f:9ee5368d85f57f371b43acd6939d58451e59dcf8ff4393140a033fafacbd3d75:62-ff2ff209c07f48acbaa5c446ff2699d9) change... 10/13/2023, 8:18:34 PM->Delete: Daily Notes/2023/10/2023-10-12-Thursday.md: Conflict revision has been deleted and resolved 10/13/2023, 8:18:34 PM->DB -> STORAGE (modify,force,plain) Daily Notes/2023/10/2023-10-12-Thursday.md 10/13/2023, 8:18:34 PM->Applied Daily Notes/2023/10/2023-10-12-Thursday.md (f:9ee5368d85f57f371b43acd6939d58451e59dcf8ff4393140a033fafacbd3d75:2-1886f600461f9b90b09f6c46603b0be4) change... 10/13/2023, 8:18:35 PM->Delete: !-To-Do.md: Conflict revision has been deleted and resolved 10/13/2023, 8:18:35 PM->Content saved:Daily Notes/2023/10/2023-10-13-Friday.md ,chunks: 2 (new:0, skip:3, cache:1) 10/13/2023, 8:18:35 PM->DB -> STORAGE (modify,plain) Events and Activities.md 10/13/2023, 8:18:35 PM->Applied Events and Activities.md (f:e32c65dfd2271860b3f9deb04e42e2231826e2a7b0d74c8d8cd2e65095e89f5a:206-d7f53a3112de4c2c8ed0a147484b944e) change... 10/13/2023, 8:18:35 PM->DB -> STORAGE (modify,plain) !-To-Do.md 10/13/2023, 8:18:35 PM->DB -> STORAGE (modify,force,plain) !-To-Do.md 10/13/2023, 8:18:35 PM->DB <- STORAGE (plain) Daily Notes/2023/10/2023-10-13-Friday.md 10/13/2023, 8:18:35 PM->DB -> STORAGE (create,plain) ERROR, Could not write: Daily Notes/2023/10/2023-10-13-Friday.md 10/13/2023, 8:18:35 PM->Applied Daily Notes/2023/10/2023-10-13-Friday.md (f:eb67351c619f00e3de4a9e7d38eea18a72e4192a3fd5a894ba2c8831e39dabb0:2-dc8b1a54f9744ae9846ebd7b03366065) change... 10/13/2023, 8:18:46 PM->Replication closed 10/13/2023, 8:18:47 PM->AbortError:The user aborted a request. 10/13/2023, 8:18:48 PM->Before LiveSync, start OneShot once... 10/13/2023, 8:18:48 PM->OneShot Sync begin... (pullOnly) 10/13/2023, 8:18:48 PM->Replication activated 10/13/2023, 8:18:48 PM->Replication completed 10/13/2023, 8:18:48 PM->LiveSync begin... 10/13/2023, 8:18:50 PM->Replication closed 10/13/2023, 8:18:51 PM->AbortError:The user aborted a request. 10/13/2023, 8:22:41 PM->Before LiveSync, start OneShot once... 10/13/2023, 8:22:41 PM->OneShot Sync begin... (pullOnly) 10/13/2023, 8:22:41 PM->Replication completed 10/13/2023, 8:22:41 PM->LiveSync begin... 10/13/2023, 8:23:10 PM->Replication closed 10/13/2023, 8:23:11 PM->AbortError:The user aborted a request. 10/13/2023, 8:39:28 PM->Before LiveSync, start OneShot once... 10/13/2023, 8:39:28 PM->OneShot Sync begin... (pullOnly) 10/13/2023, 8:39:28 PM->Replication completed 10/13/2023, 8:39:28 PM->LiveSync begin... ```

Other information, insights and intuition.

Thanks for creating livesync :)

Thank you for your report and detailed information! It has been so helpful! Without this, we could not know that the reason has been caused by write fail. However, I am still not sure why writing to the storage had failed. Would you please enable the Verbose log and check what has actually happened? The Verbose log could be enabled on the General Settings pane.

Thanks for creating livesync :)

We are creating it! I appreciate you for your patience and contribution! I hope to make it better with all together!

Hi there!

I have found a way to reproduce it on my end whenever i want create the daily note on mobile and fill it boot desktop and wait for sync to begin desktop creates daily note automatically and opens it while sync is still beginning sync begins and causes this: grafik I can reproduce it at any point in time be deleting the daily note on desktop and repeating the steps above, starting on my mobile device

Note: All other files are still being synced properly with no issue

Heres the verbose log: 0/18/2023, 1:05:59 PM->Initialized, NOW TRACKING! 10/18/2023, 1:05:59 PM->Cache initialized 300 / 50000000 10/18/2023, 1:05:59 PM->Modifying callback of the save command 10/18/2023, 1:05:59 PM->Additional safety scan.. 10/18/2023, 1:05:59 PM->Before LiveSync, start OneShot once... 10/18/2023, 1:05:59 PM->OneShot Sync begin... (pullOnly) 10/18/2023, 1:05:59 PM->There are no conflicted files 10/18/2023, 1:05:59 PM->Additional safety scan done 10/18/2023, 1:05:59 PM->Replication activated 10/18/2023, 1:05:59 PM->Applying Daily Notes/2023/10/2023-10-18-Wednesday.md (f:3ed5754fe0fdf17880c4d69e34c768fb2e714df796fc2e8aa86cbc9d3e0e459b: 10-4a6c0ffcbc184452a056645430ee302b) change... 10/18/2023, 1:05:59 PM->Skipped: i:.obsidian/workspace-mobile.json (i:f:a79fae154707da0bcbaf86f169c82611d06ae0537f8e552369d2e5dedfc95225, 195-d6f0c343b4d94d44b5c95b803cf8ce0e) Hidden file sync is disabled. 10/18/2023, 1:05:59 PM->Replication paused 10/18/2023, 1:05:59 PM->Replication completed 10/18/2023, 1:05:59 PM->LiveSync begin... 10/18/2023, 1:05:59 PM->Replication activated 10/18/2023, 1:05:59 PM->Replication paused 10/18/2023, 1:05:59 PM->Replication activated 10/18/2023, 1:06:00 PM->DB -> STORAGE (create,plain) ERROR, Could not write: Daily Notes/2023/10/2023-10-18-Wednesday.md 10/18/2023, 1:06:00 PM->Error:File already exists. 10/18/2023, 1:06:00 PM->Applied Daily Notes/2023/10/2023-10-18-Wednesday.md (f:3ed5754fe0fdf17880c4d69e34c768fb2e714df796fc2e8aa86cbc9d3e0e459b:10-4a6c0ffcbc184452a056645430ee302b) change... 10/18/2023, 1:06:00 PM->Content saved:Daily Notes/2023/10/2023-10-18-Wednesday.md ,chunks: 2 (new:0, skip:3, cache:1) 10/18/2023, 1:06:00 PM->DB <- STORAGE (plain) Daily Notes/2023/10/2023-10-18-Wednesday.md 10/18/2023, 1:06:00 PM->Replication paused 10/18/2023, 1:06:07 PM->Replication is already in progress. 10/18/2023, 1:06:11 PM->Replication closed 10/18/2023, 1:06:12 PM->HTTP:GET to:/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=9578-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX_8_K4M5iYFBND4XKMZubmGaZpqYhK4ehwl5LECSoQFI_YcbJMANNsjCPCU1xSQNXVsWABiIK4M&limit=49 -> failed 10/18/2023, 1:06:12 PM->AbortError:The user aborted a request. 10/18/2023, 1:06:13 PM->Before LiveSync, start OneShot once... 10/18/2023, 1:06:13 PM->OneShot Sync begin... (pullOnly) 10/18/2023, 1:06:13 PM->Replication activated 10/18/2023, 1:06:13 PM->Replication paused 10/18/2023, 1:06:13 PM->Replication completed 10/18/2023, 1:06:13 PM->LiveSync begin... 10/18/2023, 1:06:13 PM->Replication paused 10/18/2023, 1:06:14 PM->Replication paused 10/18/2023, 1:06:15 PM->Replication closed 10/18/2023, 1:06:16 PM->HTTP:GET to:/_changes?style=alldocs&feed=longpoll&heartbeat=30000&since=9578-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX8_K4M5iYFBND4XKMZubmGaZpqYhK4ehwl5LECSoQFI_YcbJMANNsjCPCU1xSQNXVsWABiIK4M&limit=50 -> failed 10/18/2023, 1:06:16 PM->AbortError:The user aborted a request. 10/18/2023, 1:07:38 PM->Before LiveSync, start OneShot once... 10/18/2023, 1:07:38 PM->OneShot Sync begin... (pullOnly) 10/18/2023, 1:07:38 PM->Replication paused 10/18/2023, 1:07:38 PM->Replication completed 10/18/2023, 1:07:38 PM->LiveSync begin... 10/18/2023, 1:07:38 PM->Replication paused 10/18/2023, 1:07:38 PM->Replication paused

Thank you for your detailed reproducing procedure! Thanks for it, the issue has been reproduced!

It should happen with all new notes created during boot. I will address it later!

I will address it later!

Is the fix included in https://github.com/vrtmrz/obsidian-livesync/releases/tag/0.20.6?

Thanks a lot :)

Looks like the issue has returned

1/21/2023, 7:06:23 PM->Replication paused 11/21/2023, 7:06:23 PM->Replication completed 11/21/2023, 7:06:23 PM->LiveSync begin... 11/21/2023, 7:06:23 PM->Replication activated 11/21/2023, 7:06:23 PM->DB -> STORAGE (create,plain) ERROR, Could not write: Daily Notes/2023/11/2023-11-21-Tuesday.md 11/21/2023, 7:06:23 PM->Error:File already exists. 11/21/2023, 7:06:23 PM->Applied Daily Notes/2023/11/2023-11-21-Tuesday.md (f:15a28aae6e8a7ffaf941b663ee3bcf681867422f1b20b8123bf1ec33945e8665:57-17ab32a60c824891821ffb162879460e) change... 11/21/2023, 7:06:23 PM->Applying Daily Notes/2023/11/2023-11-21-Tuesday.md (f:15a28aae6e8a7ffaf941b663ee3bcf681867422f1b20b8123bf1ec33945e8665: 4-e99f898be04f1d05efe7b92f4d30d4cb) change... 11/21/2023, 7:06:23 PM->Content saved:Daily Notes/2023/11/2023-11-21-Tuesday.md ,chunks: 2 (new:0, skip:3, cache:1) 11/21/2023, 7:06:23 PM->Delete: Daily Notes/2023/11/2023-11-21-Tuesday.md: Conflict revision has been deleted and resolved 11/21/2023, 7:06:23 PM->DB <- STORAGE (plain) Daily Notes/2023/11/2023-11-21-Tuesday.md 11/21/2023, 7:06:23 PM->DB -> STORAGE (modify,force,plain) Daily Notes/2023/11/2023-11-21-Tuesday.md 11/21/2023, 7:06:24 PM->Applied Daily Notes/2023/11/2023-11-21-Tuesday.md (f:15a28aae6e8a7ffaf941b663ee3bcf681867422f1b20b8123bf1ec33945e8665:4-e99f898be04f1d05efe7b92f4d30d4cb) change... 11/21/2023, 7:06:24 PM->Replication paused

Also seeing this on 0.23, using windows, macos and android

cc @Outlet2048

verbose log from windows instance:

6/28/2024, 10:05:32 AM->Check or pull from db:Projects/archive/van/General tips.md OK
6/28/2024, 10:05:32 AM->UPDATE STORAGE: DONE:10, FAILED:0, LAST:14
6/28/2024, 10:05:32 AM->STORAGE <- DB (create,plain) ERROR, Could not write: Projects/archive/van/Parts.md
6/28/2024, 10:05:32 AM->Error:File already exists.
6/28/2024, 10:05:32 AM->Check or pull from db:Projects/archive/van/Parts.md OK
6/28/2024, 10:05:32 AM->STORAGE <- DB (create,plain) ERROR, Could not write: Projects/archive/van/References.md
6/28/2024, 10:05:32 AM->Error:File already exists.
6/28/2024, 10:05:32 AM->Check or pull from db:Projects/archive/van/References.md OK
6/28/2024, 10:05:32 AM->STORAGE <- DB (create,plain) ERROR, Could not write: Projects/archive/van/Roof.md
6/28/2024, 10:05:32 AM->Error:File already exists.
6/28/2024, 10:05:32 AM->Check or pull from db:Projects/archive/van/Roof.md OK
6/28/2024, 10:05:32 AM->STORAGE <- DB (create,plain) ERROR, Could not write: Projects/archive/van/Seats.md
6/28/2024, 10:05:32 AM->Error:File already exists.
6/28/2024, 10:05:32 AM->Check or pull from db:Projects/archive/van/Seats.md OK
6/28/2024, 10:05:32 AM->STORAGE <- DB (create,plain) ERROR, Could not write: Projects/archive/van/Security.md
6/28/2024, 10:05:32 AM->Error:File already exists.

here's my config file if that helps

remoteType: ""
useCustomRequestHandler: false
couchDB_URI: ***************
couchDB_DBNAME: obsidian
liveSync: true
syncOnSave: false
syncOnStart: false
savingDelay: 200
lessInformationInLog: false
gcDelay: 0
versionUpFlash: ""
minimumChunkSize: 20
longLineThreshold: 250
showVerboseLog: true
suspendFileWatching: false
trashInsteadDelete: true
periodicReplication: false
periodicReplicationInterval: 60
syncOnFileOpen: false
encrypt: false
usePathObfuscation: false
doNotDeleteFolder: false
resolveConflictsByNewerFile: false
batchSave: false
batchSaveMinimumDelay: 5
batchSaveMaximumDelay: 60
deviceAndVaultName: ""
usePluginSettings: false
showOwnPlugins: false
showStatusOnEditor: true
showStatusOnStatusbar: true
showOnlyIconsOnEditor: true
usePluginSync: false
autoSweepPlugins: false
autoSweepPluginsPeriodic: false
notifyPluginOrSettingUpdated: false
checkIntegrityOnSave: false
batch_size: 25
batches_limit: 25
useHistory: true
disableRequestURI: true
skipOlderFilesOnSync: true
checkConflictOnlyOnOpen: false
showMergeDialogOnlyOnActive: false
syncInternalFiles: false
syncInternalFilesBeforeReplication: false
syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/
syncInternalFilesInterval: 60
additionalSuffixOfDatabaseName: 2d991095cb9bf88e
ignoreVersionCheck: false
lastReadUpdates: 23
deleteMetadataOfDeletedFiles: false
syncIgnoreRegEx: ""
syncOnlyRegEx: ""
customChunkSize: 50
readChunksOnline: true
watchInternalFileChanges: true
automaticallyDeleteMetadataOfDeletedFiles: 0
disableMarkdownAutoMerge: false
writeDocumentsIfConflicted: false
useDynamicIterationCount: false
syncAfterMerge: false
configPassphraseStore: ""
permitEmptyPassphrase: false
useIndexedDBAdapter: true
useTimeouts: false
writeLogToTheFile: false
doNotPaceReplication: false
hashCacheMaxCount: 300
hashCacheMaxAmount: 50
concurrencyOfReadChunksOnline: 30
minimumIntervalOfReadChunksOnline: 25
hashAlg: xxhash64
suspendParseReplicationResult: false
doNotSuspendOnFetching: false
useIgnoreFiles: false
ignoreFiles: .gitignore
syncOnEditorSave: false
pluginSyncExtendedSetting: {}
syncMaxSizeInMB: 50
settingSyncFile: live-sync.config.md
writeCredentialsForSettingSync: false
notifyAllSettingSyncFile: false
isConfigured: true
settingVersion: 0
enableCompression: false
accessKey: ""
bucket: ""
endpoint: ""
region: auto
secretKey: ""
useEden: false
maxChunksInEden: 10
maxTotalLengthInEden: 1024
maxAgeInEden: 10
disableCheckingConfigMismatch: false
displayLanguage: ""
enableChunkSplitterV2: false
disableWorkerForGeneratingChunks: false
processSmallFilesInUIThread: false

Thank you for reporting the issue! And, also thank you for your patience! So sorry for being late! It reproduced! However, this cannot be solved easily.

Resolving conflicts in case-sensitive files is probably difficult. They can be handled correctly on Mac and iOS, but may cause overwriting problems on Windows or Android.

If we can accept the risks, Self-hosted LiveSync can update the storage or database for files with incorrect case names. However, we should be warned and required to make a decision or take some action. (I think, at least, there must be an explicit configuration to automatically handle it).

I will address it later!

Middle report: I am very concerned and have a bad feeling about the case where we have several files that differ only in case. I wish we were able to simply ignore them and think about other files. (In that case, while files with duplicate names are in storage, they are ignored).

What do you think about this as the first step?

Perhaps, at least, this behaviour is safer than the current situation, so I am going to deal with it in that form once.

Hi,

I experienced the same issue with the automatic generation of the daily note but I do not understand the discussion here. To me it looks like the following happens:

On device A I created the daily note and wrote some content into it
On devbice B, not yet synced, a new, empty daily note is created before live sync has synced the daily note that came from device A (via the DB)
The sync process ends with an empy daily note, means, the note from device B somehow "survived".

To my understanding, this is a merge problem: In fact, when the sync process comes to that daily note, two versions of it exist: The one from device A (coming via the DB) and the locally created, empty one. This should be recognized and treaded as a mismatch between the contents of these two copies and hence treated as a merge conflict.

My assumtion is, that it actually is treated as a merge conflict but the auto merge feature favors the empty copy since it has been created more recently. Therefore, I enabled the option "Always resolve merge conflicts manually" which solved to problem: Now I get asked when this happens, can check the two versions in the dialog and select the right one.

vrtmrz / obsidian-livesync