Folder (and content lost) upon folder rename

[hilsonp]

Hello,

Thank you so much for this Self-hosted LiveSync possibility ! This is awesome and so easy to setup.

I created a small vault on a Win10 Laptop (Laptop 1) and put one note in it. I synched it to the server (docker running on a synology). I then added a second obsidian on an iphone (this is so amazing how easy we can configure other devices thanks to your setup string/passphrase). I did not play much on the iphone. Mostly consultation and adding a picture and a pdf. I then added a third obsidian on a second Win10 Laptop (Laptop 2).

All devices have Livesync enabled. The iOS device is most of the time with the lock screen. "Do not delete empty folder" is not set on devices.

Once I had everything in sync, I renamed a folder on Laptop 2 and got a case were the folder got duplicated on the Laptop 1 (old and new name). I continued playing, trying to find a reproductible scenario. I first though that this was because the filename had en emoticon ("🚚 TestFolder" renamed into "TestFolder") but in fact, even without any special character, I get all kind of problem when I rename folder.

This is not systematic but is really easy to produce.

The worst case being that after renaming, the folder completely disappeared on both Laptops (including, the file that was in).

How can I best reproduce it for you to investigate ? Is there any log that I should collect ? On the laptops ? On the server ?

Sorry for bringing such a worrying issue :-(

Thanks again for your huge work !

[hilsonp]

Ok. I found how to reproduce the problem.

1. Install and activate the plugin "Folder Note Core". Set Storage Strategy to "Inside Folder with same name"
2. Install and activate the plugin "AidenLx's Folder Note". Set Storage Strategy to "Inside Folder with same name"
3. Laptop1: Create a folder TempFolder.
4. Laptop2: The folder TempFolder does not happen because (empty) folders are not synced
5. Laptop1: Ctrl-click on created TempFolder folder to create a folder note. A TempFolder.md is created in the TempFolder folder.
6. Laptop2: The folder appears (because the folder TempFolder has a file TempFolder.md inside).
7. Laptop1: Rename the TempFolder to TempFolder2. This renames the folder and the TempFolder.md (to TempFolder2.md).
8. Laptop2:
   • Sometimes then TempFolder remains (emptied) while TempFolder2 appears with the TempFolder2.md inside.
   • Sometimes TempFolder and TempFolder2 are gone 😱
9. Laptop1: If I restart Obsidian:
   • Sometimes then TempFolder remains (emptied) while TempFolder2 appears with the TempFolder2.md inside.
   • Sometimes TempFolder and TempFolder2 are gone 😱

If I disable the folder, the problem disappear but honestly, this plugin is sooooo useful.

I may try the other storage strategies but this one is the recommended/default one.

I think this plugin just demonstrate a race condition effect that leads to data loss and should be further investigated. I'm ready to help is you need any information.

[hilsonp]

I went a bit further in my testing and deactivated the "Folder Note Core" and "AidenLx's Folder Note" plugins.

I can reproduce the problem almost every time like this:

1. Laptop1: Open Obsidian (with livesync on)
2. Laptop2: Open Obsidian (with livesync on) and let it sync
3. Laptop1: Create folder "test" and create a note inside it.
4. Laptop2: "test" folder appears with the note inside
5. Laptop1: Rename "test" folder into "test2"
6. Laptop2: The folder "test" disappear and the "test2" folder never appear.
7. Laptop1: The folder "test2" disappear (and so the note inside is gone)

[vrtmrz]

Thank you for the detailed procedure! I will test it later! (Sorry for the short reply!)

[vrtmrz]

@hilsonp This could not be reproduced in my vault. However, I found several suspicious implementations.
Fixed them once at v0.17.11. Could you please test this version?

[hilsonp]

Hello,

Thank you for the fast analysis.

I updated to v0.17.11 on the 3 devices and restarted obsidian.

I disabled hidden files and internal files because I'm seeing strange behaviours and don't want to mix 2 things.

It seems that a problem remain (see last trace here below) if:

• "AidenLx's Folder Note" autosync is enabled
• folder's note is renamed (wich automatically rename the folder as well)

With obsidian being opened only on 2 laptops (closed on my iphone), here is what I do:

Test with "AidenLx's Folder Note" autosync disabled

Laptop1 and Laptop2: Turn off "AidenLx's Folder Note" autosync

Laptop1: Restart Obsidian and wait until I see PM->LiveSync begin...

Laptop2: Restart Obsidian and wait until I see PM->LiveSync begin...

In the console of Laptop2, I see, regularly:

Access to fetch at '[https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4237-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXMuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADSfK_c&limit=250'](https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4237-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXMuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADSfK_c&limit=250%27) from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
plugin:obsidian-livesync:15732          GET https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4237-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXMuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADSfK_c&limit=250 net::ERR_FAILED 307

This error is not seen on Laptop1.

Laptop1: Create folder test11 with note note11 under it.

Pensine:1/16/2023, 11:15:33 PM->Content saved:test11/Untitled.md ,chunks: 1 (new:0, skip:1, cache:0)
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:33 PM->DB <- STORAGE (plain) test11/Untitled.md
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:33 PM->Replication activated
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:41 PM->Content saved:test11/note11.md ,chunks: 1 (new:0, skip:1, cache:0)
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:41 PM->DB <- STORAGE (plain) test11/note11.md
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:41 PM->Replication activated
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:41 PM->entry removed:test11/Untitled.md-2-e0db6b3c1bf5ae7acffcfd76c8daec39
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:44 PM->Replication activated
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:44 PM->Applying 1 files
plugin:obsidian-livesync:19436 Pensine:1/16/2023, 11:15:44 PM->Applied test11/Untitled.md (3-50159d0fd6babf3505c73a66e188b986) change...

Laptop2: test11/note11 appears

Access to fetch at 'https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4237-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXMuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADSfK_c&limit=250' from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
GET https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4237-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXMuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADSfK_c&limit=250 net::ERR_FAILED 307
Pensine:16/01/2023 23:15:31->TypeError:Failed to fetch
Pensine:16/01/2023 23:15:36->Replication activated
Pensine:16/01/2023 23:15:36->Applying 1 files
Pensine:16/01/2023 23:15:37->DB -> STORAGE (create,plain) test11/Untitled.md
Pensine:16/01/2023 23:15:37->Applied test11/Untitled.md (1-0209ccb53b7a099b96fef094d70e12b0) change...
Pensine:16/01/2023 23:15:43->Replication activated
Pensine:16/01/2023 23:15:44->Applying 2 files
Pensine:16/01/2023 23:15:44->DB -> STORAGE (create,plain) test11/note11.md
Pensine:16/01/2023 23:15:44->Applied test11/note11.md (1-ed509ffa62a0f25c07dd2467df52fd56) change...
Pensine:16/01/2023 23:15:44->xxx <- STORAGE (deleted) test11/Untitled.md
Pensine:16/01/2023 23:15:44->files: 1
Pensine:16/01/2023 23:15:44->Applied test11/Untitled.md (2-e0db6b3c1bf5ae7acffcfd76c8daec39) change...
Pensine:16/01/2023 23:15:44->deleteDB By path:test11/Untitled.md
Pensine:16/01/2023 23:15:44->entry removed:test11/Untitled.md-3-50159d0fd6babf3505c73a66e188b986
Pensine:16/01/2023 23:15:44->Replication activated
Access to fetch at 'https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4241-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXsuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADY7K_s&limit=247' from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
GET https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4241-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXsuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADY7K_s&limit=247 net::ERR_FAILED 307
Access to fetch at 'https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4241-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXsuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADY7K_s&limit=250' from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
GET https://obsidian-livesync.my.domain.com/myuser/_changes?style=all_docs&feed=longpoll&heartbeat=30000&since=4241-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBfXsuUIzd0swyxTTFDF09DhPyWIAkQwOQ-g83iOMW2CBDA4MkS-NUdG1ZADY7K_s&limit=250 net::ERR_FAILED 307
Pensine:16/01/2023 23:16:09->TypeError:Failed to fetch

Laptop1: Renamed folder test11 to test12

Pensine:1/16/2023, 11:24:05 PM->Content saved:test12/note11.md ,chunks: 1 (new:0, skip:1, cache:0)
Pensine:1/16/2023, 11:24:05 PM->DB <- STORAGE (plain) test12/note11.md
Pensine:1/16/2023, 11:24:05 PM->Replication activated
Pensine:1/16/2023, 11:24:05 PM->entry removed:test11/note11.md-2-1c5436fff21bf76eedf8b1edcf460f6e

Laptop2: After a while, folder test11 got renamed into test12

Pensine:16/01/2023 23:25:51->Replication activated
Pensine:16/01/2023 23:25:52->Applying 2 files
Pensine:16/01/2023 23:25:52->xxx <- STORAGE (deleted) test11/note11.md
Pensine:16/01/2023 23:25:52->files: 0
Pensine:16/01/2023 23:25:52->All files under the parent directory ([object Object]) have been deleted, so delete this one.
Pensine:16/01/2023 23:25:52->deleteDB By path:test11/note11.md
Pensine:16/01/2023 23:25:52->xxx <- STORAGE (deleted) test11
Pensine:16/01/2023 23:25:52->files: 6
Pensine:16/01/2023 23:25:52->Applied test11/note11.md (2-1c5436fff21bf76eedf8b1edcf460f6e) change...
Pensine:16/01/2023 23:25:52->entry removed:test11/note11.md-3-0e92fb903ec4e33f7cbe8753914dcb21
Pensine:16/01/2023 23:25:52->Replication activated
Pensine:16/01/2023 23:25:52->DB -> STORAGE (create,plain) test12/note11.md
Pensine:16/01/2023 23:25:52->Applied test12/note11.md (1-353358a73fa0d5915c48ba5a20d2e47e) change...

Laptop1: a bit later

I do not understand why it still manages test11...

Pensine:1/16/2023, 11:28:08 PM->Before LiveSync, start OneShot once...
Pensine:1/16/2023, 11:28:08 PM->Oneshot Sync begin... (pullOnly)
Pensine:1/16/2023, 11:28:08 PM->Replication activated
Pensine:1/16/2023, 11:28:08 PM->Replication completed
Pensine:1/16/2023, 11:28:08 PM->LiveSync begin...
Pensine:1/16/2023, 11:28:08 PM->Replication activated
Pensine:1/16/2023, 11:28:08 PM->Replication activated
Pensine:1/16/2023, 11:28:08 PM->Applying 1 files
Pensine:1/16/2023, 11:28:08 PM->Applied test11/note11.md (3-0e92fb903ec4e33f7cbe8753914dcb21) change...
Pensine:1/16/2023, 11:31:05 PM->Replication closed
Pensine:1/16/2023, 11:31:25 PM->Before LiveSync, start OneShot once...
Pensine:1/16/2023, 11:31:25 PM->Oneshot Sync begin... (pullOnly)
Pensine:1/16/2023, 11:31:25 PM->Replication completed
Pensine:1/16/2023, 11:31:25 PM->LiveSync begin...
Pensine:1/16/2023, 11:31:27 PM->Replication closed
Pensine:1/16/2023, 11:33:07 PM->Before LiveSync, start OneShot once...
Pensine:1/16/2023, 11:33:07 PM->Oneshot Sync begin... (pullOnly)
Pensine:1/16/2023, 11:33:07 PM->Replication completed
Pensine:1/16/2023, 11:33:07 PM->LiveSync begin...
Pensine:1/16/2023, 11:33:08 PM->Replication is already in progress.

Closed Obsidian on Laptop2

Closed Obsidian on Laptop1

Started Obsidian on Laptop1

No test11, only test12 👍

Started Obsidian on Laptop2

I re-did several this test and could not get a situation were test11 remained or where both test11 and test12 were deleted.

With "AidenLx's Folder Note" autosync enabled, rename folder14

Laptop1: Create a folder14 with a note folder14/folder14.md

Laptop2: folder14/folder14.md appears

Laptop1: rename the folder14 to folder15

Pensine:1/16/2023, 11:46:48 PM->Content saved:test15/note11.md ,chunks: 1 (new:0, skip:1, cache:0)
Pensine:1/16/2023, 11:46:48 PM->DB <- STORAGE (plain) test15/note11.md
Pensine:1/16/2023, 11:46:48 PM->Replication activated
Pensine:1/16/2023, 11:46:48 PM->entry removed:test14/note11.md-2-14de62709424b359bf9f0e069fe1c4a0
Pensine:1/16/2023, 11:46:48 PM->Content saved:test15/test15.md ,chunks: 2 (new:0, skip:3, cache:1)
Pensine:1/16/2023, 11:46:48 PM->DB <- STORAGE (plain) test15/test15.md
Pensine:1/16/2023, 11:46:48 PM->entry removed:test14/test14.md-3-dbf70d91ec8db1606ae8ca34cae0ecd3
Pensine:1/16/2023, 11:46:49 PM->Replication activated

Laptop2: After a while folder14 is gone and folder15 appears with the right content

Pensine:16/01/2023 23:48:01->Replication activated
Pensine:16/01/2023 23:48:01->xxx <- STORAGE (deleted) test14/note11.md
Pensine:16/01/2023 23:48:01->files: 1
Pensine:16/01/2023 23:48:01->deleteDB By path:test14/note11.md
Pensine:16/01/2023 23:48:01->entry removed:test14/note11.md-3-226159e4174ee235ddb8ca7e881d67e6
Pensine:16/01/2023 23:48:02->Applying 3 files
Pensine:16/01/2023 23:48:02->DB -> STORAGE (create,plain) test15/note11.md
Pensine:16/01/2023 23:48:02->Applied test15/note11.md (1-e2fb5995bc3daa6b64ac315e06b6ad67) change...
Pensine:16/01/2023 23:48:02->xxx <- STORAGE (deleted) test14/test14.md
Pensine:16/01/2023 23:48:02->files: 0
Pensine:16/01/2023 23:48:02->All files under the parent directory ([object Object]) have been deleted, so delete this one.
Pensine:16/01/2023 23:48:02->deleteDB By path:test14/test14.md
Pensine:16/01/2023 23:48:02->xxx <- STORAGE (deleted) test14
Pensine:16/01/2023 23:48:02->files: 7
Pensine:16/01/2023 23:48:02->Applied test14/test14.md (3-dbf70d91ec8db1606ae8ca34cae0ecd3) change...
Pensine:16/01/2023 23:48:02->entry removed:test14/test14.md-4-12525a6392bfd8a7ab1629afc5985658
no afitem found for path test14/test14.md, escaping...
no afitem found for path test14, escaping...
Pensine:16/01/2023 23:48:02->DB -> STORAGE (create,plain) test15/test15.md
Pensine:16/01/2023 23:48:02->Applied test15/test15.md (1-bbd4c758ad94806273a78598b350aa03) change...

With "AidenLx's Folder Note" autosync enabled, rename folder15/folder15.md into folder15/folder16.md which automatically gets renamed into folder16/folder16.md

Note: there is another md file in the folder: note11.md

Laptop1: Rename the folder15/folder15.md into folder15/folder16.md

The "AidenLx's Folder Note" plugin automatically renames folder15/folder16.md into folder16/folder16.md

Pensine:1/16/2023, 11:55:04 PM->Before LiveSync, start OneShot once...
Pensine:1/16/2023, 11:55:04 PM->Oneshot Sync begin... (pullOnly)
Pensine:1/16/2023, 11:55:04 PM->Replication completed
Pensine:1/16/2023, 11:55:04 PM->LiveSync begin...
Pensine:1/16/2023, 11:55:23 PM->Error:ENOENT: no such file or directory, open 'C:\Users\p.hilson\Documents\Obsidian\Pensine\Pensine\test15\test16.md'
Pensine:1/16/2023, 11:55:23 PM->Content saved:test16/note11.md ,chunks: 1 (new:0, skip:1, cache:0)
Pensine:1/16/2023, 11:55:23 PM->DB <- STORAGE (plain) test16/note11.md
Pensine:1/16/2023, 11:55:23 PM->Replication activated
Pensine:1/16/2023, 11:55:23 PM->entry removed:test15/note11.md-2-f4addd968a2887b8bbf8b142deb122fd
Pensine:1/16/2023, 11:55:23 PM->Content saved:test16/test16.md ,chunks: 2 (new:0, skip:3, cache:1)
Pensine:1/16/2023, 11:55:23 PM->DB <- STORAGE (plain) test16/test16.md
Pensine:1/16/2023, 11:55:26 PM->Replication activated
Pensine:1/16/2023, 11:55:26 PM->Applying 1 files
Pensine:1/16/2023, 11:55:26 PM->Applied test15/note11.md (3-598dbe0f5db373ff8e2d95e4efa78723) change...

**At this stage, Laptop1 UI does only show test16.

Laptop2: test16 folder appear with test16.md and note11.md inside. BUT test15 folder appears with test15.md inside (but no note11.md)

Pensine:16/01/2023 23:55:15->TypeError:Failed to fetch
Pensine:16/01/2023 23:55:24->Replication activated
Pensine:16/01/2023 23:55:24->Applying 1 files
Pensine:16/01/2023 23:55:25->DB -> STORAGE (create,plain) test16/note11.md
Pensine:16/01/2023 23:55:25->Applied test16/note11.md (1-77b7a2f625b65fa6c7ba655613125c75) change...
Pensine:16/01/2023 23:55:26->Applying 2 files
Pensine:16/01/2023 23:55:26->xxx <- STORAGE (deleted) test15/note11.md
Pensine:16/01/2023 23:55:26->files: 1
Pensine:16/01/2023 23:55:26->Applied test15/note11.md (2-f4addd968a2887b8bbf8b142deb122fd) change...
Pensine:16/01/2023 23:55:26->deleteDB By path:test15/note11.md
Pensine:16/01/2023 23:55:26->entry removed:test15/note11.md-3-598dbe0f5db373ff8e2d95e4efa78723
Pensine:16/01/2023 23:55:26->Replication activated
Pensine:16/01/2023 23:55:26->DB -> STORAGE (create,plain) test16/test16.md
Pensine:16/01/2023 23:55:26->Applied test16/test16.md (1-ef505e71655d5590a3f8db48c89d2f50) change...

Laptop1: Stop Obsidian

Laptop2: Stop Obsidian

Laptop1: Start Obsidian: test15 folder appear with test15.md inside

Pensine:1/17/2023, 12:06:30 AM->UPDATE STORAGE
Pensine:1/17/2023, 12:06:30 AM->Check or pull from db:test15/test15.md
Pensine:1/17/2023, 12:06:30 AM->DB -> STORAGE (create,plain) test15/test15.md
Pensine:1/17/2023, 12:06:30 AM->Check or pull from db:test15/test15.md OK
Pensine:1/17/2023, 12:06:30 AM->UPDATE STORAGE done.

Laptop2: Start Obsidian

Nothing special happen

[hilsonp]

Sorry for the lengthy report.

In a few words, the problem I reproduce is:

• With "AidenLx's Folder Note" plugin enabled
• With "AidenLx's Folder Note" plugin autosync enabled,
• When renaming the folder's note (eg: folder15/folder15.md => folder15/folder16.md)

Then:

• The plugin renames automatically the folder15 into folder16 (hence: folder15/folder16.md => folder16/folder16.md).
• After obsidian-livesync sync happen
  • Laptop1 has folder16/folder16.md
  • Laptop2 has folder16/folder16.md AND folder15/folder15.md
  • After obsidian restart, Laptop1 has folder16/folder16.md AND folder15/folder15.md

folder15/folder15.md shouldn't exist anywhere at the end.

The other thing I saw in the Laptop2 console were errors about cors but this is maybe unrelated and I will check https://github.com/vrtmrz/obsidian-livesync/issues/111 for this.

[vrtmrz]

Reproduced! Thank you for this information! It's good to have a long one. Nothing to be worried about! So helpful!

It probably will be fixed in a couple of days!

[vrtmrz]

Thank you for your patience! It has been fixed at 0.17.15.

This plugin caches changes and thinned if redundant to apply in batch. However, caches both deleted and created files had pointed to the same file. so it had been got weird. Fixed queueing and caching behaviour.

I am really sorry for bothering you again, but would you mind if I ask for trying v0.17.15 again?

[hilsonp]

We cannot speak about patience when fixes are coming within a few hours 😅 fully explained 😍

You are just incredible! And sure, I will test and provide feedback asap.

Thank you !

[hilsonp]

I changed my passphrase and overwrote my selfhosted couchdb db. I deleted my vault on Laptop2 and my iphone and recreated a new setup URI from Laptop1. I let Laptop1 sync finish with the couchdb.

Then I created the new vault on Laptop1 and the iphone, I installed the livesync plugin and did setup using the setup uri. I then added and enabled the AidenLx's Folder Note (and Folder Note Core) plugin.

iphone

All my tests were successful 👍

Laptop2

It half synched then stopped syncing with a 'cors' error displayed in the logs.

This is strange but because of this, I cannot fully validate your correction. (and maybe you will prefer that this is followed in another ticket ?!?) I restarted Obsidian with the console opened and see that it starts working with the database without error. When I click the replicate button, it displays:

Pensine:19/01/2023 22:03:55->Oneshot Sync begin... (sync)
index.html:1 Access to fetch at 'https://obsidian-livesync.mydomain.be/mydb/' from origin 'app://obsidian.md' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled. plugin:obsidian-livesync:15831

GET https://obsidian-livesync.mydomain.be/mydb/ net::ERR_FAILED
fetch @ plugin:obsidian-livesync:15831
ourFetch @ plugin:obsidian-livesync:11371
fetchJSON @ plugin:obsidian-livesync:11393
setup @ plugin:obsidian-livesync:11422
HttpPouch.api._info @ plugin:obsidian-livesync:11550
eval @ plugin:obsidian-livesync:8695
eval @ plugin:obsidian-livesync:6190
eval @ plugin:obsidian-livesync:6135
eval @ plugin:obsidian-livesync:6124
eval @ plugin:obsidian-livesync:6185
execute @ plugin:obsidian-livesync:8925
ready @ plugin:obsidian-livesync:8936
eval @ plugin:obsidian-livesync:9051
eval @ plugin:obsidian-livesync:11441
Item.run @ plugin:obsidian-livesync:1654
nextTick @ plugin:obsidian-livesync:1626 plugin:obsidian-livesync:19760 

Pensine:19/01/2023 22:03:56->TypeError:Failed to fetch plugin:obsidian-livesync:19760 
Pensine:19/01/2023 22:03:56->could not connect to https://obsidian-livesync.mydomain.be//mydb: TypeError:Failed to fetch
**Note** This error caused by many reasons. The only sure thing is you didn't touch the server.
To check details, open inspector. plugin:obsidian-livesync:19760 

Pensine:19/01/2023 22:03:56->Could not connect to server.

The only thing my pry eyes see is the double // on the line: could not connect to https://obsidian-livesync.mydomain.be//mydb: TypeError:Failed to fetch

But this is probably a type in a printf to the console.

Any idea why I'm getting this 'cors' error ?

---

----remote config----
Requesting information to the remote CouchDB has been failed. If you are using
IBM Cloudant, it is the normal behaviour.

---- Plug-in config ---
couchDB_URI: self-hosted
couchDB_USER: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
liveSync: false
syncOnSave: false
syncOnStart: false
savingDelay: 200
lessInformationInLog: false
gcDelay: 0
versionUpFlash: ""
minimumChunkSize: 20
longLineThreshold: 250
showVerboseLog: false
suspendFileWatching: false
trashInsteadDelete: false
periodicReplication: false
periodicReplicationInterval: 60
syncOnFileOpen: false
encrypt: true
passphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
doNotDeleteFolder: false
resolveConflictsByNewerFile: false
batchSave: false
deviceAndVaultName: ""
usePluginSettings: false
showOwnPlugins: false
showStatusOnEditor: true
usePluginSync: false
autoSweepPlugins: false
autoSweepPluginsPeriodic: false
notifyPluginOrSettingUpdated: false
checkIntegrityOnSave: false
batch_size: 250
batches_limit: 40
useHistory: true
disableRequestURI: true
skipOlderFilesOnSync: true
checkConflictOnlyOnOpen: false
syncInternalFiles: false
syncInternalFilesBeforeReplication: false
syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/
syncInternalFilesInterval: 60
additionalSuffixOfDatabaseName: ""
ignoreVersionCheck: false
lastReadUpdates: 0
deleteMetadataOfDeletedFiles: false
syncIgnoreRegEx: ""
syncOnlyRegEx: ""
customChunkSize: 100
readChunksOnline: true
watchInternalFileChanges: false
automaticallyDeleteMetadataOfDeletedFiles: 0
disableMarkdownAutoMerge: false
writeDocumentsIfConflicted: false
useDynamicIterationCount: false
syncAfterMerge: false
configPassphraseStore: ""
encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
permitEmptyPassphrase: false
workingUseDynamicIterationCount: false

[hilsonp]

And my couchdb local.ini

[couchdb]
single_node=true
max_document_size = 50000000

[chttpd]
require_valid_user = true
max_http_request_size = 4294967296

[chttpd_auth]
require_valid_user = true
authentication_redirect = /_utils/session.html

[httpd]
WWW-Authenticate = Basic realm="couchdb"
enable_cors = true

[cors]
origins = app://obsidian.md,capacitor://localhost,http://localhost
credentials = true
headers = accept, authorization, content-type, origin, referer
methods = GET, PUT, POST, HEAD, DELETE
max_age = 3600

It is behind an nginx that handles the https with a zerossl certificate.

[vrtmrz]

I am wondering why inside ----remote config---- had not been shown if you are using a self-hosted server. If you once have synchronised, the configuration should be correct. (And looks fine) Could you please check what will be shown on DevTools when you selected the errored request?

And, at nginx, we could not set add_header for CORS. May I see your configuration if you do not mind?

[hilsonp]

iphone is syncing nicely over the 4G network (no error in the log and ----remote config---- well reported). This let me thing that the nginx and the couchdb server access is fine. (note: once I'm confident and have a fully working sync, I'll share with you the docker-compose and nginx configuration)

Laptop2 is a corporate computer on which I'm admin but do not control everything. I suspect things being done under the wood like antivirus/proxying, ...

I can see that bitwarden all of a sudden does not sync with my self-hosted vaultwarden server.

I'll do my duty and make sure everything is ok on that side first.

Sorry for the annoyance.

I'll report here once I find the root cause.

[vrtmrz]

I have so relieved to hear that!

LiveSync uses PouchDB for synchronisation. And it will group the stored chunks together to some extent and transfer them in sequence. If by chance, large chunks are then sent together in consecutive parts, CouchDB may reject them. Of course, WAFs such as Akamai may reject consecutive requests, so in practice, it is often difficult to aim to reproduce this problem.

Looking forward to your report!

[hilsonp]

Setup

• CouchDB running as a docker on a Synology listening on standard port 5984
• Nginx reverse proxy running as a docker (SWAG) as well on the Synology and
  • Listening on https://obsidian-livesync.mydomain.be
  • Doing the SSL endpoint, geoip+fail2ban protections and transfering data to the CouchDB
• pfSense router listening on port 443 on my registered domain name and forwarding incomming traffic to the nginx.

Obsidian + obsidian-livesync plugin are installed on:

• Laptop1 (home Win10 laptop)
• Laptop2 (corporate Win10 laptop)
• iPhone

Problem Symptoms

• obsidian-livesync on Laptop1 (home Win10 laptop) and iPhone are synchrozing correctly.
• obsidian-livesync on Laptop2 (corporate Win10 laptop) is not synchronizing correctly (cors errors). The strangiest thing is that it sometimes sync a few things then stop for no reason.

Analysis

Digging further, I found out that, on Laptop2, I had trouble reaching other services running behind nginx (and therefore reached over https using the fqdn.

So for example, if I have another service setup which is reachable on nginx on https://niceservice.mydomain.be, then:

• When Laptop2 was in the office, it could reach the niceservice using any browser.
• From home, Laptop1 and iPhone (hence with a local ip address in the same subnet than the Synology) where able to reach the niceservice using any browser.
• From home, Laptop2 (hence with a local ip address in the same subnet than the Synology) was NOT able to reach the niceservice.

The error seen in the browser was: McAfee Security Gateway - Bad Response -The proxy did not receive a valid response in time.

I first thought it was due to DNS Reflection but the strange thing was that Laptop1 and iPhone were not having the problem.

The problem is linked to McAfee running on my corporate Laptop2 that makes a check with the McAfee Security Gateway running at my company. The Security Gateway invalidates the request for some obscure resolution issues.

Solution

I then tried changing the way DNS resolution occured in my local network for my own services (exposed using xxx.mydomain.be).

I changed for NAT Reflection to Split DNS and all of a sudden I could reach all "hosts" declared in my DNS Resolver/Forwarder Overrides (and allowed in nginx) from Laptop2 (and Laptop1 and iPhone).

Conclusion

I know this problem has nothing to do with obsidian-livesync but I promised you the glory details once I would have solved my problem. Hopefully those can help other people.

Annex: Nginx couchdb configuration

The swag\nginx\proxy-confs\obsidian_livesync.subdomain.conf contains:

# Those are fake uri/ips
# Redirecting http[s]://obsidian_sync.mydomain.be to http://192.168.1.2:5984

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name obsidian-livesync.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for geo blocking
    # See /config/nginx/geoip2.conf for more information.
    if ($allow_list = yes) {
        set $allowed_country yes;
    }
    if ($allowed_country = no) {
        return 444;
    }

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_proto http;
        set $upstream_app 192.168.1.2;
        set $upstream_port 5984;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        proxy_max_temp_file_size 2048m;
    }
}

⚠ I should probably set the proxy_max_temp_file_size to the livesync plugin config. Any suggestion is welcome.

[vrtmrz]

Thank you for your detailed information! It might be great help for everyone! You know that the FAQ is not yet here.

enterprise environment makes us worried. I also had got stuck with the network gateway’s disconnection. Replication made a bit lot of reconnection, and looks like that felt them suspicious.