Replication and Plugin Error When Syncing Larger Vaults

[ghost]

Just following up on a post I'd made on the Obsidian forum (here) over a month ago.

I'd noticed that you'd made several revisions to the plugin since then so thought I'd give it another go today.

Unfortunately, I've found that this issue has not yet been fixed.

---

I've had some time to investigate the problem and have concluded that it occurs only when attempting to sync a larger vault.

• Syncing a small vault (68 .md files across 48 folders, total size of 700KB) works quickly and efficiently across all devices. All changes sync promptly when made.
• Syncing a slightly larger vault (200 files of mixed images and .md across 25 folders, total size of 13.4MB) leads to a replication error.
• Syncing an even larger vault (300 files of mixed images and .md across 45 folders, total size of 67.7MB) leads to a replication error.

From what I'm able to make of the logs, it seems that the plugin attempts to update the database, gets stuck, and then gives up syncing the files without any particular reason. Checking the Cloudant database reveals that only the "obsydian_livesync_version" file is ever synced, and no real vault data makes it across. This problem occurs both if I add all files to the vault in one go to sync in one attempt, or if I slowly add the files to the vault one by one to sync across several attempts.

After the plugin gets stuck, it is frozen in an errored state. Numerous Obsidian restarts and local/remote database initialising and resets do not resolve the problem as long as all files within the large vault are still present. The only way to recover the plugin is to reset the local/remote databases using a small vault, e.g., the 700KB vault mentioned above.

Please see the log below captured after the plugin has entered the errored state:

21/12/2021, 16:52:56->loading plugin
21/12/2021, 16:52:56->Open Database...
21/12/2021, 16:52:56->Database Info
21/12/2021, 16:52:56->{
  "doc_count": 4861,
  "update_seq": 4861,
  "idb_attachment_format": "binary",
  "db_name": "livesync-test",
  "auto_compaction": true,
  "adapter": "idb"
}
21/12/2021, 16:52:56->Database is now ready.
21/12/2021, 16:52:57->Database closed (by close)
21/12/2021, 16:52:57->Open Database...
21/12/2021, 16:52:57->Database Info
21/12/2021, 16:52:57->{
  "doc_count": 4861,
  "update_seq": 4861,
  "idb_attachment_format": "binary",
  "db_name": "livesync-test",
  "auto_compaction": true,
  "adapter": "idb"
}
21/12/2021, 16:52:57->Database is now ready.
21/12/2021, 16:52:57->Initialize and checking database files
21/12/2021, 16:52:57->Updating database by new files
21/12/2021, 16:52:57->UPDATE DATABASE
21/12/2021, 16:52:57->UPDATE STORAGE
21/12/2021, 16:52:57->CHECK FILE STATUS
21/12/2021, 16:52:57->CHECK FILE STATUS : 25/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 50/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 75/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 100/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 125/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 150/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 175/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 200/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 225/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 250/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 275/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 300/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 325/368
21/12/2021, 16:52:57->CHECK FILE STATUS : 350/368
21/12/2021, 16:52:57->Initialized,NOW TRACKING!
21/12/2021, 16:52:57->Replication closed
21/12/2021, 16:52:57->Lock:batchSave:enter
21/12/2021, 16:52:57->Lock:batchSave:aqquired
21/12/2021, 16:52:57->Lock:batchSave:released
21/12/2021, 16:53:25->Replication closed
21/12/2021, 16:53:25->Lock:batchSave:enter
21/12/2021, 16:53:25->Lock:batchSave:aqquired
21/12/2021, 16:53:25->Lock:batchSave:released
21/12/2021, 16:53:27->Replication closed
21/12/2021, 16:53:27->Lock:batchSave:enter
21/12/2021, 16:53:27->Lock:batchSave:aqquired
21/12/2021, 16:53:27->Lock:batchSave:released
21/12/2021, 16:53:35->Lock:batchSave:enter
21/12/2021, 16:53:35->Lock:batchSave:aqquired
21/12/2021, 16:53:35->Lock:batchSave:released
21/12/2021, 16:53:35->Lock:batchSave:enter
21/12/2021, 16:53:35->Lock:batchSave:aqquired
21/12/2021, 16:53:35->Lock:batchSave:released
21/12/2021, 16:53:35->Lock:replicate:enter
21/12/2021, 16:53:35->Lock:replicate:aqquired
21/12/2021, 16:53:35->Replication activated
21/12/2021, 16:53:36->Replication error
21/12/2021, 16:53:36->Lock:replicate:released
21/12/2021, 16:53:37->Error while loading Self-hosted LiveSync
21/12/2021, 16:53:37->Error:Failed to fetch

This is quite a significant problem for me at the moment as it means that I'm unable to sync my entire vault across my devices.

Any help or suggestions would be very much appreciated.

[vrtmrz]

Thank you for the detailed log! Maybe it was caused by a network problem as like the timeout of the replication. Could you open the DevTools (press Ctrl+Shift+i or option+command+i) and check the network window. And check there's any errored request (usually red). If the errored request existed, please send the capture me.

[ghost]

Tested again this morning, it does indeed seem to be a network error. Once the plugin was stuck, I reloaded Obsidian and logged the network window from start-up.

Below is the console log:

app.js:1 Obsidian Developer Console
main.ts:12 Clear Unused Images plugin loaded...
main.ts:18 loading plugin "tasks"
main.ts:400 WikiSpace:22/12/2021, 10:46:59->loading plugin
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Open Database...
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Database Info
main.ts:30 loading markdown-table-editor plugin
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Database is now ready.
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Database closed (by close)
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Open Database...
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Database Info
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Database is now ready.
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Initialize and checking database files
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Updating database by new files
main.ts:400 WikiSpace:22/12/2021, 10:46:59->UPDATE DATABASE
main.ts:400 WikiSpace:22/12/2021, 10:46:59->UPDATE STORAGE
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 25/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 50/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 75/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 100/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 125/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 150/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 175/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 200/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 225/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 250/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 275/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 300/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 325/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 350/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 375/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 400/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 425/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 450/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 475/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 500/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 525/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 550/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->CHECK FILE STATUS : 575/575
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Initialized,NOW TRACKING!
main.ts:400 WikiSpace:22/12/2021, 10:46:59->Replication closed
main.ts:400 WikiSpace:22/12/2021, 10:47:00->Replication activated
pouchdb-browser.js:1 POST https://REDACTED.cloudant.com/sync/_bulk_docs net::ERR_HTTP2_PROTOCOL_ERROR
a @ pouchdb-browser.js:1
c @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
r._bulkDocs @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
q @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
C @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
p.run @ pouchdb-browser.js:1
l @ pouchdb-browser.js:1
IndexedDB (async)
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
c @ pouchdb-browser.js:1
main.ts:400 WikiSpace:22/12/2021, 10:47:01->Replication error
main.ts:400 WikiSpace:22/12/2021, 10:47:01->Error while loading Self-hosted LiveSync

[vrtmrz]

Thank you! Perhaps I found why it occurs. (is response code 413?). In that phase, the browser and the server send and receive the diff of each other. But cloudant doesn't accepts the request that over 10MB.

I haven't been able to reproduce it, so I'll check it.

[ghost]

That does make a lot of sense, and would also explain why I was receiving the same error when trying to sync attachments above 10MB (here). As far as I'm aware, the attachment issue is also still present.

As the 10MB request limit is a hard limit imposed by Cloudant, I wonder if it might be possible to have requests larger than 10MB split into smaller chunks? I can see this becoming a large problem in several scenarios otherwise, e.g.:

• Making many large changes to a vault at once.
• Attempting to sync large attachments, such as PDFs, above 10MB.
• Syncing an entire vault (of total size above 10MB) to a new device.
• Syncing to a device which doesn't get used very often.
• Editing several files on one device, adding large attachments, and then syncing to another device at a later point.

While this all could be overcome by self-hosting a CouchDB instance and lifting the request limit, there will be users who do not have the hardware facilities or infrastructure to do this, and so will be reliant on Cloudant for syncing.

[ghost]

After some further testing, I've found that the limit is actually somewhere around 7MB in this use case rather than 10MB. The maximum I've been able to sync to Cloudant in one go without the plugin becoming stuck is just under 7MB.

I've just about managed to sync a portion of my vault across my devices using a workaround. The steps are as follows:

1. Add a chunk of data around ~1-1.5MB in size to the vault of device one.
2. Open Obsidian and have the plugin sync the data to Cloudant.
3. Close Obsidian on device one and open Obsidian on device two.
4. Have the plugin download the data from Cloudant.
5. Close Obsidian on device two and repeat for all devices until the vaults are up-to-date on each.
6. Repeat steps 1-5 until all data has been transferred to all devices.

1-1.5MB chunks seem to be most reliable to ensure efficient data transfer without the plugin throwing out any other errors. It's also important to keep a delay of ~20 seconds minimum between sync attempts to prevent overload.

The above workaround is currently the only way I've found to sync a database larger than ~7MB in total size. It's very tedious though, which makes using the plugin quite frustrating. There's also a constant sense of nervousness while using this workaround, in case the plugin suddenly becomes stuck on one device. If this happens, the only way to recover the plugin is to reset the local and remote databases on all devices and start from scratch.

---

Perhaps having the plugin detect a difference of 7MB+ between the remote and local databases, and then automate this process of chunking and receiving/transmitting ~1-1.5MB of data until syncing is complete would resolve the problem?

The time taken to sync would be far longer, but the operation would be significantly more reliable.

[vrtmrz]

I'm sorry for your frustrations... But after some investigations, I found the workaround!

Self-hosted LiveSync using PouchDB and synchronized with the remote by this protocol. And, it splits every entry into chunks to be acceptable by the database with limited payload size and document size.

However, it was not enough. According to 2.4.2.5.2. Upload Batch of Changed Documents in Replicate Changes, it might become a bigger request.

Unfortunately, there is no way to deal with this automatically by size for every request. Therefore, I made it possible to configure this at version 0.3.0.

If you set these values lower number, the number of requests will increase.
Therefore, if you are far from the server, the total throughput will be low, and the traffic will increase. But it must be better than could not sync.

I tried with IBM Cloudant with the large vault (200MB, 963 files)

Batch size: 50
Batch limit: 10

was work well.

Could you please try it?

[ghost]

Just updated and gave it another go with those settings. The replication process seems to get further than before, but still fails, albeit with a different error.

Here is the new log:

Obsidian Developer Console
main.ts:12 Clear Unused Images plugin loaded...
main.ts:18 loading plugin "tasks"
main.ts:400 WikiSpace:23/12/2021, 12:41:05->loading plugin
main.ts:400 WikiSpace:23/12/2021, 12:41:05->Open Database...
main.ts:400 WikiSpace:23/12/2021, 12:41:05->Database Info
main.ts:30 loading markdown-table-editor plugin
main.ts:400 WikiSpace:23/12/2021, 12:41:05->{
  "doc_count": 9562,
  "update_seq": 9563,
  "idb_attachment_format": "binary",
  "db_name": "WikiSpace-livesync",
  "auto_compaction": true,
  "adapter": "idb"
}
main.ts:400 WikiSpace:23/12/2021, 12:41:05->Database is now ready.
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Database closed (by close)
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Open Database...
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Database Info
main.ts:400 WikiSpace:23/12/2021, 12:41:06->{
  "doc_count": 9562,
  "update_seq": 9563,
  "idb_attachment_format": "binary",
  "db_name": "WikiSpace-livesync",
  "auto_compaction": true,
  "adapter": "idb"
}
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Database is now ready.
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Initialize and checking database files
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Updating database by new files
main.ts:400 WikiSpace:23/12/2021, 12:41:06->UPDATE DATABASE
main.ts:400 WikiSpace:23/12/2021, 12:41:06->UPDATE STORAGE
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 25/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 50/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 75/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 100/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 125/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 150/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 175/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 200/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 225/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 250/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 275/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 300/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 325/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 350/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 375/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 400/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 425/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 450/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 475/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 500/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 525/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 550/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->CHECK FILE STATUS : 575/577
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Initialized,NOW TRACKING!
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Replication closed
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Lock:batchSave:enter
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Lock:batchSave:aqquired
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Lock:batchSave:released
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Lock:replicate:enter
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Lock:replicate:aqquired
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Sync Main Started
main.ts:400 WikiSpace:23/12/2021, 12:41:06->Replication activated
index.html:1 Access to fetch at 'https://REDACTED.cloudant.com/sync/_local/t5YhN9cP3c0CaZjHTYZMew%3D%3D' from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
pouchdb-browser.js:1 PUT https://REDACTED.cloudant.com/sync/_local/t5YhN9cP3c0CaZjHTYZMew%3D%3D net::ERR_FAILED
a @ pouchdb-browser.js:1
c @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
r._put @ pouchdb-browser.js:1
u @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
Hn @ pouchdb-browser.js:1
Xn.updateSource @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
Xn.writeCheckpoint @ pouchdb-browser.js:1
S @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
C @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
p.run @ pouchdb-browser.js:1
l @ pouchdb-browser.js:1
main.ts:400 WikiSpace:23/12/2021, 12:41:08->Replication error
main.ts:400 WikiSpace:23/12/2021, 12:41:08->Error:Failed to fetch
main.ts:400 WikiSpace:23/12/2021, 12:41:08->Lock:replicate:released
index.html:1 Access to fetch at 'https://REDACTED.cloudant.com/sync/_local/tv.MvfiFhWilXlToKC6eSg%3D%3D' from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
pouchdb-browser.js:1 PUT https://REDACTED.cloudant.com/sync/_local/tv.MvfiFhWilXlToKC6eSg%3D%3D net::ERR_FAILED

[ghost]

After some further testing, with batch size: 200 and batch limit: 40, I'm able to successfully sync a larger proportion of my vault than I was initially able to without errors - 368 files, 94 folders, 68.2MB total size.

If I try to sync the entire vault, I receive the above fetch error regardless of whether I use batch size: 200 and batch limit: 40, or batch size: 50 and batch limit: 10 which you had suggested - 576 files, 126 folders, 81.7MB total size.

I should also point out that with batch size: 50 and batch limit: 10, I receive the same error even while trying to sync a small vault - 69 files, 48 folders, 500KB total size.

[vrtmrz]

This would be reached the access frequency restrictions for IBM Cloudant. But it's not the same situation as before, The replication is progressed little by little.

[kmod-midori]

I would assume that large binary files in a vault are not changed frequently (are we doing diff on them anyway?). If that's the case, is it possible to store binary files on places such as S3 or some S3-compatible storage services (some are quite cheap and affordable) and only store a hash/URL to that file in the database?

[vrtmrz]

@chengyuhui

Yes, Self-hosted LiveSync check the file in the four cases

• When Obsidian says "New file has been created"
• When Obsidian says "This file has been changed"
• When the file on the storage and its on the database have different last modified times in the boot-up scan
• When the file is replicated.

In this scene, LiveSync checks its contents and if changed, updates to the latest version, if the changes are not conflicted. If conflicted, the merging dialog will be shown and we have to choose.

By the way, some types of attachments won't be changed so frequently. Therefore, your idea has a good point. Adding minio or something like a server into my container would be great. But perhaps it is not easy for the users that just want to use this plugin: create an AWS account and make the settings sure in addition to creating a Cloudant account.

I think it is a very useful feature, in other software do like this. Outlook saves the attachment into SharePoint and links automatically.

But I think it is out of the scope of my plugin. This feature could be realized and be useful even without Self-hosted LiveSync.

[kmod-midori]

Are we actually comparing the contents of binary files ?

My particular use case is that my self-hosted database server is quite limited in bandwidth (~5Mbps, although latency is quite good), S3 (or even S3 + Cloudflare or other CDN provider) can be a great solution if the user has large files to be synced. After all, one is not supposed to store several gigabytes of data in a database.

[vrtmrz]

You right. It will be useful that mark large files and synchronize them to another place, like Git LFS.

Using a bucket is a good idea, but there are a few limitations. We need the server that proxies requests between obsidian and AWS. (I checked docs of AWS, OCI, GCP, these bucket servers couldn't set up the CORS origin to use from obsidian). I think It's the hard way. If I found a good way to make it realized with fewer servers, I will implement it! Thank you.

And even the file is binary, it is split into chunks and dedupe chunks all through the vault. So if your file doesn't rewrite completely, store and synchronize effectively. PDF or some appendable file have matched this situation.

[kmod-midori]

https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html Setting "AllowedOrigins": ["*"] should be fine? It's not optimal, but AWS probably won't allow capacitor:// in their configuration.

[kmod-midori]

Also, according to denolehov/obsidian-git#57, CORS can be bypassed with Obsidian's API.

[vrtmrz]

If set allowed origins to "*", usually access-control-allow-credentials can not be true.

But referred some topics in StackOverflow, AWS may respond as echo back the request's origin. I have to try this.

"request" API can treat only text data. but perhaps it's enough for this use case. (Sorry, I dismissed this API once and completely forgot. It couldn't be able to synchronize)

Could you mind if I make this a new issue?

[kmod-midori]

Yes, feel free to do so.

[vrtmrz]

@kwjfef I think the problem with the synchronization not completing will be solved by some re-runs of the replication. How about it? (Even it has been stopped by Cloudant request limit, next synchronization will start from the continuation.) And once synchronized, you can increase the value again.

[ghost]

I have tried this, but it's impossible as the plugin refuses to sync at all after the replication/fetch error.

The only way to recover is by resetting the local and remote databases on all devices and starting the replication process from scratch.

[vrtmrz]

Oh, thanks... If batch size is not effect to solve this, perhaps the chunk size hit the limit. May I ask content-length of the request that error has occured?

[ghost]

Could you clarify what you mean by content-length?

[vrtmrz]

Sorry, "content-length" of "Request Headers" in the errored request. (573 in below request) .

[ghost]

Perfect, thanks!

For the _bulk_docs error that occurs when using a batch size of 250 and batch limit of 40, the content-length is 16561431. For the CORS error that occurs when using a batch size of 50 and a batch limit of 10, there's actually two content-length values. One is 1856 and the other is 380.

[vrtmrz]

Thanks too! 16561431 of _bulk_docs is 16MB. We have to set a lower batch size. CORS error is a little different. Could you please show me the request header, response header, request, and response? I'm sorry for asking so many questions over and over again.

[ghost]

No worries! I'm more than happy to help. I'd love to see this problem get resolved.

Using a batch size of 50 and a batch limit of 10, after 10 refreshes the plugin finally stops erroring and syncs without any problems. With each refresh, the content-length for the request and response headers change: CORS error: Refresh 1 Response - 109 Request - 240

CORS error: Refresh 2 Response - 109 Request - 1801

CORS error: Refresh 3 Response - 109 Request - 380

CORS error: Refresh 4 Response - 109 Request - 3520

CORS error: Refresh 5 Response - 109 Request - 520

CORS error: Refresh 6 Response - 109 Request - 3603 AND _bulk_docs: Refresh 1 Response - 109 Request - 19407

_bulk_docs: Refresh 2 Response - 109 Request - 25961 and so on until after 10 refreshes it starts to work.

[vrtmrz]

Thank you so much! I think we're getting closer to a solution. I'm sorry for absent and hasty reply, but may I get full capture of the CORS error? (Any single one of them is fine. and please hide your personal information)

[ghost]

app.js:1 Obsidian Developer Console
main.ts:462 syncmodel:03/01/2022, 12:08:17->loading plugin
main.ts:462 syncmodel:03/01/2022, 12:08:17->Open Database...
main.ts:462 syncmodel:03/01/2022, 12:08:17->Database Info
main.ts:462 syncmodel:03/01/2022, 12:08:17->Database is now ready.
main.ts:462 syncmodel:03/01/2022, 12:08:17->Database closed (by close)
main.ts:462 syncmodel:03/01/2022, 12:08:17->Open Database...
main.ts:462 syncmodel:03/01/2022, 12:08:17->Database Info
main.ts:462 syncmodel:03/01/2022, 12:08:18->Database is now ready.
main.ts:462 syncmodel:03/01/2022, 12:08:18->Initialize and checking database files
main.ts:462 syncmodel:03/01/2022, 12:08:18->Updating database by new files
main.ts:462 syncmodel:03/01/2022, 12:08:18->UPDATE DATABASE
main.ts:462 syncmodel:03/01/2022, 12:08:18->UPDATE STORAGE
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 25/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 50/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 75/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 100/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 125/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 150/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 175/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 200/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 225/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 250/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 275/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 300/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 325/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 350/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 375/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 400/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 425/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 450/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 475/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 500/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 525/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 550/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->CHECK FILE STATUS : 575/582
main.ts:462 syncmodel:03/01/2022, 12:08:18->Initialized,NOW TRACKING!
main.ts:462 syncmodel:03/01/2022, 12:08:18->Replication closed
main.ts:462 syncmodel:03/01/2022, 12:08:18->Sync Main Started
main.ts:462 syncmodel:03/01/2022, 12:08:19->Replication activated
pouchdb-browser.js:1 GET https://REDACTED-bluemix.cloudant.com/syncmodel/_local/REDACTED? 404
a @ pouchdb-browser.js:1
c @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Hn @ pouchdb-browser.js:1
Xn.updateSource @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
Xn.writeCheckpoint @ pouchdb-browser.js:1
S @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
C @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
p.run @ pouchdb-browser.js:1
l @ pouchdb-browser.js:1
pouchdb-browser.js:1 The above 404 is totally normal. PouchDB is just checking if a remote checkpoint exists.
main.ts:462 syncmodel:03/01/2022, 12:08:19->replicated 1 doc(s)
index.html:1 Access to fetch at 'https://REDACTED-bluemix.cloudant.com/syncmodel/_local/REDACTED' from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
pouchdb-browser.js:1 PUT https://REDACTED-bluemix.cloudant.com/syncmodel/_local/REDACTED net::ERR_FAILED
a @ pouchdb-browser.js:1
c @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
r._put @ pouchdb-browser.js:1
u @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
Promise.then (async)
Hn @ pouchdb-browser.js:1
Xn.updateTarget @ pouchdb-browser.js:1
Xn.writeCheckpoint @ pouchdb-browser.js:1
S @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
Promise.then (async)
E @ pouchdb-browser.js:1
C @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
p.run @ pouchdb-browser.js:1
l @ pouchdb-browser.js:1
IndexedDB (async)
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
eval @ pouchdb-browser.js:1
c @ pouchdb-browser.js:1
main.ts:462 syncmodel:03/01/2022, 12:08:20->Replication error
main.ts:462 syncmodel:03/01/2022, 12:08:20->Error:Failed to fetch

[vrtmrz]

@kwjfef

Thank you for the detailed log. It was so helpful. There were two problems, request size, and checkpoint creation.

It's probably fixed in v0.4.1. And you don't have to set the batch size and batch limit. You can set these values back to default. If the request was failed (or would be failed) with the larger request, the plugin retries with the half-size batch configurations. And, the plugin doesn't use the remote checkpoint.

Could you please try this version?

[ghost]

I've just given 0.4.1 a try and I can confirm that everything works perfectly! My entire vault synced flawlessly without any problems. I then tried challenging the plugin, attempting to sync a batch of MP3 files and a ~300MB MP4. All of these eventually synced, and all automatically. Great work @vrtmrz!

I have noticed that the time to sync singular very large files (e.g., the ~300MB MP4) can be quite long, but I suppose there's not much that can be done about that. As I said before, I'd much rather have reliable syncing than fast syncing and that's exactly what you've done.

[vrtmrz]

@kwjfef I'm very happy to hear that!

Yes. If you are far from CouchDB's instance (logically), increasing request counts slows down the replication. If you want to speed up, you'd like to host your CouchDB or use the Cloudant's region which is near to you.

[vrtmrz]

Close this issue once. If there's a new problem, please re-open or create a new issue.