_bulk_docs request exceed Cloudant's 1M limit

[link89]

I use NoteSelf with the IBM Cloudant (free tier). It worked well in the last few months until it started to throw (failed)net::ERR_HTTP2_PROTOCOL_ERROR when calling the _bulk_docs API.

I find the content length of the error request is 54341667 (52M), which exceeds the 1M limit of Cloudant. Is it possible to break down the bulk operation into smaller ones to work around this issue?

[danielo515]

Hello @link89 give us more info about your setup. Are you using the official online version of noteself? If you are using your own build, which tiddlywiki version do you use? What is the size of your wiki? How many tiddlers?

My first guess is that, recently, tiddlywiki just decided to request all the tiddlers to the backend instead of the skinny versions. But that should not increase the number of requests, just the size of them. I have been using cloudant free tier for 7 years now and I never reached such limit, this is weird.

[link89]

Hello @link89 give us more info about your setup. Are you using the official online version of noteself? If you are using your own build, which tiddlywiki version do you use?

I setup my own one by mirroring master branch of this repo. I just update mine so now it should be same as the online verison. But this problem still exists.

What is the size of your wiki? How many tiddlers?

Only 14.7M in total, about 50 tiddlers.

Here is how this error looks like

[danielo515]

Wow, that is quite a big wiki for such small amount of tiddlers. Do you store images on your wiki? Seems that your bulk_docs is becoming too big. This is probably because you are requesting not only your current wiki size but also every version that is stored. I suggest you to run a compactation on your server side, there is no point on keeping versions of images (if that is what you are storign on your wiki). By the way,images are not a god fit for tiddlywiki, because the weight 3x the normal size of the image and are very innefficient.

Regards

[link89]

I do store images in my notebook. I find a work around of this issue by cleaning up the db of my browser, which was about 170M when the problem occurred.

[danielo515]

while I am not sure how that could be causing the problem, with such amount of images your local DB can become quite big because it stores versions of each tiddler, so if you change any image, that will create a copy of that image and the new one.

[link89]

After I clear the site data and sync content from couchdb, the total size of local storage is only 11.8M. I guess there is some kind of leaking there.