File Uploads lead to "unknown database error" (status 500) sometimes

[sleidig]

This bug seems hard to reproduce but on the most important machine (one in the Kolkata office) the error consistently occurs:

Sync between local (PouchDB) and remote server (CouchDB) works fine when changing normal information. After adding a photo however, sync is broken. For any changes (even changes similar to what was successfully synced before) there are errors reported by PouchDB in the Javascript console:

Could not connect to remote database (is CORS enabled?)

(the default error message when remote connections fail)

"unknown database error" (status 500)

[sanindya-iiitd]

Could you please also find out the browser (with version) and the OS being used at the Kolkata office?

[sleidig]

Okay, this seems to be connected to special files or connection problems: I successfully uploaded a picture from the Kolkata office computer but the same error persists with the "real" photos. I suspected corrupted files but those exact files upload just fine from my computer again ...

Maybe this is a problem with large file size when the upload is disturbed in between?

[sleidig]

In any case, we need something to detect upload (attachment) problems in our code and revert to a consistent state if necessary.

Also it could make sense to contract files before adding them to the database - or saving them outside of the database on the server (and just writing the file location into the database).

[sleidig]

maybe this is our error and can be reproduced: https://github.com/pouchdb/pouchdb/issues/4614

or this has something to do with it? https://github.com/pouchdb/pouchdb/issues/3802#issuecomment-100386108

[sanindya-iiitd]

Ok, I'll look into it and see what can be done by me. (How did you find this?)

[sleidig]

Let's split this up into different issues:

• Compress images before upload (#60)
• Save files and images separate from main database (#61)

and this issue itself remains until the upload problem for users in Kolkata finally disappears ...

[fschoell]

Ok, this issue seems to be way more serious than we thought. Just cloned the project on a new machine and I cannot login due to the sync not finishing. This is not only uploading but syncing in general.

It seems that the server doesn't respond to a HTTP request within 10 seconds for some pictures (which probably is the PouchDB timeout). Could it be that the CouchDB server has some strict limits for concurrent connections (I counted ~150 served requests before the sync failed)? This way the larger picture requests are maybe processed slower than PouchDB opens new HTTP requests.

[sleidig]

Yeah, this is getting really serious. As a temporary quick fix I cleaned up the dev (demo-db) database, removing most of the photos.

We should look into those settings concerning timeout / concurrent connections in addition to #61, #60 .

[fschoell]

Another "fix" has been introduced in 7d4f1af8142b605c74c67fe06249f2463f77a495, it's now possible to set the timeout for the pouch db http requests in the app-config

[fschoell]

And there's another thing that came to my mind: How are offline applications handled by the browsers in regards to cache size? Could it become a problem if our database grows too large, so the browsers decide they need some space and delete our offline replication? This would be the absolute worst case if the user hasn't synchronized his work yet.

[sleidig]

created a separate issue for the general question of storage size limitations: #81