Synching with PouchDB with attachments silently fails on Android

[ahodgkinson]

Overview

1. Our Android/Ionic/Angular app fails while syncing data with attachments from a CouchDB server to the local device's PouchDB
2. We are calling PouchDB replicate.from(..) a CouchDB on a server, to an empty local PouchDB
3. There are approximately 220 records, with 90 attachment files, a total of 60 MB, in the CouchDB
4. There are two main symptoms:
   1. Neither the replicate.from() success nor error callback are ever triggered
   2. The app exits with message: "Unfortunately, Issue83 has stopped"

      Environment

Ionic/Cordova/Angular hybrid app, using PouchDB & cordova-plugin-sqlite-2, for Android for phones and tablets

• Android version 4.x desired, but limited to version 5.1+ would be acceptable
• Ionic 1.7.15
• Cordova 6.0.0
• Angular 1.5.5
• PouchDB 5.4.4 and 5.4.5 tested
• Version 1.6.1-1 (on OSX)

  Details

  1. Everything works if there are no attachments in the database
  2. The synch appears to initially run normally and lots of requests can be seen in the CouchDB server's log file
  3. Records without attachments appear to always be synced successfully
  4. The problem always occurs when records with attachments are being synced
  5. The problem often occurs at the 'same place'. E.g. the same number of records are transferred at the time of crash
     • The crash reports an alert with the text: "Unfortunately, Issue83 has stopped"
  6. We have tried changing the sync options: _batch_size: 1, batcheslimit: 1, retry: true
     • This reduces but does not eliminate the error, on some devices
     • We always ensure that only our app is open when we start the sync
  7. The problem is intermittent and appears to depend on the device's capabilities and Android version
     • Newer, higher capacity devices appear to crash less. This makes us suspect it is a resource related problem
     • Restarting the app and the sync sometimes allows us to sync the missing data
  8. The problem goes away completely if we switch from PouchDB's sqlite plugin to the idb adapter
     • But with the idb adapter and larger data sets we hit the Quota limit
  9. Enabling PouchDB logging does not produce any useful log output
  10. Once we have the attachments in the device's PouchDB, we can read and display them. E.g. it appears to work normally
  11. We haven't tested syncing attachments from the device to the CouchDB server

      Alternative

  12. Does anyone knows of a way to enable Android/Chrome's HTML5 "unlimitedStorage" or expand the Quota via JavaScript?
      • See: Managing HTML5 Offline Storage
      • Is so then we could simply switch over to the idb adapter and avoid this problem

        Risk

  13. Given we plan to manage and sync over 1 GB of PouchDB data with attachments, is the SQLite-2 adapter actually considered production ready for our use case?

      Reproducing the Problem

  14. On the CouchDB create a database:
      • Approximately 200 records
      • About 90 of of the records should have an attachment, of 500KB to 8MB (creating all 8MB is probably OK)
      • The total DB size, as reported by CouchDB should be about 60 MB
  15. Download the attached code: All required plugins are included
  16. Make sure you have the following tools (see version numbers above):
      • ionic
      • cordova
  17. Build the android app
      • ionic platform ann android
      • Plug in an Android device (with USD debugging enabled) with a USB cable
      • ionic run android
      • In the application on the device
      • Set the Server URL to the CouchDB server and database.
      • Or set the default in file: www/services/pouchdb.service.js
      • Open the Chrome debugger and display the console output
      • Press the Replicate From
      • The synchronization should start, and depending on the device, it will either hang on crash the app

issue-38.zip

Test App Source Code:

1. Key files:
   • www/services/pouchdb.service.js
   • www/components/home/home.controller.js

[nolanlawson]

Thanks for the reproducible test case! I will take a look at this when I get a chance.

In the meantime I can tell you this: I do not recommend SQLite Plugin 2 for Android for 99% of use cases, and I especially don't recommend it for large binary attachments. In WebSQL and in Cordova plugins, binary data needs to be serialized when sent back and forth between the JavaScript context and the native code, which is enormously inefficient and may lead to crashes if you try to do it all at once.

Whereas in IndexedDB on Android, there is native support for Blobs, and so those Blobs can be directly saved to IndexedDB without ever reading anything into memory in JavaScript. (Try this with a <input type="file">, notice that if you save that Blob/File to PouchDB, then your memory usage inside the JavaScript environment will never increase, even if the file is huge.)

Another issue: since your documents are huge, you may want to use a smaller batch_size when replicating. PouchDB will try to insert all records in a batch into IndexedDB/WebSQL/SQLite at once, which can exceed your memory limitations if you're not careful. Another option is to break up your attachments into smaller attachments and spread them across documents.

Given we plan to manage and sync over 1 GB of PouchDB data with attachments, is the SQLite-2 adapter actually considered production ready for our use case?

No, not unless you break up your attachments into smaller pieces and/or sync with a very low batch size. And unfortunately you probably can't use IndexedDB for that much data, because you'll hit the quota limit very quickly, and the Persistent Storage API has not officially shipped yet in any browser.

If I were you, I would probably not use PouchDB, because I would want more fine-grained control over the attachment sizes and how I sync them. I might store md5 sums in PouchDB, but I would not store attachments. Instead, I would sync those separately using AJAX/fetch and then insert the data directly into SQLite Plugin 2, into as small chunks as possible.

[stefanhuber]

I have a similar problem! We are in the process of trying the following approach:

• Store only hashes inside couchdb
• Downloading files via cordova file transfer plugin (https://github.com/apache/cordova-plugin-file-transfer)

approach 1: PouchDB triggers change events after each document is replicated. Within this event the process of downloading could be done for the files in this document, based on the hash.

approach 2: After the whole pouchDB replication is finished download files.

There are definitely many challenges, e.g.:

• deleting old files manually on file system
• handing bad connections, which often could occur on mobile
• maybe an approach like plupload (http://www.plupload.com/) could be used for large files to chunk them in javascript

[csakbalint]

I had the same problem, PouchDB sync fails silently, when documents are loaded with attachments.

I read the code, and I found out, the problem is with the readAsBinaryString function, more specifically with the FileReader.onloadend function, which wasn't triggered. The solution for this problem was a little bit more obvious and described here:

https://stackoverflow.com/questions/36283445/filereader-not-firing-onloadend-in-ionic-2/43703344

Make sure that polyfills is loaded before cordova!

[nolanlawson]

Closing old issues, please reopen as necessary. Seems this is an issue with ordering of dependencies in user code

[knubie]

Could be related to #46