opensrp / opensrp-server

OpenSRP backend
https://smartregister.atlassian.net/wiki/display/Documentation/OpenSRP+Developer%27s+Guide
Other
24 stars 37 forks source link

Optimize/upgrade couchdb #171

Open maina opened 8 years ago

maina commented 8 years ago

Couchdb is consuming a lot of disk space in the v2 branch code within a couple of days. Optimize and possibly upgrade to couchdb 2.0

sohelsarder commented 8 years ago

Few optimization we are using at Bangladesh site like limit versioning in couchDB. By default it saves all of its updates with different version but we limited it just only latest two.

keymane commented 8 years ago

Okay, that's a great optimization technique. I'm also trying Automatic Compaction where disk space usage will be reduced by removing unused and old data just like SQLite vacuum. See the links below: http://docs.couchdb.org/en/1.6.1/maintenance/compaction.html http://www.sqlite.org/lang_vacuum.html

sohelsarder commented 8 years ago

View files are the evil, its compaction from futon sometimes show space reduce but actual physical storage remain occupied. I did not dig it further but look forward to see some solution of it. I guess couch community should answer of it and latest version may have some fixation on it. Let us know your findings.

keymane commented 8 years ago

okay cool.

keymane commented 8 years ago

So I configured Ona server couchdb instance to trigger compaction automatically and the results are quite impressive...

screen shot 2016-11-08 at 10 02 23 am
mberg commented 8 years ago

We should spend a bit of time and look into switching to couch 2. It supposedly has big improvements in sync performance.

On Tue, Nov 8, 2016 at 10:08 AM, Ephraim (Keyman) Muhia < notifications@github.com> wrote:

So I configured Ona server couchdb instance to trigger compaction automatically and the results are quite impressive...

[image: screen shot 2016-11-08 at 10 02 23 am] https://cloud.githubusercontent.com/assets/13656416/20089939/1c15a230-a59b-11e6-9936-845148f59307.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenSRP/opensrp-server/issues/171#issuecomment-259062461, or mute the thread https://github.com/notifications/unsubscribe-auth/AADiQdinERQCIwpH1og-1yD6BZpIlpWQks5q8B_vgaJpZM4Kq4_O .

keymane commented 8 years ago

okay. I'll check it out

keymane commented 8 years ago

Apache CouchDB 2 can be integrated with OpenSRP. I set it up locally and I was able to sync records between the client app and the server. The only issue that may be a blocker was linking CouchDB 2.0 with couchdb-lunene (used extensively in OpenSRP server for full text search).

According to these github issues couchdb-lucene needs an update to be compatible with 2.0 https://github.com/rnewson/couchdb-lucene/issues/235 https://github.com/rnewson/couchdb-lucene/issues/236

An alternative would be dreyfus https://cloudant.com/blog/enable-full-text-search-in-apache-couchdb/ which uses Apache Lucene 4.x on Java 6 but we use Java 7 which is compatible with couchdb-lucene.

@mberg @maina could we revisit this issue once couchdb-lucene has been updated?