search

medic / medic-os

A space-efficient Linux-based operating system, designed specifically to run Medic's mHealth platform. This is the build system and source code; Docker images will be available on Medic's website.
https://medic.org
GNU Affero General Public License v3.0
10 stars 6 forks source link

Update default couchdb configuration #48

Closed garethbowen closed 5 years ago

garethbowen commented 5 years ago

Recently we won a free couchdb configuration evaluation which resulted in some recommendations for improvements we can make. The raw report is available here: https://drive.google.com/drive/folders/1FLQLigNJ7O1msTbTECa5JdGWAPCE_7yd

The changes I recommend we make are:

Low setting for OS process limit

The nodes identified below has have the configuration value [query_server_config] os_process_limit set lower than expected. CouchDB query servers are external processes that help build secondary indexes. The most commonly used CouchDB query server is the couchjs JavaScript engine.

This means that performance may be affected. Setting this value too low can result in starvation of query servers, and manifest in os_process_timeout errors. Setting this parameter too high can potentially use too many system resources, slowing all CouchDB reads and writes.

From the couchdb documentation:

Setting os_process_limit too low can result in starvation of Query Servers, and manifest in os_process_timeout errors, while setting it too high can potentially use too many system resources. Production settings are typically 10-20 times the default value.

Using this recommendation we should change the setting to from 100 to 1,000.

Socket options are misconfigured

Because of a very old CouchDB bug (COUCHDB-1986 and #1409), CouchDB decided to limit the size of Erlang network buffers in such a way that attachment performance was greatly slowed in CouchDB 1.x and 2.x up through the 2.2.x series. The issue was fixed in 2.3.0, but inadvertently created a secondary bug where some requests could not complete successfully, including replication of busy databases (#1810). CouchDB 2.3.1 fully resolved the issue, but only when the new default socket options are used. Prior default settings, as well as workarounds applied for the above issues, are no longer the best settings for CouchDB.

This means that the settings in your installation are outdated, or your CouchDB version is outdated, or both, resulting in reduced network performance. If your CouchDB use includes the use of attachments, performance will be especially bad.

We use attachments extensively and are on 2.3.1 so this is likely a significant issue for us.

Documentation.

Set this to: [{sndbuf, 262144}, {nodelay, true}]

garethbowen commented 5 years ago

I understand this has been built into the latest medic-os and is now ready for AT.

newtewt commented 5 years ago

I am looking at the couchdb configs in fauxton and the settings described here are not the settings I'm finding. I'm pretty sure I'm on the latest medic-os. @Hareet has there been a new image built since the announcement recently?

newtewt commented 5 years ago

Spoke with @Hareet, waiting on a new image from him. Moved back to in progress.

garethbowen commented 5 years ago

Should now be fixed in medic-os:cht-3.7-pre4 in dockerhub.

newtewt commented 5 years ago

Confirmed in the new image the settings are there. Also did a quick run through of the app to ensure replicating and creating reports works still.