Performance regression on CouchDB v3 while using custom reduce function

I was trying to upgrade my database installation from CouchDB v2.3.1 to v3.1.1 and experienced about 6 time slower performance on particular view invocations.

Description

Steps to Reproduce

To make it easier to reproduce, I created a script which sets up CouchDB 2.3.1 and CouchDB 3.1.1 in docker/podman, generates example data and imports the problematic view. The test script generates 12000 documents and a single design document including the slow view. The script can be used to measure response times against the local system.

2021_04_19-couchdb-test.zip

Show usage

$ ./couchdb-test.sh 
Usage: couchdb-test.sh setup|query|cleanup

Initial setup of scenario to reproduce

The following step takes about 13 min to run, the data generation is not very efficient with curl, but I did not want to force to much requirements on the host except of having bash, curl and docker/podman installed.

$ ./couchdb-test.sh setup
CouchDB performance regression test script
Initializing clean CouchDB setups
Cleanup containers if present
Error: failed to evict container: "": failed to find container "couchdb2" in state: no container with name or ID couchdb2 found: no such container
Create container
e0077b6673c9751053d4c9613c38c5de0015a283315a3dd4f125a6242de835af
Cleanup containers if present
Error: failed to evict container: "": failed to find container "couchdb3" in state: no container with name or ID couchdb3 found: no such container
Create container
d114d4a2e6eb28f8f322e85083e46aa033701206a13c1789c6beccc8d457f656
Setup CouchDB2
Waiting for CouchDB2 to become available at http://localhost:3002...
CouchDB is up and running
Creating _users
{"ok":true}
Creating _replicator
{"ok":true}
Creating _global_changes
{"ok":true}
Creating demo
{"ok":true}
Setup CouchDB3
Waiting for CouchDB2 to become available at http://localhost:3003...
CouchDB is up and running
Creating _users
{"ok":true}
Creating _replicator
{"ok":true}
Creating _global_changes
{"ok":true}
Creating demo
{"ok":true}
Creating design doc
{"ok":true,"id":"_design/demo","rev":"1-6732411a147da115d79425ba6aeb71de"}
Creating design doc
{"ok":true,"id":"_design/demo","rev":"1-6732411a147da115d79425ba6aeb71de"}
Generating data
inserting docs cycle 1/2000
{"ok":true,"id":"5d933844b700c58fc1b422deb0000687","rev":"1-8353a76e296fb10a71bfb93b7b0ae73f"}
{"ok":true,"id":"5d933844b700c58fc1b422deb0000cf3","rev":"1-60f7912abd240b2433b1ea010f0699c4"}
{"ok":true,"id":"5d933844b700c58fc1b422deb0001801","rev":"1-1a83628882bcb89401df77e7fcc42fd1"}
{"ok":true,"id":"5d933844b700c58fc1b422deb0002000","rev":"1-8a91cdc871ffb7623369ab81d2b6e980"}
{"ok":true,"id":"5d933844b700c58fc1b422deb00023ce","rev":"1-66ac9947820d8f32ca321a91d11b8794"}
...
inserting docs cycle 1999/2000
{"ok":true,"id":"01594d00a265804fd144a3e55575806e","rev":"1-6620f7ce5b499af7486ba404b43fcf6a"}
{"ok":true,"id":"01594d00a265804fd144a3e555758337","rev":"1-6595ca907b8e04dd56f83c56a1ccb5ee"}
{"ok":true,"id":"01594d00a265804fd144a3e5557584f1","rev":"1-31bfdc81ae9ab7c8d5eb16007f479309"}
{"ok":true,"id":"01594d00a265804fd144a3e555758f9e","rev":"1-b3109111048a9a14bce96b1a6a5df642"}
{"ok":true,"id":"01594d00a265804fd144a3e555759188","rev":"1-37fc4bd5012c2e90c13324cc7911d053"}
{"ok":true,"id":"01594d00a265804fd144a3e555759880","rev":"1-7457236eacd2fc5742491a431bccabdb"}
inserting docs cycle 2000/2000
{"ok":true,"id":"01594d00a265804fd144a3e55575a0a7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"01594d00a265804fd144a3e55575ab65","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"01594d00a265804fd144a3e55575b4de","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"01594d00a265804fd144a3e55575c456","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"01594d00a265804fd144a3e55575cbcb","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"01594d00a265804fd144a3e55575d758","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}

Query performance

$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"9772359d7167d42054df2c9bca1388c9","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10

real    0m6.258s
user    0m0.013s
sys 0m0.012s
Round: 2/10

real    0m6.526s
user    0m0.014s
sys 0m0.011s
Round: 3/10

real    0m6.477s
user    0m0.010s
sys 0m0.015s
Round: 4/10

real    0m6.156s
user    0m0.011s
sys 0m0.014s
Round: 5/10

real    0m6.185s
user    0m0.015s
sys 0m0.011s
Round: 6/10

real    0m6.082s
user    0m0.015s
sys 0m0.015s
Round: 7/10

real    0m5.932s
user    0m0.012s
sys 0m0.014s
Round: 8/10

real    0m5.594s
user    0m0.013s
sys 0m0.013s
Round: 9/10

real    0m6.258s
user    0m0.012s
sys 0m0.014s
Round: 10/10

real    0m5.796s
user    0m0.009s
sys 0m0.017s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"5f8a19fe389af901b8044f541b0d3c9f","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10

real    0m42.177s
user    0m0.012s
sys 0m0.016s
Round: 2/10

real    0m39.650s
user    0m0.013s
sys 0m0.016s
Round: 3/10

real    0m40.073s
user    0m0.013s
sys 0m0.014s
Round: 4/10

real    0m40.407s
user    0m0.011s
sys 0m0.017s
Round: 5/10

real    0m39.927s
user    0m0.013s
sys 0m0.015s
Round: 6/10

real    0m38.747s
user    0m0.012s
sys 0m0.016s
Round: 7/10

real    0m38.286s
user    0m0.014s
sys 0m0.013s
Round: 8/10

real    0m38.962s
user    0m0.015s
sys 0m0.014s
Round: 9/10

real    0m37.821s
user    0m0.017s
sys 0m0.012s
Round: 10/10

real    0m37.505s
user    0m0.015s
sys 0m0.013s

Cleanup

$ ./couchdb-test.sh cleanup
CouchDB performance regression test script
CONTAINER ID  IMAGE                            COMMAND               CREATED        STATUS            PORTS                   NAMES
b885253d64ea  docker.io/library/couchdb:3.1.1  /opt/couchdb/bin/...  2 minutes ago  Up 2 minutes ago  0.0.0.0:3003->5984/tcp  couchdb3
908b29168c7e  docker.io/library/couchdb:2.3.1  /opt/couchdb/bin/...  2 minutes ago  Up 2 minutes ago  0.0.0.0:3002->5984/tcp  couchdb2
Removing containers couchdb2 and couchdb3 if present
908b29168c7e35b6a0dda9d2723c54ff972e44b8467ccd05b28421d033c48a9d
b885253d64ea11a40cbb2401d4f9e04d6a34cdffcf375c37eb989d19a4f9db3f
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

$ podman rmi docker.io/library/couchdb:2.3.1 docker.io/library/couchdb:3.1.1

Expected Behaviour

The performance of the view should not decrease, or at least not at that magnitude.

Your Environment

CouchDB version used: v2.3.1 and v3.1.1 (official unmodified Container images from https://hub.docker.com/_/couchdb)
Browser name and version: curl
Operating system and version: I was running the script on RHEL 8.3 with podman version 2.2.1.

Additional Context

Used map function:

function (doc) {
    if(doc.object === "ainst" && doc.result) {
        var subscription = doc.subscription || doc.tenant;
        var stats = {
            successfulCount : 0,
            failedCount : 0,
            canceledCount : 0,
            executingCount : 0,
            unsuccessfulCount : 0,
            unknownCount : 0,
            totalCount : 1,
            execTimeSum : 0,
            execTimeCount : 0,
            execTimeMin : Infinity,
            execTimeMax : 0
        }

        if (doc.result.status === "successful") {
            stats.successfulCount = 1;

            if (doc.result.executionTime) {
                stats.execTimeSum = doc.result.executionTime;
                stats.execTimeCount = 1;
                stats.execTimeMin = doc.result.executionTime;
                stats.execTimeMax = doc.result.executionTime;
            }
        } else if (doc.result.status === "failed") {
            stats.failedCount = 1;
        } else if (doc.result.status === "canceled") {
            stats.canceledCount = 1;
        } else if (doc.result.status === "executing") {
            stats.executingCount = 1;
        } else if (doc.result.status === "unsuccessful") {
            stats.unsuccessfulCount = 1;
        } else if (doc.result.status === "unknown") {
            stats.unknownCount = 1;
        }

        emit([subscription, doc.aDefId], stats);
    }
}

Custom reduce function:

function (keys, values, rereduce) {
    var stats = {
        successfulCount : 0,
        failedCount : 0,
        canceledCount : 0,
        executingCount : 0,
        unsuccessfulCount : 0,
        unknownCount : 0,
        totalCount : 0,
        execTimeSum : 0,
        execTimeCount : 0,
        execTimeMin : Infinity,
        execTimeMax : 0
    }

    for (var index in values) {
        stats.successfulCount += values[index].successfulCount;
        stats.failedCount += values[index].failedCount;
        stats.canceledCount += values[index].canceledCount;
        stats.executingCount += values[index].executingCount;
        stats.unsuccessfulCount += values[index].unsuccessfulCount;
        stats.unknownCount += values[index].unknownCount;
        stats.totalCount += values[index].totalCount;
        if (values[index].execTimeCount) {
            stats.execTimeSum += values[index].execTimeSum;
            stats.execTimeCount += values[index].execTimeCount;
            stats.execTimeMin = Math.min(stats.execTimeMin, values[index].execTimeMin);
            stats.execTimeMax = Math.max(stats.execTimeMax, values[index].execTimeMax);
        }
    }

    return stats;
}

CouchDB v2.3.1 logs

[notice] 2021-04-19T15:15:56.957812Z nonode@nohost <0.8597.3> ea0847955d 127.0.0.1:3002 10.0.2.100 admin GET / 200 ok 1
[info] 2021-04-19T15:15:57.024251Z nonode@nohost <0.8708.3> -------- Starting index update for db: shards/20000000-3fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:15:57.024263Z nonode@nohost <0.8710.3> -------- Starting index update for db: shards/00000000-1fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:15:57.027396Z nonode@nohost <0.8714.3> -------- Starting index update for db: shards/80000000-9fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:15:57.027402Z nonode@nohost <0.8716.3> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:15:57.027414Z nonode@nohost <0.8717.3> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:15:57.027510Z nonode@nohost <0.8720.3> -------- Starting index update for db: shards/60000000-7fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:15:57.033749Z nonode@nohost <0.8726.3> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:15:57.037107Z nonode@nohost <0.8729.3> -------- Starting index update for db: shards/40000000-5fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.545524Z nonode@nohost <0.8714.3> -------- Index update finished for db: shards/80000000-9fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.586724Z nonode@nohost <0.8708.3> -------- Index update finished for db: shards/20000000-3fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.602767Z nonode@nohost <0.8710.3> -------- Index update finished for db: shards/00000000-1fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.621844Z nonode@nohost <0.8720.3> -------- Index update finished for db: shards/60000000-7fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.634404Z nonode@nohost <0.8716.3> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.639086Z nonode@nohost <0.8729.3> -------- Index update finished for db: shards/40000000-5fffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.642827Z nonode@nohost <0.8726.3> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1618843969 idx: _design/demo
[info] 2021-04-19T15:16:00.660120Z nonode@nohost <0.8717.3> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1618843969 idx: _design/demo
[notice] 2021-04-19T15:16:04.871512Z nonode@nohost <0.8598.3> 0ff4f8d0e3 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 7901
[notice] 2021-04-19T15:16:09.628157Z nonode@nohost <0.8904.3> 7c47c98e62 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4749
[notice] 2021-04-19T15:16:14.377989Z nonode@nohost <0.9008.3> 15f2ef83f8 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4742
[notice] 2021-04-19T15:16:19.214627Z nonode@nohost <0.9086.3> c1231fddeb 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4830

CouchDB v3.1.1 logs

[notice] 2021-04-19T15:16:50.323986Z nonode@nohost <0.16853.3> 105c664a0b 127.0.0.1:3003 10.0.2.100 admin GET / 200 ok 1
[notice] 2021-04-19T15:17:26.534828Z nonode@nohost <0.16854.3> e06cf40a83 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 36202
[notice] 2021-04-19T15:18:05.599146Z nonode@nohost <0.17182.3> 3db1f4acb4 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 39056
[notice] 2021-04-19T15:18:44.043762Z nonode@nohost <0.17518.3> 4b7c9097f9 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 38437

It might be related to the change in the JavaScript engine in CouchDB 3.0, but I am just guessing https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-update-to-javascript-engine/

Thanks in advance for any help Konrad

Friendly reminder, if anyone could have a look, please? It is blocking us to upgrade to CouchDB v3 at the moment.

3.x decreased the default sharding factor (Q) from 8 to 2. I noticed your test script creates dbs with a default sharding factor 2 and before it was 8, and that might impact indexing times. Try explicitly passing ?q=8 on db creation.

Also 3.x has an index auto-builder and auto-compactor and those might be working the background while the test run and consume some resources.

Thanks @nickva for investigating the issue. I tried to adjust the function from

setupDbs() {
    COUCHDB_PORT=${1}
    waitForDb "${COUCHDB_PORT}"
    SYSTEM_DBS=( _users _replicator _global_changes demo )
    for CURRENT_DB in "${SYSTEM_DBS[@]}"
    do
        echo "Creating ${CURRENT_DB}"
        curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}"
    done
}

setupDbs() {
    COUCHDB_PORT=${1}
    waitForDb "${COUCHDB_PORT}"
    SYSTEM_DBS=( _users _replicator _global_changes demo )
    for CURRENT_DB in "${SYSTEM_DBS[@]}"
    do
        echo "Creating ${CURRENT_DB}"
        curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}?q=8"
    done
}

Unfortunately that did not solve the performance degradation of CouchDB 3.1.1 compared to v2.3.1.

I checked that the databases are using the same number of shards:

CouchDB 2:

$ curl -u "xxx:xxx" http://localhost:3002/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgYg9f____lZiQx41R2AqLuPW11SApBMqgcqImTWA4hZeNQlKYDMsifCrAUQs_bjVpfIkCSP3zIHkGXxEDVZAMg5YvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjMlMiTJ____PyuJgYH1Cw41SQpAMskeqoyNFZcyB5CyeJhpD3EpSwApq4cpO4hDWR4LkGRoAFJAlfPBStfjVboAonQ_WOl-vEoPQJTex-triNIHEKUQtz7JAgDbh2kI","sizes":{"file":11425440,"external":11923785,"active":11287083},"other":{"data_size":11923785},"doc_del_count":0,"doc_count":12001,"disk_size":11425440,"disk_format_version":7,"data_size":11287083,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}

CouchDB 3:

$ curl -u "xxx:xxx" http://localhost:3003/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUy4FV3AKLuPiF1CyDq9hNS1wBRNx-3uqQEIJlUj9dtSQ4gNfH41SiA1NjjVZPIkCQPUZAFAN6AYvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUxMLDew6v0AETpfbDSg3iVLoAo3Q9W-gKv0gaI0vlgpa9wKE1KAJJJ9TB33sGlzAGkLB6mDKdpCiBl9jBll3AoS2RIkoepuZEFAAsmagc","sizes":{"file":14468808,"external":9785687,"active":11343864},"props":{},"doc_del_count":0,"doc_count":12001,"disk_format_version":8,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}

Both state:

"cluster":{"q":8,"n":1,"w":1,"r":1}

After injecting the data I waited multiple minutes before running the queries:

$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"5584fb6a7a94ff8ab44bbee93823352d","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10

real    0m8.534s
user    0m0.017s
sys 0m0.013s
Round: 2/10

real    0m5.050s
user    0m0.012s
sys 0m0.010s
Round: 3/10

real    0m4.677s
user    0m0.010s
sys 0m0.011s
Round: 4/10

real    0m4.820s
user    0m0.013s
sys 0m0.009s
Round: 5/10

real    0m4.798s
user    0m0.011s
sys 0m0.011s
Round: 6/10

real    0m5.093s
user    0m0.013s
sys 0m0.009s
Round: 7/10

real    0m5.285s
user    0m0.015s
sys 0m0.007s
Round: 8/10

real    0m5.627s
user    0m0.010s
sys 0m0.014s
Round: 9/10

real    0m5.459s
user    0m0.013s
sys 0m0.010s
Round: 10/10

real    0m5.589s
user    0m0.009s
sys 0m0.015s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"6445c39f1422b264fbf7837dbb055bb9","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10

real    0m53.993s
user    0m0.014s
sys 0m0.011s
Round: 2/10

real    0m53.707s
user    0m0.016s
sys 0m0.009s
Round: 3/10

real    0m56.130s
user    0m0.011s
sys 0m0.016s
Round: 4/10

real    0m56.537s
user    0m0.012s
sys 0m0.014s
Round: 5/10

real    0m55.933s
user    0m0.012s
sys 0m0.013s
Round: 6/10

real    0m52.887s
user    0m0.013s
sys 0m0.014s
Round: 7/10

real    1m1.758s
user    0m0.012s
sys 0m0.012s
Round: 8/10

real    0m52.182s
user    0m0.010s
sys 0m0.018s
Round: 9/10

real    0m53.072s
user    0m0.011s
sys 0m0.016s
Round: 10/10

real    0m53.720s
user    0m0.014s
sys 0m0.013s

I am not sure if the default log level does not indicate the auto-builder and auto-compactor actions, but at least the container logs are not reflecting such actions:

$ podman logs -f couchdb3
...
[notice] 2021-07-05T12:34:42.581798Z nonode@nohost <0.19894.3> 14ea633501 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 19
[notice] 2021-07-05T12:34:42.618414Z nonode@nohost <0.19902.3> 96877188fc 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 22
[notice] 2021-07-05T12:34:42.650066Z nonode@nohost <0.19910.3> c2fe735612 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 20
[info] 2021-07-05T12:35:10.329291Z nonode@nohost <0.3652.0> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.749276Z nonode@nohost <0.3652.0> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.936249Z nonode@nohost <0.3778.0> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.008010Z nonode@nohost <0.3778.0> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.291704Z nonode@nohost <0.3817.0> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:14.564826Z nonode@nohost <0.3817.0> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.314002Z nonode@nohost <0.3907.0> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.373015Z nonode@nohost <0.3975.0> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.510076Z nonode@nohost <0.3621.0> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.010943Z nonode@nohost <0.4169.0> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.654203Z nonode@nohost <0.3975.0> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.677526Z nonode@nohost <0.3907.0> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.788062Z nonode@nohost <0.3621.0> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:17.306988Z nonode@nohost <0.4169.0> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:18.494925Z nonode@nohost <0.3686.0> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:19.388880Z nonode@nohost <0.3686.0> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[notice] 2021-07-05T14:32:23.887866Z nonode@nohost <0.13346.5> 61a2177aae 127.0.0.1:3003 10.0.2.100 admin GET / 200 ok 9
[notice] 2021-07-05T14:33:17.881591Z nonode@nohost <0.13347.5> 39c531c67e 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53986
[notice] 2021-07-05T14:34:11.588538Z nonode@nohost <0.13889.5> d3cc5e0fa8 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53700
[notice] 2021-07-05T14:35:07.718592Z nonode@nohost <0.14346.5> 05b6354796 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56122
[notice] 2021-07-05T14:36:04.255503Z nonode@nohost <0.14842.5> 1535baf654 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56529
[notice] 2021-07-05T14:37:00.187963Z nonode@nohost <0.15334.5> 00a9c90ecf 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 55926
[notice] 2021-07-05T14:37:53.074959Z nonode@nohost <0.15818.5> e48cbf750a 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52880
[notice] 2021-07-05T14:38:54.833050Z nonode@nohost <0.16272.5> 07b2646c16 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 61751
[notice] 2021-07-05T14:39:47.015319Z nonode@nohost <0.16808.5> a26c18d19d 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52175
[notice] 2021-07-05T14:40:40.086929Z nonode@nohost <0.17255.5> 5267817a12 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53065
[notice] 2021-07-05T14:41:33.807566Z nonode@nohost <0.17723.5> 21e5c4941c 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53713

Compared to couchdb2:

$ podman logs couchdb2
...
[notice] 2021-07-05T12:27:09.139663Z nonode@nohost <0.23495.2> 714b3e7a37 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 30
[notice] 2021-07-05T12:27:09.165180Z nonode@nohost <0.23502.2> 9a12b27d94 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 18
[info] 2021-07-05T13:22:11.293974Z nonode@nohost <0.12126.4> -------- Starting compaction for db "shards/20000000-3fffffff/demo.1625487744" at 1541
[info] 2021-07-05T13:22:32.858211Z nonode@nohost <0.12139.4> -------- Starting compaction for db "shards/00000000-1fffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:22:54.395354Z nonode@nohost <0.12142.4> -------- Starting compaction for db "shards/40000000-5fffffff/demo.1625487744" at 1505
[info] 2021-07-05T13:23:15.835989Z nonode@nohost <0.12145.4> -------- Starting compaction for db "shards/60000000-7fffffff/demo.1625487744" at 1473
[info] 2021-07-05T13:23:37.277484Z nonode@nohost <0.12151.4> -------- Starting compaction for db "shards/a0000000-bfffffff/demo.1625487744" at 1471
[info] 2021-07-05T13:23:58.739812Z nonode@nohost <0.12157.4> -------- Starting compaction for db "shards/e0000000-ffffffff/demo.1625487744" at 1508
[info] 2021-07-05T13:24:20.179296Z nonode@nohost <0.12148.4> -------- Starting compaction for db "shards/80000000-9fffffff/demo.1625487744" at 1455
[info] 2021-07-05T13:24:41.714250Z nonode@nohost <0.12154.4> -------- Starting compaction for db "shards/c0000000-dfffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:24:54.231782Z nonode@nohost <0.12450.4> -------- Starting compaction for db "shards/c0000000-dfffffff/_global_changes.1625487744" at 554
[notice] 2021-07-05T14:31:28.936974Z nonode@nohost <0.15912.6> 312602f8c0 127.0.0.1:3002 10.0.2.100 admin GET / 200 ok 2
[info] 2021-07-05T14:31:28.989213Z nonode@nohost <0.15983.6> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.989257Z nonode@nohost <0.15984.6> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990506Z nonode@nohost <0.16010.6> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990507Z nonode@nohost <0.16011.6> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990573Z nonode@nohost <0.16012.6> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990650Z nonode@nohost <0.16016.6> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990715Z nonode@nohost <0.16020.6> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.991237Z nonode@nohost <0.16025.6> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.578442Z nonode@nohost <0.16012.6> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.592533Z nonode@nohost <0.16025.6> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.624614Z nonode@nohost <0.16011.6> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.630679Z nonode@nohost <0.16010.6> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.641088Z nonode@nohost <0.16016.6> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.682452Z nonode@nohost <0.15984.6> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.729299Z nonode@nohost <0.16020.6> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.746659Z nonode@nohost <0.15983.6> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[notice] 2021-07-05T14:31:37.472657Z nonode@nohost <0.15913.6> 3b98ddbf30 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 8516
[notice] 2021-07-05T14:31:42.522912Z nonode@nohost <0.16231.6> 139c6ac71b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5042
[notice] 2021-07-05T14:31:47.199986Z nonode@nohost <0.16315.6> 63625be9aa 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4671
[notice] 2021-07-05T14:31:52.020073Z nonode@nohost <0.16417.6> 06140e6373 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4813
[notice] 2021-07-05T14:31:56.818616Z nonode@nohost <0.16495.6> 73aef1519b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4791
[notice] 2021-07-05T14:32:01.911785Z nonode@nohost <0.16590.6> e7371ec8db 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5086
[notice] 2021-07-05T14:32:07.196507Z nonode@nohost <0.16725.6> 19201fdd74 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5278
[notice] 2021-07-05T14:32:12.823661Z nonode@nohost <0.16833.6> 1cfc7a7f8a 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5618
[notice] 2021-07-05T14:32:18.282645Z nonode@nohost <0.16924.6> a70a1462dc 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5453
[notice] 2021-07-05T14:32:23.871410Z nonode@nohost <0.17021.6> 8364bc2506 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5581

Thanks for trying with q=8, @konrad-ohms. The JS engine could be responsible as you indicated then.

Hello folks, is there an update on this issue? I'm also facing this issue when I migrate from 2.3.1 to 3.1.1. View queries with reduce are significantly slower.

I confirmed that the JS engine is implicated. I ran 2.3.1 and 3.x latest on Ubuntu 20.04. For 3.x I installed the libmozjs185 couchdb-build package for Ubuntu 18.04 on my Ubuntu 20.04 system and ran with q=8 to minimize difference and similar performance numbers as 2.3.1.

Packages I downloaded and installed manually:

couch-libmozjs185-1.0_1.8.5-1.0.0+couch-2_bionic_amd64.deb
couch-libmozjs185-dev_1.8.5-1.0.0+couch-2_bionic_amd64.deb
libffi6_3.2.1-8_amd64.deb
libffi-dev_3.2.1-8_amd64.deb

(libffi6 was needed as a dependency of couch-libmozjs185)

Configure 3.x to build with version 1.8.5

./configure --spidermonkey-version 1.8.5 --dev

Test script was modified to not set up docker containers, just run locally with 2.3.1 and 3.1.1 running on different ports.

./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"fake_uuid_for_dev","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15994
Round: 1/10

real    0m13.022s
user    0m0.023s
sys 0m0.023s
Round: 2/10

real    0m7.234s
user    0m0.020s
sys 0m0.008s
Round: 3/10

real    0m6.715s
user    0m0.026s
sys 0m0.000s
Round: 4/10

real    0m7.021s
user    0m0.018s
sys 0m0.009s
Round: 5/10

real    0m7.159s
user    0m0.005s
sys 0m0.021s
Round: 6/10

real    0m7.318s
user    0m0.014s
sys 0m0.014s
Round: 7/10

real    0m7.464s
user    0m0.007s
sys 0m0.021s
Round: 8/10

real    0m7.175s
user    0m0.016s
sys 0m0.011s
Round: 9/10

real    0m7.096s
user    0m0.004s
sys 0m0.023s
Round: 10/10

real    0m7.235s
user    0m0.006s
sys 0m0.022s

q=8
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1-ba63878","git_sha":"ba63878","uuid":"fake_uuid_for_dev","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15984
Round: 1/10

real    0m5.979s
user    0m0.017s
sys 0m0.010s
Round: 2/10

real    0m6.757s
user    0m0.013s
sys 0m0.013s
Round: 3/10

real    0m7.112s
user    0m0.012s
sys 0m0.016s
Round: 4/10

real    0m7.359s
user    0m0.017s
sys 0m0.011s
Round: 5/10

real    0m7.210s
user    0m0.017s
sys 0m0.013s
Round: 6/10

real    0m7.233s
user    0m0.008s
sys 0m0.021s
Round: 7/10

real    0m6.284s
user    0m0.028s
sys 0m0.000s
Round: 8/10

real    0m7.101s
user    0m0.020s
sys 0m0.008s
Round: 9/10

real    0m6.943s
user    0m0.022s
sys 0m0.006s
Round: 10/10

real    0m6.736s
user    0m0.016s
sys 0m0.011s

With default mozjs68 library on Ubuntu I was getting around 30 seconds

Thank you @nickva for your investigation. Good to know that it is really located in the Javascript engine and not elsewhere within the CouchDB code base. I am not too familiar with CouchDB's dependencies, are you stating, that the Problem is basically a performance regression in Mozilla’s SpiderMonkey implementation?

Discussing this on couchdb-dev channel, CouchDB with JS runtimes later than mozjs 60 need to re-write their design doc functions [1]. Also, unlike pure map views, where the JS runtime gets to process (learn) the function first, then only handle the keys and values [2], custom reduce functions have the function body sent with each query server command. Therefore, any overhead in that JS code re-writing is amplified for the custom reducers.

Looking back at the reduce function, I wonder if it would be possible to rewrite it using two _sum[3] and _stats[4] built-in reducers, which should not suffer from the same rewriting overhead.

For example, the _sum reducer can be applied to objects, such as:

"views":{"v1":{ "map": "function(doc){ emit(null, {x:doc.x, y:doc.y}) };", "reduce":"_sum"}}
...
{"x": 1, "y": 3}
{"x": 2, "y": 4}

And would emit:

{"key":null,"value":{"x":3,"y":7}}

That could be used for "unsuccessful" results. For the "successful" case, execution stats could be accumulated using _stats [3]

[1] https://github.com/apache/couchdb/blob/0059b8f90e58e10b199a4b768a06a762d12a30d3/share/server/60/rewrite_fun.js#L29-L38 [2] https://docs.couchdb.org/en/stable/query-server/protocol.html?highlight=add_fun#add-fun [3] https://docs.couchdb.org/en/stable/ddocs/ddocs.html#_sum [4] https://docs.couchdb.org/en/stable/ddocs/ddocs.html#_stats

Thanks @nickva and sorry for my delayed response, it took me a while to gather the right data to check if that change would be feasible for our application code.

TLDR: The proposed solution works and is about 130 times faster on the given dataset, that are great results, thanks so much for your input :-)

View adjustments

I tried to use the build-in reducers for a small set of documents on CouchDB 3.

For the testing out your proposal, I created 7 documents, all shared the same aDefId to return one single reduced output object. I have one document per execution result and an additional document which executed successful to be able to check the computation.

The original view was using this map function:

function (doc) {
    if(doc.object === "ainst" && doc.result) {
        var subscription = doc.subscription || doc.tenant;
        var stats = {
            successfulCount : 0,
            failedCount : 0,
            canceledCount : 0,
            executingCount : 0,
            unsuccessfulCount : 0,
            unknownCount : 0,
            totalCount : 1,
            execTimeSum : 0,
            execTimeCount : 0,
            execTimeMin : Infinity,
            execTimeMax : 0
        }

        if (doc.result.status === "successful") {
            stats.successfulCount = 1;

            if (doc.result.executionTime) {
                stats.execTimeSum = doc.result.executionTime;
                stats.execTimeCount = 1;
                stats.execTimeMin = doc.result.executionTime;
                stats.execTimeMax = doc.result.executionTime;
            }
        } else if (doc.result.status === "failed") {
            stats.failedCount = 1;
        } else if (doc.result.status === "canceled") {
            stats.canceledCount = 1;
        } else if (doc.result.status === "executing") {
            stats.executingCount = 1;
        } else if (doc.result.status === "unsuccessful") {
            stats.unsuccessfulCount = 1;
        } else if (doc.result.status === "unknown") {
            stats.unknownCount = 1;
        }

        emit([subscription, doc.aDefId], stats);
    }
}

and the following custom reduce function:

function (keys, values, rereduce) {
    var stats = {
        successfulCount : 0,
        failedCount : 0,
        canceledCount : 0,
        executingCount : 0,
        unsuccessfulCount : 0,
        unknownCount : 0,
        totalCount : 0,
        execTimeSum : 0,
        execTimeCount : 0,
        execTimeMin : Infinity,
        execTimeMax : 0
    }

    for (var index in values) {
        stats.successfulCount += values[index].successfulCount;
        stats.failedCount += values[index].failedCount;
        stats.canceledCount += values[index].canceledCount;
        stats.executingCount += values[index].executingCount;
        stats.unsuccessfulCount += values[index].unsuccessfulCount;
        stats.unknownCount += values[index].unknownCount;
        stats.totalCount += values[index].totalCount;
        if (values[index].execTimeCount) {
            stats.execTimeSum += values[index].execTimeSum;
            stats.execTimeCount += values[index].execTimeCount;
            stats.execTimeMin = Math.min(stats.execTimeMin, values[index].execTimeMin);
            stats.execTimeMax = Math.max(stats.execTimeMax, values[index].execTimeMax);
        }
    }

    return stats;
}

The original view returned the following as response which I will use to validate the result of the following adjustments:

$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo/_view/slow?reduce=true&group=true" | jq "."
{
  "rows": [
    {
      "key": [
        "xxx",
        "135600b7487db5804c1961408a115e0270e5"
      ],
      "value": {
        "successfulCount": 2,
        "failedCount": 1,
        "canceledCount": 1,
        "executingCount": 1,
        "unsuccessfulCount": 1,
        "unknownCount": 1,
        "totalCount": 7,
        "execTimeSum": 1557,
        "execTimeCount": 2,
        "execTimeMin": 200,
        "execTimeMax": 1357
      }
    }
  ]
}

As you mentioned, the build-in _sum reducer can be used to count different executions in a single view.

This is my updated demo-build-in-reduce/count view:

function (doc) {
    if(doc.object === "ainst" && doc.result) {
        var subscription = doc.subscription || doc.tenant;
        var stats = {
            successfulCount : 0,
            failedCount : 0,
            canceledCount : 0,
            executingCount : 0,
            unsuccessfulCount : 0,
            unknownCount : 0,
            totalCount : 1
        }

        if (doc.result.status === "successful") {
            stats.successfulCount = 1;
        } else if (doc.result.status === "failed") {
            stats.failedCount = 1;
        } else if (doc.result.status === "canceled") {
            stats.canceledCount = 1;
        } else if (doc.result.status === "executing") {
            stats.executingCount = 1;
        } else if (doc.result.status === "unsuccessful") {
            stats.unsuccessfulCount = 1;
        } else if (doc.result.status === "unknown") {
            stats.unknownCount = 1;
        }

        emit([subscription, doc.aDefId], stats);
    }
}

The reduce function is set to _sum on that view.

When I query that view, I get results as expected for the counts:

$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/count?reduce=true&group=true" | jq "."
{
  "rows": [
    {
      "key": [
        "xxx",
        "135600b7487db5804c1961408a115e0270e5"
      ],
      "value": {
        "canceledCount": 1,
        "executingCount": 1,
        "failedCount": 1,
        "successfulCount": 2,
        "totalCount": 7,
        "unknownCount": 1,
        "unsuccessfulCount": 1
      }
    }
  ]
}

The response looks good, but as expected is lacking the execution time stats. This can be done with a second view which I named demo-build-in-reduce/successfulExecutionTime.

The second view returns the following:

$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true" | jq "."
{
  "rows": [
    {
      "key": [
        "xxx",
        "135600b7487db5804c1961408a115e0270e5"
      ],
      "value": {
        "sum": 1557,
        "count": 2,
        "min": 200,
        "max": 1357,
        "sumsqr": 1881449
      }
    }
  ]
}

This looks good either, I do not care for the sumsqr, but for sum, count, min and max. The count is also included in the demo-build-in-reduce/count view, which can be used in the application theoretically to render a more high level view of data. The successfulExecutionTime view is providing the other data.

So, your proposal works for the given data model, thank you very much. The downside is that to keep our API compatible, we need to submit two requests concurrently instead of using the single original view. Nevertheless, the performance is way more consistent, let me share a few measurements based on the new views.

Test setup to measure performance improvements of the new implementation

I used the initial setup script to deploy both CouchDBs and to load data. Afterwards I injected the design doc into the demo database:

$ curl -X POST \
    "http://admin:password@localhost:3003/demo" \
    -H "Content-Type: application/json" \
    -d '{
        "_id": "_design/demo-build-in-reduce",
        "views": {
            "count": {
            "reduce": "_sum",
            "map": "function (doc) {\n                if(doc.object === \"ainst\" && doc.result) {\n                    var subscription = doc.subscription || doc.tenant;\n                    var stats = {\n                        successfulCount : 0,\n                        failedCount : 0,\n                        canceledCount : 0,\n                        executingCount : 0,\n                        unsuccessfulCount : 0,\n                        unknownCount : 0,\n                        totalCount : 1\n                    }\n\n                    if (doc.result.status === \"successful\") {\n                        stats.successfulCount = 1;\n                    } else if (doc.result.status === \"failed\") {\n                        stats.failedCount = 1;\n                    } else if (doc.result.status === \"canceled\") {\n                        stats.canceledCount = 1;\n                    } else if (doc.result.status === \"executing\") {\n                        stats.executingCount = 1;\n                    } else if (doc.result.status === \"unsuccessful\") {\n                        stats.unsuccessfulCount = 1;\n                    } else if (doc.result.status === \"unknown\") {\n                        stats.unknownCount = 1;\n                    }\n\n                    emit([subscription, doc.aDefId], stats);\n                }\n            }"
            },
            "successfulExecutionTime": {
            "reduce": "_stats",
            "map": "function (doc) {\n    if(doc.object === \"ainst\" && doc.result && doc.result.status === \"successful\") {\n        var subscription = doc.subscription || doc.tenant;\n        emit([subscription, doc.aDefId], doc.result.executionTime);\n    }\n}"
            }
        },
        "language": "javascript"
    }'
{"ok":true,"id":"_design/demo-build-in-reduce","rev":"1-e769ee9d7efdeb5f80075e16a1113968"}

I used the following Node.js script to query data and combine them within the application. reducers-demo.zip

'use strict';
const parallel = require('async').parallel;
const http = require('http');
const user = 'admin';
const password = 'password';
const port = 3003;
const dbname = 'demo';

let newRequest = (async () => {
    console.log("==================== New approach ====================");
    for( let i=1; i<=10; i++) {
        console.log(`===== Iteration ${i} ======`);
        console.time(`Fast iteration ${i} took`);
        let responseMapCount = {};
        let responseMapSuccessfulExecutionTime = {};
        await parallel([
            (cb) => {
                let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/count?reduce=true&group=true`;
                console.log(`Requesting: ${currentUrl}`)
                let req = http.get(currentUrl, (res) => {
                    console.log('statusCode:', res.statusCode);

                    let body = ""
                    res.on('data', (d) => {
                        body += d;
                    });
                    res.on('end', function () {
                        // store rows in map to combine them after http requests are completed
                        body = JSON.parse(body);
                        body.rows.forEach(currentRow => {
                            responseMapCount[currentRow.key.join('-')] = currentRow.value;
                        });
                        cb()
                    });
                }).on('error', (e) => {
                    console.error(e);
                    cb(e);
                });
                req.end();
            },
            (cb) => {
                let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true`;
                console.log(`Requesting: ${currentUrl}`)
                let req = http.get(currentUrl, (res) => {
                    console.log('statusCode:', res.statusCode);

                    let body = '';
                    res.on('data', (d) => {
                        body += d;
                    });
                    res.on('end', function () {
                        // store rows in map to combine them after http requests are completed
                        body = JSON.parse(body);
                        body.rows.forEach(currentRow => {
                            responseMapSuccessfulExecutionTime[currentRow.key.join('-')] = currentRow.value;
                        });
                        cb()
                    });
                }).on('error', (e) => {
                    console.error(e);
                    cb(e);
                });
                req.end();
            }
        ]);
        console.log(`Combining results`);
        // iterate over responseMapCount, as there is always an entry for every ainst result
        let lastMergedKey;
        Object.keys(responseMapCount).forEach((currentKey) => {
            // check if the aInst result had a successful invocation, merge it back in that case
            if(responseMapSuccessfulExecutionTime[currentKey]) {
                // merge second result property back into responseMapCount
                responseMapCount[currentKey].execTimeSum = responseMapSuccessfulExecutionTime[currentKey].sum;
                responseMapCount[currentKey].execTimeMin = responseMapSuccessfulExecutionTime[currentKey].min;
                responseMapCount[currentKey].execTimeMax = responseMapSuccessfulExecutionTime[currentKey].max;
            }
        });
        console.log(`Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: ${JSON.stringify(responseMapCount['xxx-100200b7487db5804c1961408a115e0270e5'], null, 4)}`)
        console.timeEnd(`Fast iteration ${i} took`);
    }
});

let oldRequest = (async () => {
    console.log("==================== Old approach ====================");
    for( let i=1; i<=10; i++) {
        console.log(`===== Iteration ${i} ======`);
        console.time(`Slow iteration ${i} took`);
        let results = await parallel([
            (cb) => {
                let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo/_view/slow?reduce=true&group=true`;
                console.log(`Requesting: ${currentUrl}`)
                let req = http.get(currentUrl, (res) => {
                    console.log('statusCode:', res.statusCode);

                    let body = ''
                    res.on('data', (d) => {
                        body += d;
                    });
                    res.on('end', function () {
                        body = JSON.parse(body);
                        cb(null, body);
                    });
                }).on('error', (e) => {
                    console.error(e);
                    cb(e);
                });
                req.end();
            }
        ]);
        console.log(`Result for aDefId=100200b7487db5804c1961408a115e0270e5:`);
        // the view could have been invoked directly with the key, but all results should be retrieved to measure the performance
        // only a single result is picked to compare those are equal in both query approaches. 
        results[0].rows.forEach((currentRow) => {
            if(currentRow.key[1] === '100200b7487db5804c1961408a115e0270e5') {
                console.log(JSON.stringify(currentRow.value, null, 4));
            }
        })
        console.timeEnd(`Slow iteration ${i} took`);
    }
});

(async () => {
    await newRequest();
    await oldRequest();
})()

The output shows an impressive difference, even if the application logic gets more complicated (full output: couchdb3-test-output.txt):

==================== New approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
    "canceledCount": 1,
    "executingCount": 1,
    "failedCount": 1,
    "successfulCount": 1,
    "totalCount": 6,
    "unknownCount": 1,
    "unsuccessfulCount": 1,
    "execTimeSum": 1002,
    "execTimeMin": 1002,
    "execTimeMax": 1002
}
Fast iteration 1 took: 270.495ms
...
==================== Old approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
    "successfulCount": 1,
    "failedCount": 1,
    "canceledCount": 1,
    "executingCount": 1,
    "unsuccessfulCount": 1,
    "unknownCount": 1,
    "totalCount": 6,
    "execTimeSum": 1002,
    "execTimeCount": 1,
    "execTimeMin": 1002,
    "execTimeMax": 1002
}
Slow iteration 1 took: 34.973s
...

Conclusion

On average (10 iterations), I could get combined results of all records within 274 ms compared to 35714 ms, so the workaround on my given dataset is actually ~ 130 times faster.

I repeated the test for CouchDB 2 as well (couchdb2-test-output) and got an average of 447 ms with the new approach compared to 6917 ms which is still more than 15 times faster. So independent on the question if we should migrate to CouchDB 3 or not, it might still be a useful optimization also for CouchDB 2 setups.

Excellent write-up, @konrad-ohms. Thank you for getting back to us and sharing the results. 130x speedup is not something we see every day!

I’ve attempted a quick demo that caches the transpilation inside the JS process, this would still do one transpile per couchjs process, but should still be a lot better.

@konrad-ohms or @nickva would you be able to retry this with the original setup?

diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
index 1b27a9d14..9b85f953f 100644
--- a/share/server/60/rewrite_fun.js
+++ b/share/server/60/rewrite_fun.js
@@ -15,30 +15,39 @@
 //
 //  https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js

+
+const cache = {}
 function rewriteFunInt(fun) {
-    const ast = esprima.parse(fun);
-    let idx = ast.body.length - 1;
-    let decl = {};
-
-    // Search for the first FunctionDeclaration beginning from the end
-    do {
-        decl = ast.body[idx--];
-    } while (idx >= 0 && decl.type !== "FunctionDeclaration");
-    idx++;
-
-    // If we have a function declaration without an Id, wrap it
-    // in an ExpressionStatement and change it into
-    // a FuntionExpression
-    if (decl.type == "FunctionDeclaration" && decl.id == null) {
-        decl.type = "FunctionExpression";
-        ast.body[idx] = {
-            type: "ExpressionStatement",
-            expression: decl
-        };
-    }
+    const crc = crc32_str(fun)
+    if (cache[crc]) {
+        return cache[crc];
+    } else {
+        const ast = esprima.parse(fun);
+        let idx = ast.body.length - 1;
+        let decl = {};
+
+        // Search for the first FunctionDeclaration beginning from the end
+        do {
+            decl = ast.body[idx--];
+        } while (idx >= 0 && decl.type !== "FunctionDeclaration");
+        idx++;
+
+        // If we have a function declaration without an Id, wrap it
+        // in an ExpressionStatement and change it into
+        // a FuntionExpression
+        if (decl.type == "FunctionDeclaration" && decl.id == null) {
+            decl.type = "FunctionExpression";
+            ast.body[idx] = {
+                type: "ExpressionStatement",
+                expression: decl
+            };
+        }

-    // Generate source from the rewritten AST
-    return escodegen.generate(ast);
+        // Generate source from the rewritten AST
+        const gen = escodegen.generate(ast);
+        cache[crc] = gen;
+        return gen;
+    }
 }

@@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
         return rewriteFunInt(fun);
     });
     return JSON.stringify(results);
-}
\ No newline at end of file
+}
+
+// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
+
+function signed_crc_table() {
+   var c = 0, table = new Array(256);
+
+   for(var n =0; n != 256; ++n){
+       c = n;
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+       table[n] = c;
+   }
+
+   return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
+}
+
+var T = signed_crc_table();
+
+function crc32_str(str, seed) {
+   var C = seed ^ -1;
+   for(var i = 0, L=str.length, c, d; i < L;) {
+       c = str.charCodeAt(i++);
+       if(c < 0x80) {
+           C = (C>>>8) ^ T[(C ^ c)&0xFF];
+       } else if(c < 0x800) {
+           C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
+           C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
+       } else if(c >= 0xD800 && c < 0xE000) {
+           c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
+           C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
+           C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
+           C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
+           C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
+       } else {
+           C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
+           C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
+           C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
+       }
+   }
+   return C ^ -1;
+}

Thank you @janl for your code change. I will try to compare the performance of your change within the test suite above, I try to fit that in next week.

Okay, I validated your fix, please find the outcome below:

Creating a custom build

First, I wanted to create a patch build which is as close as possible to what I have tested before, so I used the same OS as in the container image I used initially:

$ podman run --rm -it docker.io/library/couchdb:3.1.1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

To get a build env matching the OS and its required libraries, I had a look into https://github.com/apache/couchdb-ci and built the following image on an 18.04.5 LTS amd64 VM with Docker version 20.10.8, build 3967b7d:

$ git clone https://github.com/apache/couchdb-ci.git
Cloning into 'couchdb-ci'...
remote: Enumerating objects: 1742, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1742 (delta 55), reused 56 (delta 29), pack-reused 1655
Receiving objects: 100% (1742/1742), 10.24 MiB | 16.11 MiB/s, done.
Resolving deltas: 100% (998/998), done.
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
Successfully built d1c29b763a2c
Successfully tagged apache/couchdbci-debian:buster-erlang-20.3.8.26-1

From there I used CouchDB v3.1.1 and applied your patch within the container:

$ docker run --name couchdb-3.1.1-fix-validation -it apache/couchdbci-debian:buster-erlang-20.3.8.26-1
jenkins@2665e52069d0:/$ cd ~
jenkins@2665e52069d0:~$ whoami
jenkins
jenkins@2665e52069d0:~$ pwd
/home/jenkins
jenkins@2665e52069d0:~$ git clone --depth=1 -b 3.1.1 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1207, done.
remote: Counting objects: 100% (1207/1207), done.
remote: Compressing objects: 100% (1099/1099), done.
remote: Total 1207 (delta 245), reused 380 (delta 55), pack-reused 0
Receiving objects: 100% (1207/1207), 1.80 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Note: checking out 'ce596c65d9d7f0bc5d9937bcaf6253b343015690'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

jenkins@2665e52069d0:~$ # Apply Jan's patch
jenkins@2665e52069d0:~$ cat <<'EOF' > jan.patch
> diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
> index 1b27a9d14..9b85f953f 100644
> --- a/share/server/60/rewrite_fun.js
> +++ b/share/server/60/rewrite_fun.js
> @@ -15,30 +15,39 @@
>  //
>  //  https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
>  
> +
> +const cache = {}
>  function rewriteFunInt(fun) {
> -    const ast = esprima.parse(fun);
> -    let idx = ast.body.length - 1;
> -    let decl = {};
> -
> -    // Search for the first FunctionDeclaration beginning from the end
> -    do {
> -        decl = ast.body[idx--];
> -    } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> -    idx++;
> -
> -    // If we have a function declaration without an Id, wrap it
> -    // in an ExpressionStatement and change it into
> -    // a FuntionExpression
> -    if (decl.type == "FunctionDeclaration" && decl.id == null) {
> -        decl.type = "FunctionExpression";
> -        ast.body[idx] = {
> -            type: "ExpressionStatement",
> -            expression: decl
> -        };
> -    }
> +    const crc = crc32_str(fun)
> +    if (cache[crc]) {
> +        return cache[crc];
> +    } else {
> +        const ast = esprima.parse(fun);
> +        let idx = ast.body.length - 1;
> +        let decl = {};
> +
> +        // Search for the first FunctionDeclaration beginning from the end
> +        do {
> +            decl = ast.body[idx--];
> +        } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> +        idx++;
> +
> +        // If we have a function declaration without an Id, wrap it
> +        // in an ExpressionStatement and change it into
> +        // a FuntionExpression
> +        if (decl.type == "FunctionDeclaration" && decl.id == null) {
> +            decl.type = "FunctionExpression";
> +            ast.body[idx] = {
> +                type: "ExpressionStatement",
> +                expression: decl
> +            };
> +        }
>  
> -    // Generate source from the rewritten AST
> -    return escodegen.generate(ast);
> +        // Generate source from the rewritten AST
> +        const gen = escodegen.generate(ast);
> +        cache[crc] = gen;
> +        return gen;
> +    }
>  }
>  
>  
> @@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
>          return rewriteFunInt(fun);
>      });
>      return JSON.stringify(results);
> -}
> \ No newline at end of file
> +}
> +
> +// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
> +
> +function signed_crc_table() {
> +var c = 0, table = new Array(256);
> +
> +for(var n =0; n != 256; ++n){
> +c = n;
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +table[n] = c;
> +}
> +
> +return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
> +}
> +
> +var T = signed_crc_table();
> +
> +function crc32_str(str, seed) {
> +var C = seed ^ -1;
> +for(var i = 0, L=str.length, c, d; i < L;) {
> +c = str.charCodeAt(i++);
> +if(c < 0x80) {
> +C = (C>>>8) ^ T[(C ^ c)&0xFF];
> +} else if(c < 0x800) {
> +C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +} else if(c >= 0xD800 && c < 0xE000) {
> +c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
> +C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
> +} else {
> +C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +}
> +}
> +return C ^ -1;
> +}
> EOF

jenkins@2665e52069d0:~$ cd couchdb/
jenkins@2665e52069d0:~/couchdb$ git status
Not currently on any branch.
nothing to commit, working tree clean
jenkins@2665e52069d0:~/couchdb$ git checkout -b 3.1.1-patch-jan
Switched to a new branch '3.1.1-patch-jan'
jenkins@2665e52069d0:~/couchdb$ git apply ../jan.patch 
jenkins@2665e52069d0:~/couchdb$ git status
On branch 3.1.1-patch-jan
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   share/server/60/rewrite_fun.js

no changes added to commit (use "git add" and/or "git commit -a")

Afterwards I compiled the source:

jenkins@2665e52069d0:~/couchdb$ ./configure
...
==> couchdb (update-deps)
Updating config from {git,"https://github.com/apache/couchdb-config.git",
                          {tag,"2.1.7"}}
Updating b64url from {git,"https://github.com/apache/couchdb-b64url.git",
                          {tag,"1.0.2"}}
Updating ets_lru from {git,"https://github.com/apache/couchdb-ets-lru.git",
                           {tag,"1.1.0"}}
Updating khash from {git,"https://github.com/apache/couchdb-khash.git",
                         {tag,"1.1.0"}}
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
                          {tag,"CouchDB-1.0.4"}}
Updating docs from {git,"https://github.com/apache/couchdb-documentation",
                        {tag,"3.1.1-RC2"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
                           {tag,"v1.2.6"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
                          {tag,"CouchDB-0.8.3"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
                         {tag,"CouchDB-2.2.0-6"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
                           {tag,"CouchDB-4.0.1-1"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
                         {tag,"CouchDB-1.0.4-1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
                            {tag,"v2.20.0"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
                        {tag,"0.8.8"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
                         {tag,"2.5.0"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
                          {tag,"v1.3"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
                        "008f48aff819126e281d5ccae80a258bf9bf9c30"}
You have configured Apache CouchDB, time to relax. Relax.

# Compile
jenkins@2665e52069d0:~/couchdb$ make release
...
The HTML pages are in build/html.
sphinx-build -b man -a -W -n -A local=1 -D latex_elements.papersize=a4 -d build/doctree src/ build/man
Running Sphinx v1.8.4
making output directory...
loading pickled environment... done
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... apachecouchdb.1 { intro/index intro/overview intro/why intro/consistency intro/curl intro/security intro/tour intro/api replication/index replication/intro replication/replicator replication/conflicts replication/protocol ddocs/index ddocs/ddocs ddocs/views/index ddocs/views/intro ddocs/views/collation ddocs/views/joins ddocs/views/nosql ddocs/views/pagination ddocs/search best-practices/index best-practices/documents best-practices/forms best-practices/iso-date best-practices/jsdevel best-practices/views best-practices/reverse-proxies install/index install/unix install/windows install/mac install/freebsd install/docker install/snap install/kubernetes install/search install/upgrading install/troubleshooting setup/index setup/single-node setup/cluster config/index config/intro config/couchdb config/cluster config/couch-peruser config/http config/auth config/compaction config/indexbuilds config/ioq config/logging config/replicator config/query-servers config/misc config/resharding cluster/index cluster/theory cluster/nodes cluster/databases cluster/sharding cluster/purging maintenance/index maintenance/compaction maintenance/performance maintenance/backups fauxton/index fauxton/install experimental api/index api/basics api/server/index api/server/common api/server/authn api/server/configuration api/database/index api/database/common api/database/bulk-api api/database/find api/database/shard api/database/changes api/database/compact api/database/security api/database/misc api/document/index api/document/common api/document/attachments api/ddoc/index api/ddoc/common api/ddoc/views api/ddoc/search api/ddoc/render api/ddoc/rewrites api/partitioned-dbs api/local json-structure query-server/index query-server/protocol query-server/javascript query-server/erlang partitioned-dbs/index whatsnew/index whatsnew/3.1 whatsnew/3.0 whatsnew/2.3 whatsnew/2.2 whatsnew/2.1 whatsnew/2.0 whatsnew/1.7 whatsnew/1.6 whatsnew/1.5 whatsnew/1.4 whatsnew/1.3 whatsnew/1.2 whatsnew/1.1 whatsnew/1.0 whatsnew/0.11 whatsnew/0.10 whatsnew/0.9 whatsnew/0.8 cve/index cve/2010-0009 cve/2010-2234 cve/2010-3854 cve/2012-5641 cve/2012-5649 cve/2012-5650 cve/2014-2668 cve/2017-12635 cve/2017-12636 cve/2018-11769 cve/2018-17188 cve/2018-8007 cve/2020-1955 about contributing } 
build succeeded.

The manual pages are in build/man.
make[1]: Leaving directory '/home/jenkins/couchdb/src/docs'
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN:  'generate' command does not apply to directory /home/jenkins/couchdb
... done

    You can now copy the rel/couchdb directory anywhere on your system.
    Start CouchDB with ./bin/couchdb from within that directory.

jenkins@2665e52069d0:~/couchdb$ echo $?
0

The compilation seem to have worked, so I created deb packages based on that. The make dist seemed not to like uncommited changes, so I added your patch as a commit:

jenkins@2665e52069d0:~/couchdb$ git config --global user.email "couchdb-validation@no-reply.com"
jenkins@2665e52069d0:~/couchdb$ git config --global user.name "CouchDB validation"
jenkins@2665e52069d0:~/couchdb$ git commit -am "fix: Apply caching"
[3.1.1-patch-jan 6c2d01a] fix: Apply caching
 1 file changed, 79 insertions(+), 23 deletions(-)
jenkins@2665e52069d0:~/couchdb$ make dist
==> config (compile)
==> b64url (compile)
==> ets_lru (compile)
==> khash (compile)
==> snappy (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> hyper (compile)
==> ibrowse (compile)
==> jiffy (compile)
==> mochiweb (compile)
==> recon (compile)
==> proper (compile)
==> couch_epi (compile)
==> couch_log (compile)
==> chttpd (compile)
==> couch (compile)
Compiling priv/couch_js/1.8.5/http.c
Compiling priv/couch_js/1.8.5/main.c
Compiling priv/couch_js/1.8.5/utf8.c
Compiling priv/couch_js/1.8.5/util.c
==> couch_event (compile)
==> mem3 (compile)
==> couch_index (compile)
==> couch_mrview (compile)
==> couch_replicator (compile)
==> couch_plugins (compile)
==> couch_pse_tests (compile)
==> couch_stats (compile)
==> couch_peruser (compile)
==> couch_tests (compile)
==> ddoc_cache (compile)
==> dreyfus (compile)
==> fabric (compile)
==> global_changes (compile)
==> ioq (compile)
==> jwtf (compile)
==> ken (compile)
==> mango (compile)
==> rexi (compile)
==> setup (compile)
==> smoosh (compile)
==> rel (compile)
==> couchdb (compile)
COUCHDB_GIT_SHA:        6c2d01a
COUCHDB_VERSION:        3.1.1-6c2d01a
COUCHDB_VERSION_SUFFIX: 6c2d01a
DIRTY:                  
IN_RC:                  
IN_RELEASE:             
ON_TAG:                 
REL_TAG:                3.1.1
SUB_VSN:                
Building Apache CouchDB 3.1.1-6c2d01a
fatal: ambiguous argument '6c976bd..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Done: apache-couchdb-3.1.1-6c2d01a.tar.gz

I copied that tar back to the host OS:

$ docker cp couchdb-3.1.1-fix-validation:/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 723, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 723 (delta 40), reused 57 (delta 20), pack-reused 629
Receiving objects: 100% (723/723), 25.31 MiB | 23.49 MiB/s, done.
Resolving deltas: 100% (335/335), done.
$ cd couchdb-pkg/

The assumed image name was couchdbdev/<osname>-<codename>-erlang-<erlang-version>, so I retagged my created one before running the packaging job:

$ docker tag apache/couchdbci-debian:buster-erlang-20.3.8.26-1 couchdbdev/debian-buster-erlang-20.3.8.26-1
$ ERLANGVERSION=20.3.8.26-1 ./build.sh couch debian-buster /root/apache-couchdb-3.1.1-6c2d01a.tar.gz
Using apache-couchdb-3.1.1-6c2d01a.tar.gz to build packages...
mkdir -p ../couchdb
cp apache-couchdb-3.1.1-6c2d01a.tar.gz ../couchdb
cd ../couchdb && tar xfz *.tar.gz
cp debian/control.in debian/control
sed -i 's/%SPIDERMONKEY%/libmozjs-60-0/g' debian/control
sed -i 's/%SPIDERMONKEY_DEV%/libmozjs-60-dev/g' debian/control
echo 'SM_VER = 60' > debian/sm_ver.mk
rm -rf /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/debian
cp -R debian /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dch -v 3.1.1-6c2d01a~buster "Automatically generated package from upstream."
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dpkg-buildpackage -b -us -uc
dpkg-buildpackage: info: source package couchdb
dpkg-buildpackage: info: source version 3.1.1-6c2d01a~buster
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by "CouchDB Developers" <"dev@couchdb.apache.org">
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
 debian/rules clean
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh clean --with=systemd
   debian/rules override_dh_auto_clean
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_clean
    make -j1 distclean
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (clean)
==> b64url (clean)
==> ets_lru (clean)
==> khash (clean)
==> snappy (clean)
==> bear (clean)
==> meck (clean)
==> folsom (clean)
==> hyper (clean)
==> ibrowse (clean)
==> jiffy (clean)
==> mochiweb (clean)
==> recon (clean)
==> couch_epi (clean)
==> couch_log (clean)
==> chttpd (clean)
==> couch (clean)
==> couch_event (clean)
==> mem3 (clean)
==> couch_index (clean)
==> couch_mrview (clean)
==> couch_replicator (clean)
==> couch_plugins (clean)
==> couch_pse_tests (clean)
==> couch_stats (clean)
==> couch_peruser (clean)
==> couch_tests (clean)
==> ddoc_cache (clean)
==> dreyfus (clean)
==> fabric (clean)
==> global_changes (clean)
==> ioq (clean)
==> jwtf (clean)
==> ken (clean)
==> mango (clean)
==> rexi (clean)
==> setup (clean)
==> smoosh (clean)
==> rel (clean)
==> apache-couchdb-3.1.1-6c2d01a (clean)
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
mv src/mango/src/mango_cursor_text.erl.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.erl.nocompile': No such file or directory
make[2]: [debian/rules:21: override_dh_auto_clean] Error 1 (ignored)
mv src/mango/src/mango_cursor_text.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.nocompile': No such file or directory
make[2]: [debian/rules:22: override_dh_auto_clean] Error 1 (ignored)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
   dh_clean
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
 debian/rules build
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh build --with=systemd
   dh_update_autotools_config
   debian/rules override_dh_auto_configure
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
./configure --spidermonkey-version 60
==> configuring couchdb in rel/couchdb.config
You have configured Apache CouchDB, time to relax. Relax.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
   debian/rules override_dh_auto_build
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_build -- release
    make -j1 release
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (compile)
Compiled src/config_util.erl
Compiled src/config_writer.erl
Compiled src/config_sup.erl
Compiled src/config_notifier.erl
Compiled src/config_listener_mon.erl
Compiled src/config_listener.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
     } catch(std::bad_alloc e) {
                            ^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13:   required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3:   required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78:   required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
     if (offset - 1u < op_ptr_ - op_base_) {
         ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_proc.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
==> folsom (compile)
Compiled src/folsom_utils.erl
Compiled src/folsom_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_const.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary_rle.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:245: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:288: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:333: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
Compiled src/hyper.erl
Compiled src/hyper_binary.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_lib.erl:372: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse_app.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:83: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:93: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:171: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:182: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_test.erl
Compiled src/ibrowse.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:428: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:1831: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiweb_html.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochijson2.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_alloc.erl
Compiled src/recon_lib.erl
Compiled src/recon.erl
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_types.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_symb.erl
Compiled src/proper_transformer.erl
Compiled src/proper_shrink.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_queue.erl
Compiled src/proper_sa.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_gen.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_fsm.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_trunc_io.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_uuids.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_stream.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_external.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_flags.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_drv.erl
Compiled src/couch_debug.erl
Compiled src/couch_doc.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_db.erl
Compiled src/couch_btree.erl
Compiled src/couch_base32.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_att.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch.erl
Compiled src/couch_app.erl
Compiling priv/couch_js/60/http.cpp
Compiling priv/couch_js/60/main.cpp
Compiling priv/couch_js/60/util.cpp
Compiling priv/icu_driver/couch_icu_driver.c
Compiling priv/couch_ejson_compare/couch_ejson_compare.c
==> couch_event (compile)
Compiled src/couch_event_listener.erl
Compiled src/couch_event_sup2.erl
Compiled src/couch_event_os_listener.erl
Compiled src/couch_event_app.erl
Compiled src/couch_event.erl
Compiled src/couch_event_server.erl
Compiled src/couch_event_listener_mfa.erl
==> mem3 (compile)
Compiled src/mem3_sync_nodes.erl
Compiled src/mem3_sync_security.erl
Compiled src/mem3_sync_event.erl
Compiled src/mem3_sync_event_listener.erl
Compiled src/mem3_sup.erl
Compiled src/mem3_sync.erl
Compiled src/mem3_util.erl
Compiled src/mem3_seeds.erl
Compiled src/mem3_reshard_validate.erl
Compiled src/mem3_reshard_sup.erl
Compiled src/mem3_shards.erl
Compiled src/mem3_rpc.erl
Compiled src/mem3_reshard_job_sup.erl
Compiled src/mem3_reshard_store.erl
Compiled src/mem3_reshard_index.erl
Compiled src/mem3_reshard_httpd.erl
Compiled src/mem3_reshard_job.erl
Compiled src/mem3_reshard_api.erl
Compiled src/mem3_reshard_dbdoc.erl
Compiled src/mem3_plugin_couch_db.erl
Compiled src/mem3_nodes.erl
Compiled src/mem3_httpd_handlers.erl
Compiled src/mem3_reshard.erl
Compiled src/mem3_hash.erl
Compiled src/mem3_epi.erl
Compiled src/mem3_httpd.erl
Compiled src/mem3_rep.erl
Compiled src/mem3_app.erl
Compiled src/mem3_cluster.erl
Compiled src/mem3.erl
==> couch_index (compile)
Compiled src/couch_index_sup.erl
Compiled src/couch_index_util.erl
Compiled src/couch_index_plugin_couch_db.erl
Compiled src/couch_index_updater.erl
Compiled src/couch_index_epi.erl
Compiled src/couch_index_plugin.erl
Compiled src/couch_index_app.erl
Compiled src/couch_index_server.erl
Compiled src/couch_index_compactor.erl
Compiled src/couch_index.erl
==> couch_mrview (compile)
Compiled src/couch_mrview_update_notifier.erl
Compiled src/couch_mrview_updater.erl
Compiled src/couch_mrview_test_util.erl
Compiled src/couch_mrview_show.erl
Compiled src/couch_mrview_util.erl
Compiled src/couch_mrview_index.erl
Compiled src/couch_mrview_compactor.erl
Compiled src/couch_mrview_cleanup.erl
Compiled src/couch_mrview_http.erl
Compiled src/couch_mrview.erl
==> couch_replicator (compile)
Compiled src/couch_replicator_httpc.erl
Compiled src/couch_replicator_auth.erl
Compiled src/couch_replicator_utils.erl
Compiled src/couch_replicator_sup.erl
Compiled src/couch_replicator_stats.erl
Compiled src/couch_replicator_scheduler_sup.erl
Compiled src/json_stream_parse.erl
Compiled src/couch_replicator_worker.erl
Compiled src/couch_replicator_rate_limiter_tables.erl
Compiled src/couch_replicator_rate_limiter.erl
Compiled src/couch_replicator_notifier.erl
Compiled src/couch_replicator_job_sup.erl
Compiled src/couch_replicator_scheduler_job.erl
Compiled src/couch_replicator_scheduler.erl
Compiled src/couch_replicator_ids.erl
Compiled src/couch_replicator_httpd_util.erl
Compiled src/couch_replicator_httpd.erl
Compiled src/couch_replicator_httpc_pool.erl
Compiled src/couch_replicator_filters.erl
Compiled src/couch_replicator_fabric_rpc.erl
Compiled src/couch_replicator_doc_processor_worker.erl
Compiled src/couch_replicator_fabric.erl
Compiled src/couch_replicator_db_changes.erl
Compiled src/couch_replicator_connection.erl
Compiled src/couch_replicator_docs.erl
Compiled src/couch_replicator_changes_reader.erl
Compiled src/couch_replicator_doc_processor.erl
Compiled src/couch_replicator_clustering.erl
Compiled src/couch_replicator_app.erl
Compiled src/couch_replicator_auth_noop.erl
Compiled src/couch_replicator_auth_session.erl
Compiled src/couch_replicator.erl
Compiled src/couch_replicator_api_wrap.erl
==> couch_plugins (compile)
Compiled src/couch_plugins_httpd.erl
Compiled src/couch_plugins.erl
==> couch_pse_tests (compile)
Compiled src/cpse_test_ref_counting.erl
Compiled src/cpse_util.erl
Compiled src/cpse_test_read_write_docs.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/couch_pse_tests/src/cpse_test_purge_seqs.erl:14: Warning: export_all flag enabled - all functions will be exported
Compiled src/cpse_test_purge_seqs.erl
Compiled src/cpse_test_purge_replication.erl
Compiled src/cpse_test_purge_docs.erl
Compiled src/cpse_test_open_close_delete.erl
Compiled src/cpse_test_purge_bad_checkpoints.erl
Compiled src/cpse_test_get_set_props.erl
Compiled src/cpse_test_fold_purge_infos.erl
Compiled src/cpse_test_fold_changes.erl
Compiled src/cpse_test_fold_docs.erl
Compiled src/cpse_test_copy_purge_infos.erl
Compiled src/cpse_gather.erl
Compiled src/cpse_test_attachments.erl
Compiled src/cpse_test_compaction.erl
==> couch_stats (compile)
Compiled src/couch_stats_sup.erl
Compiled src/couch_stats_app.erl
Compiled src/couch_stats_process_tracker.erl
Compiled src/couch_stats.erl
Compiled src/couch_stats_httpd.erl
Compiled src/couch_stats_aggregator.erl
==> couch_peruser (compile)
Compiled src/couch_peruser_app.erl
Compiled src/couch_peruser_sup.erl
Compiled src/couch_peruser.erl
==> couch_tests (compile)
Compiled setups/couch_epi_dispatch.erl
Compiled src/couch_tests_combinatorics.erl
Compiled src/couch_tests.erl
==> ddoc_cache (compile)
Compiled src/ddoc_cache_value.erl
Compiled src/ddoc_cache_sup.erl
Compiled src/ddoc_cache_opener.erl
Compiled src/ddoc_cache_entry_validation_funs.erl
Compiled src/ddoc_cache_entry_ddocid_rev.erl
Compiled src/ddoc_cache_entry_ddocid.erl
Compiled src/ddoc_cache_entry_custom.erl
Compiled src/ddoc_cache_app.erl
Compiled src/ddoc_cache_lru.erl
Compiled src/ddoc_cache.erl
Compiled src/ddoc_cache_entry.erl
==> dreyfus (compile)
Compiled src/dreyfus_sup.erl
Compiled src/dreyfus_plugin_couch_db.erl
Compiled src/dreyfus_rpc.erl
Compiled src/dreyfus_index_updater.erl
Compiled src/dreyfus_util.erl
Compiled src/dreyfus_index_manager.erl
Compiled src/dreyfus_httpd_handlers.erl
Compiled src/dreyfus_index.erl
Compiled src/dreyfus_fabric_search.erl
Compiled src/dreyfus_fabric_info.erl
Compiled src/dreyfus_httpd.erl
Compiled src/dreyfus_fabric_group2.erl
Compiled src/dreyfus_fabric_cleanup.erl
Compiled src/dreyfus_fabric_group1.erl
Compiled src/dreyfus_epi.erl
Compiled src/dreyfus_config.erl
Compiled src/dreyfus_app.erl
Compiled src/dreyfus_fabric.erl
Compiled src/dreyfus_bookmark.erl
Compiled src/clouseau_rpc.erl
==> fabric (compile)
Compiled src/fabric_db_update_listener.erl
Compiled src/fabric_view_reduce.erl
Compiled src/fabric_view_map.erl
Compiled src/fabric_view_all_docs.erl
Compiled src/fabric_view_changes.erl
Compiled src/fabric_view.erl
Compiled src/fabric_streams.erl
Compiled src/fabric_util.erl
Compiled src/fabric_rpc.erl
Compiled src/fabric_ring.erl
Compiled src/fabric_group_info.erl
Compiled src/fabric_doc_update.erl
Compiled src/fabric_doc_purge.erl
Compiled src/fabric_doc_missing_revs.erl
Compiled src/fabric_doc_open_revs.erl
Compiled src/fabric_doc_atts.erl
Compiled src/fabric_doc_open.erl
Compiled src/fabric_dict.erl
Compiled src/fabric_doc_attachments.erl
Compiled src/fabric_design_doc_count.erl
Compiled src/fabric_db_partition_info.erl
Compiled src/fabric_db_meta.erl
Compiled src/fabric_db_doc_count.erl
Compiled src/fabric_db_info.erl
Compiled src/fabric_db_delete.erl
Compiled src/fabric_db_create.erl
Compiled src/fabric.erl
==> global_changes (compile)
Compiled src/global_changes_sup.erl
Compiled src/global_changes_util.erl
Compiled src/global_changes_plugin.erl
Compiled src/global_changes_httpd_handlers.erl
Compiled src/global_changes_listener.erl
Compiled src/global_changes_server.erl
Compiled src/global_changes_epi.erl
Compiled src/global_changes_app.erl
Compiled src/global_changes_httpd.erl
==> ioq (compile)
Compiled src/ioq_app.erl
Compiled src/ioq_sup.erl
Compiled src/ioq.erl
==> jwtf (compile)
Compiled src/jwtf_app.erl
Compiled src/jwtf_sup.erl
Compiled src/jwtf.erl
Compiled src/jwtf_keystore.erl
==> ken (compile)
Compiled src/ken_sup.erl
Compiled src/ken_event_handler.erl
Compiled src/ken_app.erl
Compiled src/ken.erl
Compiled src/ken_server.erl
==> mango (compile)
Compiled src/mango_sup.erl
Compiled src/mango_sort.erl
Compiled src/mango_util.erl
Compiled src/mango_selector_text.erl
Compiled src/mango_selector.erl
Compiled src/mango_opts.erl
Compiled src/mango_json.erl
Compiled src/mango_json_bookmark.erl
Compiled src/mango_native_proc.erl
Compiled src/mango_idx_special.erl
Compiled src/mango_idx_text.erl
Compiled src/mango_httpd_handlers.erl
Compiled src/mango_idx_view.erl
Compiled src/mango_fields.erl
Compiled src/mango_execution_stats.erl
Compiled src/mango_idx.erl
Compiled src/mango_epi.erl
Compiled src/mango_httpd.erl
Compiled src/mango_error.erl
Compiled src/mango_doc.erl
Compiled src/mango_cursor_text.erl
Compiled src/mango_cursor_special.erl
Compiled src/mango_cursor.erl
Compiled src/mango_app.erl
Compiled src/mango_crud.erl
Compiled src/mango_cursor_view.erl
==> rexi (compile)
Compiled src/rexi_server_sup.erl
Compiled src/rexi_sup.erl
Compiled src/rexi_utils.erl
Compiled src/rexi_server_mon.erl
Compiled src/rexi_monitor.erl
Compiled src/rexi_app.erl
Compiled src/rexi_server.erl
Compiled src/rexi_buffer.erl
Compiled src/rexi.erl
==> setup (compile)
Compiled src/setup_sup.erl
Compiled src/setup_httpd_handlers.erl
Compiled src/setup_app.erl
Compiled src/setup_epi.erl
Compiled src/setup_httpd.erl
Compiled src/setup.erl
==> smoosh (compile)
Compiled src/smoosh_sup.erl
Compiled src/smoosh_utils.erl
Compiled src/smoosh_priority_queue.erl
Compiled src/smoosh_app.erl
Compiled src/smoosh_channel.erl
Compiled src/smoosh_server.erl
Compiled src/smoosh.erl
==> rel (compile)
==> apache-couchdb-3.1.1-6c2d01a (compile)
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN:  'generate' command does not apply to directory /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
... done

    You can now copy the rel/couchdb directory anywhere on your system.
    Start CouchDB with ./bin/couchdb from within that directory.

make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
rm -rf rel/couchdb/var/log
rm -rf rel/couchdb/data
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
 debian/rules binary
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh binary --with=systemd
   dh_testroot
   dh_prep
   dh_installdirs
   dh_install
   dh_installdocs
   dh_installchangelogs
   dh_installman
   dh_installdebconf
   dh_systemd_enable
   dh_installinit
   dh_systemd_start
   dh_installlogrotate
   dh_lintian
   dh_perl
   dh_link
   dh_strip_nondeterminism
   dh_compress
   dh_fixperms
   dh_missing
   dh_strip
   dh_makeshlibs
   debian/rules override_dh_shlibdeps
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_shlibdeps -- --ignore-missing-info -xlibmozjs185-1.0
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_ejson_compare.so debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_icu_driver.so were not linked against libicudata.so.63 (they use none of the library's symbols)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
   dh_installdeb
   dh_gencontrol
   dh_md5sums
   debian/rules override_dh_builddeb
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_builddeb -- -Zgzip
dpkg-deb: building package 'couchdb-dbgsym' in '../couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb'.
dpkg-deb: building package 'couchdb' in '../couchdb_3.1.1-6c2d01a~buster_amd64.deb'.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
 dpkg-genbuildinfo --build=binary
 dpkg-genchanges --build=binary >../couchdb_3.1.1-6c2d01a~buster_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-buildpackage: info: binary-only upload (no source included)
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/.. && lintian --profile couchdb couch*.deb || true
warning: the authors of lintian do not recommend running it with root privileges!
W: couchdb: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:207
N: 3 tags overridden (2 errors, 1 info); 2 unused overrides
chmod a+rwx ../couchdb/couchdb*
mkdir -p pkgs/couch/debian-buster && chmod 777 pkgs/couch/debian-buster
cp ../couchdb/couchdb* pkgs/couch/debian-buster
if [ -f debian/control.bak ]; then mv -f debian/control.bak debian/control; fi
if [ -f rpm/SPECS/couchdb.spec.bak ]; then mv -f rpm/SPECS/couchdb.spec.bak rpm/SPECS/couchdb.spec; fi
rm -rf parts prime stage js/build debian/sm_ver.mk

# packages were created
$ ls -lah pkgs/couch/debian-buster/
total 30M
drwxrwxrwx 2 root root  207 Aug 31 01:58 .
drwxrwxrwx 3 root root   45 Aug 31 01:58 ..
-rwxr-xr-x 1 root root  11K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo
-rwxr-xr-x 1 root root 1.5K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.changes
-rwxr-xr-x 1 root root  30M Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.deb
-rwxr-xr-x 1 root root 474K Aug 31 01:58 couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb

Next use https://github.com/apache/couchdb-docker/tree/main/3.1.1 to produce the container images, but adjust them to include the custom couchdb version.

$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1010, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 1010 (delta 64), reused 64 (delta 32), pack-reused 892
Receiving objects: 100% (1010/1010), 248.11 KiB | 1.19 MiB/s, done.
Resolving deltas: 100% (490/490), done.
$ cd ~/couchdb-docker/3.1.1
$ cp -r ~/couchdb-pkg/pkgs/couch/debian-buster/ .
$ ls debian-buster/
couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo  couchdb_3.1.1-6c2d01a~buster_amd64.changes  couchdb_3.1.1-6c2d01a~buster_amd64.deb  couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb

$ git diff
diff --git a/3.1.1/Dockerfile b/3.1.1/Dockerfile
index 75b4b29..59a90d9 100644
--- a/3.1.1/Dockerfile
+++ b/3.1.1/Dockerfile
@@ -60,6 +60,10 @@ RUN . /etc/os-release; \
     echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
         tee /etc/apt/sources.list.d/couchdb.list >/dev/null

+# I know, we should not include deb packages in images, but it should be good enough to validate the fix
+COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+
 # https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
 RUN set -eux; \
     apt-get update; \
@@ -67,7 +71,7 @@ RUN set -eux; \
     echo "couchdb couchdb/mode select none" | debconf-set-selections; \
 # we DO want recommends this time
     DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
-            couchdb="$COUCHDB_VERSION"~buster \
+    /tmp/packages/* \
     ; \
 # Undo symlinks to /var/log and /var/lib
     rmdir /var/lib/couchdb /var/log/couchdb; \

docker build -t "couchdb:3.1.1-patch-jan" .
Sending build context to Docker daemon  31.45MB
Step 1/20 : FROM debian:buster-slim
 ---> 5c3d57f3e47a
Step 2/20 : LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
 ---> Using cache
 ---> 60f37c14a0e1
Step 3/20 : RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
 ---> Using cache
 ---> 78202346b500
Step 4/20 : RUN set -ex;     apt-get update;     apt-get install -y --no-install-recommends         apt-transport-https         ca-certificates         dirmngr         gnupg      ;     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> fad0f8a736e0
Step 5/20 : RUN set -eux;     apt-get update;     apt-get install -y --no-install-recommends gosu tini;     rm -rf /var/lib/apt/lists/*;     gosu nobody true;     tini --version
 ---> Using cache
 ---> aca1db0792a3
Step 6/20 : ENV GPG_COUCH_KEY     390EF70BB1EA12B2773962950EE62FB37A00258D
 ---> Using cache
 ---> d28f3c244c10
Step 7/20 : RUN set -eux;     apt-get update;     apt-get install -y curl;     export GNUPGHOME="$(mktemp -d)";     curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc;     gpg --batch --import keys.asc;     gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg;     command -v gpgconf && gpgconf --kill all || :;     rm -rf "$GNUPGHOME";     apt-key list;     apt purge -y --autoremove curl;     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> e2949d6cc031
Step 8/20 : ENV COUCHDB_VERSION 3.1.1
 ---> Using cache
 ---> baa8089d1bea
Step 9/20 : RUN . /etc/os-release;     echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" |         tee /etc/apt/sources.list.d/couchdb.list >/dev/null
 ---> Using cache
 ---> a7933b88e802
Step 10/20 : COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
 ---> Using cache
 ---> ded52d34b729
Step 11/20 : COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
 ---> Using cache
 ---> eb9b215cb430
Step 12/20 : RUN set -eux;     apt-get update;         echo "couchdb couchdb/mode select none" | debconf-set-selections;     DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages     /tmp/packages/*     ;     rmdir /var/lib/couchdb /var/log/couchdb;     rm /opt/couchdb/data /opt/couchdb/var/log;     mkdir -p /opt/couchdb/data /opt/couchdb/var/log;     chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log;     chmod 777 /opt/couchdb/data /opt/couchdb/var/log;     rm /opt/couchdb/etc/default.d/10-filelog.ini;     find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +;     find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +;     find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +;     chmod -f 0777 /opt/couchdb/etc/local.d;     rm -rf /var/lib/apt/lists/*;
 ---> Running in 6f530f886258
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb buster InRelease [5151 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb buster/main amd64 Packages [3889 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8473 kB in 3s (3017 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2 libicu63 libk5crypto3
  libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0 libncurses6
  libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps psmisc
  publicsuffix
Suggested packages:
  gpm krb5-doc krb5-user
The following NEW packages will be installed:
  couchdb couchdb-dbgsym curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2
  libicu63 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0
  libncurses6 libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps
  psmisc publicsuffix
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.5 MB/45.0 MB of archives.
After this operation, 104 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 krb5-locales all 1.17-3+deb10u2 [95.5 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5support0 amd64 1.17-3+deb10u2 [65.7 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 libk5crypto3 amd64 1.17-3+deb10u2 [122 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5-3 amd64 1.17-3+deb10u2 [369 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u2 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libprocps7 amd64 2:3.3.15-2 [61.7 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 procps amd64 2:3.3.15-2 [259 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:11 /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb amd64 3.1.1-6c2d01a~buster [30.9 MB]
Get:12 /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym amd64 3.1.1-6c2d01a~buster [485 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u2 [332 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u2 [265 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8300 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libmozjs-60-0 amd64 60.2.3-3 [2785 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 libgpm2 amd64 1.20.7-5 [35.1 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 psmisc amd64 23.2-1 [126 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 13.5 MB in 1s (12.8 MB/s)
Selecting previously unselected package libncurses6:amd64.
(Reading database ... 7118 files and directories currently installed.)
Preparing to unpack .../00-libncurses6_6.1+20181013-2+deb10u2_amd64.deb ...
Unpacking libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Selecting previously unselected package libprocps7:amd64.
Preparing to unpack .../01-libprocps7_2%3a3.3.15-2_amd64.deb ...
Unpacking libprocps7:amd64 (2:3.3.15-2) ...
Selecting previously unselected package procps.
Preparing to unpack .../02-procps_2%3a3.3.15-2_amd64.deb ...
Unpacking procps (2:3.3.15-2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../03-krb5-locales_1.17-3+deb10u2_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../09-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../11-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../12-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../13-libcurl4_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package curl.
Preparing to unpack .../14-curl_7.64.0-4+deb10u2_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u2) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../15-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) ...
Selecting previously unselected package libmozjs-60-0:amd64.
Preparing to unpack .../16-libmozjs-60-0_60.2.3-3_amd64.deb ...
Unpacking libmozjs-60-0:amd64 (60.2.3-3) ...
Selecting previously unselected package couchdb.
Preparing to unpack .../17-couchdb_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package couchdb-dbgsym.
Preparing to unpack .../18-couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../19-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../20-psmisc_23.2-1_amd64.deb ...
Unpacking psmisc (23.2-1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../21-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up psmisc (23.2-1) ...
Setting up libprocps7:amd64 (2:3.3.15-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u2) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u2) ...
Setting up procps (2:3.3.15-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/w.1.gz because associated file /usr/share/man/man1/w.procps.1.gz (of link group w) doesn't exist
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u2) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libmozjs-60-0:amd64 (60.2.3-3) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u2) ...
Setting up curl (7.64.0-4+deb10u2) ...
Setting up couchdb (3.1.1-6c2d01a~buster) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Processing triggers for libc-bin (2.28-10) ...
+ rmdir /var/lib/couchdb /var/log/couchdb
+ rm /opt/couchdb/data /opt/couchdb/var/log
+ mkdir -p /opt/couchdb/data /opt/couchdb/var/log
+ chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log
+ chmod 777 /opt/couchdb/data /opt/couchdb/var/log
+ rm /opt/couchdb/etc/default.d/10-filelog.ini
+ find /opt/couchdb ! ( -user couchdb -group couchdb ) -exec chown -f couchdb:couchdb {} +
+ find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 {} +
+ find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 {} +
+ chmod -f 0777 /opt/couchdb/etc/local.d
+ rm -rf /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_InRelease /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_buster_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_main_binary-amd64_Packages.lz4
Removing intermediate container 6f530f886258
 ---> eaea07a7886e
Step 13/20 : COPY --chown=couchdb:couchdb 10-docker-default.ini /opt/couchdb/etc/default.d/
 ---> f1c9d8e86732
Step 14/20 : COPY --chown=couchdb:couchdb vm.args /opt/couchdb/etc/
 ---> a984087a4ee3
Step 15/20 : COPY docker-entrypoint.sh /usr/local/bin
 ---> 66cd3ececce1
Step 16/20 : RUN ln -s usr/local/bin/docker-entrypoint.sh /docker-entrypoint.sh # backwards compat
 ---> Running in ae10e0e02906
Removing intermediate container ae10e0e02906
 ---> bf8817264923
Step 17/20 : ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
 ---> Running in 1485a79c319f
Removing intermediate container 1485a79c319f
 ---> 737d9f7a9b97
Step 18/20 : VOLUME /opt/couchdb/data
 ---> Running in 6aa4eafbc054
Removing intermediate container 6aa4eafbc054
 ---> 3f111d557168
Step 19/20 : EXPOSE 5984 4369 9100
 ---> Running in 07b6a3204c82
Removing intermediate container 07b6a3204c82
 ---> c70917bea3fd
Step 20/20 : CMD ["/opt/couchdb/bin/couchdb"]
 ---> Running in d61719ef6dea
Removing intermediate container d61719ef6dea
 ---> ca3940266314
Successfully built ca3940266314
Successfully tagged couchdb:3.1.1-patch-jan

I pushed the image and pulled it to my workstation again to have the same underlying hardware as in the previous test.

Results

I adjusted the couchdb-test.sh couchdb-test-validate-patch-3517.zip script to use the patched image and compare it against the vanilla CouchDB 3.1.1 image.

$ ./couchdb-test.sh setup
...
inserting docs cycle 2000/2000
{"ok":true,"id":"a479f05c01bf22c5168ee73617783eb7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784bb6","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784e38","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617785d5b","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617786743","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617787715","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}

$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"84f2eae60ea04fe192284667cb0e4640","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10

real    0m36.391s
user    0m0.014s
sys 0m0.010s
Round: 2/10

real    0m36.426s
user    0m0.009s
sys 0m0.015s
Round: 3/10

real    0m37.448s
user    0m0.015s
sys 0m0.010s
Round: 4/10

real    0m37.182s
user    0m0.010s
sys 0m0.016s
Round: 5/10

real    0m34.802s
user    0m0.013s
sys 0m0.013s
Round: 6/10

real    0m34.877s
user    0m0.009s
sys 0m0.016s
Round: 7/10

real    0m34.204s
user    0m0.007s
sys 0m0.017s
Round: 8/10

real    0m34.092s
user    0m0.009s
sys 0m0.015s
Round: 9/10

real    0m35.571s
user    0m0.014s
sys 0m0.011s
Round: 10/10

real    0m35.790s
user    0m0.010s
sys 0m0.015s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"7df6b96ce9f321de839f65a9a6f264f3","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10

real    0m11.194s
user    0m0.012s
sys 0m0.010s
Round: 2/10

real    0m11.224s
user    0m0.014s
sys 0m0.011s
Round: 3/10

real    0m10.966s
user    0m0.012s
sys 0m0.012s
Round: 4/10

real    0m11.212s
user    0m0.013s
sys 0m0.012s
Round: 5/10

real    0m10.696s
user    0m0.016s
sys 0m0.009s
Round: 6/10

real    0m10.859s
user    0m0.013s
sys 0m0.012s
Round: 7/10

real    0m11.342s
user    0m0.012s
sys 0m0.013s
Round: 8/10

real    0m11.276s
user    0m0.011s
sys 0m0.013s
Round: 9/10

real    0m11.357s
user    0m0.015s
sys 0m0.009s
Round: 10/10

real    0m11.542s
user    0m0.014s
sys 0m0.010s

That's actually great news, we have an average response time from about 11.2s now compared to 35.7s on the vanilla image :-) If possible, I would vote for including your patch in a new 3.x build, as it would be more than 3 times faster for the same queries with unchanged client code (at least for the given dataset).

Still, nothing seem to beat build-in reducers, but at least it would not be way slower to run customer reducers on CouchDB 3 vs. CouchDB 2 installations.

@konrad-ohms thank you for confirming this, we’ll see about getting this into 3.x ✌️

Perfect, thank you very much

@konrad-ohms would you be able to once more test this PR: https://github.com/apache/couchdb/pull/3728 — it uses SHA-256 over crc32, and thus is production ready, we just wanna make sure this still speeds you up :)

Thanks @janl, I compiled your PR and ran the test suite again.

Unfortunately, the second fix in the PR is not performing as good as the prior patch, but it is still better than the original 3.1.1 performance.

Version	Average Performance
3.1.1	36.43s
3.1.1 patch 1	11.45s
3.1.1 patch 2 (PR #3728)	25.25s

Please find my full result output below:

$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"ae7c1c70f02a0b873d0a4f414e1a4d7e","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10

real    0m35.155s
user    0m0.019s
sys 0m0.014s
Round: 2/10

real    0m35.645s
user    0m0.011s
sys 0m0.015s
Round: 3/10

real    0m37.915s
user    0m0.008s
sys 0m0.018s
Round: 4/10

real    0m36.850s
user    0m0.008s
sys 0m0.018s
Round: 5/10

real    0m35.977s
user    0m0.016s
sys 0m0.012s
Round: 6/10

real    0m36.915s
user    0m0.010s
sys 0m0.016s
Round: 7/10

real    0m35.969s
user    0m0.010s
sys 0m0.017s
Round: 8/10

real    0m37.248s
user    0m0.014s
sys 0m0.013s
Round: 9/10

real    0m36.506s
user    0m0.013s
sys 0m0.012s
Round: 10/10

real    0m36.101s
user    0m0.012s
sys 0m0.014s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"54f521952cafc33bdb70987c4fc6cd69","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10

real    0m11.213s
user    0m0.015s
sys 0m0.010s
Round: 2/10

real    0m11.737s
user    0m0.008s
sys 0m0.017s
Round: 3/10

real    0m11.524s
user    0m0.013s
sys 0m0.012s
Round: 4/10

real    0m11.387s
user    0m0.011s
sys 0m0.014s
Round: 5/10

real    0m11.257s
user    0m0.013s
sys 0m0.011s
Round: 6/10

real    0m11.325s
user    0m0.012s
sys 0m0.013s
Round: 7/10

real    0m11.846s
user    0m0.012s
sys 0m0.013s
Round: 8/10

real    0m11.652s
user    0m0.009s
sys 0m0.015s
Round: 9/10

real    0m11.508s
user    0m0.013s
sys 0m0.012s
Round: 10/10

real    0m11.351s
user    0m0.015s
sys 0m0.010s
================ Query CouchDB 3.1.1 patch2 jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"65fd7ec","uuid":"73454c85dfb63b0826f42e25310b6dec","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3005
Round: 1/10

real    0m25.617s
user    0m0.017s
sys 0m0.011s
Round: 2/10

real    0m25.098s
user    0m0.014s
sys 0m0.015s
Round: 3/10

real    0m24.765s
user    0m0.010s
sys 0m0.015s
Round: 4/10

real    0m26.233s
user    0m0.010s
sys 0m0.016s
Round: 5/10

real    0m25.144s
user    0m0.011s
sys 0m0.016s
Round: 6/10

real    0m25.586s
user    0m0.015s
sys 0m0.012s
Round: 7/10

real    0m24.918s
user    0m0.013s
sys 0m0.014s
Round: 8/10

real    0m24.787s
user    0m0.011s
sys 0m0.015s
Round: 9/10

real    0m25.519s
user    0m0.015s
sys 0m0.011s
Round: 10/10

real    0m24.812s
user    0m0.014s
sys 0m0.011s

Thanks for the update! We expected a reduction in performance, but this is quite a jump. Maybe we can think about making this configurable, most folks only ever run trusted design docs anyway and a simpler and faster hashing function might do the trick.

After all, this might not make it into 3.2, but we will remain on it.

Hello,

Do you have any ideas on a date of integration for this patch? We are stuck on couchdb 2.3.1 because of the performance on the map/ reduce design, as our application is mainly based on this feature

Thank

@konrad-ohms @AltenTMAOrion — I put up a new super-simple PR that I’d love for you to try: https://github.com/apache/couchdb/pull/4292

Hey @janl, thanks for picking the issue up again. I tried to rebuild but hit a dependency problem in the docker build, I guess I might have missed overwriting certain dependency versions somewhere...

By any chance, did your CI system put out a docker image I could just use for running the validation?

This is what I tried:

$ git clone https://github.com/apache/couchdb-ci.git
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
==> Downloading Elixir from https://repo.hex.pm/builds/elixir/v1.13.4-otp-23.zip
--> dbab54520ba
STEP 12/14: RUN echo "jenkins ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/jenkins
--> c59f6b6aca0
STEP 13/14: USER jenkins
--> 39e19399d38
STEP 14/14: CMD ["/bin/bash"]
COMMIT apache/couchdbci-debian:buster-erlang-23.3.4.15
--> 617bc6f2f04
Successfully tagged localhost/apache/couchdbci-debian:buster-erlang-23.3.4.15
617bc6f2f04e049eb6ab418828de3239b06ce1a74b756a031dd5f33ea342c05b

$ podman run --name couchdb-fix-3517 -it localhost/apache/couchdbci-debian:buster-erlang-23.3.4.15
jenkins@b730dc2b6711:/$ cd ~
jenkins@b730dc2b6711:~$ whoami
jenkins
jenkins@b730dc2b6711:~$ pwd
/home/jenkins
jenkins@b730dc2b6711:~$ git clone --depth=1 -b fix/3517-part7 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1508, done.
remote: Counting objects: 100% (1508/1508), done.
remote: Compressing objects: 100% (1309/1309), done.
remote: Total 1508 (delta 290), reused 625 (delta 138), pack-reused 0
Receiving objects: 100% (1508/1508), 4.95 MiB | 1.64 MiB/s, done.
Resolving deltas: 100% (290/290), done.
jenkins@b730dc2b6711:~$ cd couchdb/
jenkins@b730dc2b6711:~/couchdb$ git log
commit 05628e891b5145c09e429eb79a21631cddf63c43 (grafted
, HEAD -> fix/3517-part7, origin/fix/3517-part7
)
Author: Jan Lehnardt <jan@apache.org>
Date:   Tue Dec 6 12:50:15 2022 +0100

    fix(3517): super-simplistic fix to avoid costly AST transforms when they are
 not needed
jenkins@b730dc2b6711:~/couchdb$ ./configure
...
jenkins@b730dc2b6711:~/couchdb$ make release
...
WARN:  Missing plugins: [covertool]
WARN:  Missing plugins: [pc]
==> rel (generate)
WARN:  'generate' command does not apply to directory /home/jenkins/couchdb
... done

    You can now copy the rel/couchdb directory anywhere on your system.
    Start CouchDB with ./bin/couchdb from within that directory.

jenkins@b730dc2b6711:~/couchdb$ echo $?
0
jenkins@b730dc2b6711:~/couchdb$ make dist
...
Done: apache-couchdb-3.2.2-05628e8.tar.gz

I copied back the build results and packaged them up:

$ podman cp couchdb-fix-3517:/home/jenkins/couchdb/apache-couchdb-3.2.2-05628e8.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 860, done.
remote: Counting objects: 100% (234/234), done.
remote: Compressing objects: 100% (143/143), done.
remote: Total 860 (delta 103), reused 173 (delta 75), pack-reused 626
Receiving objects: 100% (860/860), 25.34 MiB | 10.53 MiB/s, done.
Resolving deltas: 100% (397/397), done.
$ cd couchdb-pkg/
$ podman tag apache/couchdbci-debian:buster-erlang-23.3.4.15 docker.io/couchdbdev/debian-buster-erlang-23.3.4.15
$ ERLANGVERSION=23.3.4.15 ./build.sh couch debian-buster /home/ohmsk/Downloads/couchdb-test/apache-couchdb-3.2.2-05628e8.tar.gz
...
$ ls -lah pkgs/couch/debian-buster/
total 35M
drwxrwxrwx. 2 ohmsk ohmsk  207 Dec  6 13:48 .
drwxrwxrwx. 3 ohmsk ohmsk   45 Dec  6 13:48 ..
-rwxr-xr-x. 1 ohmsk ohmsk 7.6K Dec  6 13:48 couchdb_3.2.2-05628e8~buster_amd64.buildinfo
-rwxr-xr-x. 1 ohmsk ohmsk 1.5K Dec  6 13:48 couchdb_3.2.2-05628e8~buster_amd64.changes
-rwxr-xr-x. 1 ohmsk ohmsk  27M Dec  6 13:48 couchdb_3.2.2-05628e8~buster_amd64.deb
-rwxr-xr-x. 1 ohmsk ohmsk 8.6M Dec  6 13:48 couchdb-dbgsym_3.2.2-05628e8~buster_amd64.deb

And include the binaries in the image:

$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1093, done.
remote: Counting objects: 100% (201/201), done.
remote: Compressing objects: 100% (97/97), done.
remote: Total 1093 (delta 124), reused 164 (delta 100), pack-reused 892
Receiving objects: 100% (1093/1093), 265.20 KiB | 498.00 KiB/s, done.
Resolving deltas: 100% (550/550), done.

$ cd couchdb-docker/
$ ls
2.3.1      3.1.2-ubi           3.2.0-ubi-clouseau  build.sh     LICENSE
2.3.1-ubi  3.1.2-ubi-clouseau  3.2.1               dev          README.md
3.1.2      3.2.0               3.2.2               dev-cluster

$ cd 3.2.2/
$ cp -r ~/Downloads/couchdb-test/couchdb-pkg/pkgs/couch/debian-buster/ .

$ ls debian-buster/
couchdb_3.2.2-05628e8~buster_amd64.buildinfo
couchdb_3.2.2-05628e8~buster_amd64.changes
couchdb_3.2.2-05628e8~buster_amd64.deb
couchdb-dbgsym_3.2.2-05628e8~buster_amd64.deb

I adjusted the Dockerfile again to just copy deb packages into the image to validate the patch:

$ git diff Dockerfile
diff --git a/3.2.2/Dockerfile b/3.2.2/Dockerfile
index 62177b5..ec8e679 100644
--- a/3.2.2/Dockerfile
+++ b/3.2.2/Dockerfile
@@ -60,6 +60,9 @@ RUN . /etc/os-release; \
     echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
         tee /etc/apt/sources.list.d/couchdb.list >/dev/null

+COPY debian-buster/couchdb_3.2.2-05628e8~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.2.2-05628e8~buster_amd64.deb /tmp/packages/
+
 # https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
 RUN set -eux; \
     apt-get update; \
@@ -67,7 +70,7 @@ RUN set -eux; \
     echo "couchdb couchdb/mode select none" | debconf-set-selections; \
 # we DO want recommends this time
     DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
-            couchdb="$COUCHDB_VERSION"~bullseye \
+        /tmp/packages/* \
     ; \
 # Undo symlinks to /var/log and /var/lib
     rmdir /var/lib/couchdb /var/log/couchdb; \

$ podman build -t "couchdb:3.2.2-patch-jan-fix-3517" .
STEP 1/20: FROM debian:bullseye-slim
STEP 2/20: LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
--> Using cache 09f4784df121998589bdcab66be2643748d0e2db2f49a87c19b52c0a771837f3
--> 09f4784df12
STEP 3/20: RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
--> Using cache 081f8b18791f9a3459ad1f40641f7eb4590a3da68e42a46a636d61f58aff0e4d
--> 081f8b18791
STEP 4/20: RUN set -ex;     apt-get update;     apt-get install -y --no-install-recommends         apt-transport-https         ca-certificates         dirmngr         gnupg      ;     rm -rf /var/lib/apt/lists/*
--> Using cache e683e09e78629b6bff7b85d7f8fd2835c840a299ecbf16d848ca71b69e628edd
--> e683e09e786
STEP 5/20: RUN set -eux;     apt-get update;     apt-get install -y --no-install-recommends gosu tini;     rm -rf /var/lib/apt/lists/*;     gosu nobody true;     tini --version
--> Using cache b0c10606eae929b003afec60212eef33809a96f5acd81bb037f304c1dc0bd7b7
--> b0c10606eae
STEP 6/20: ENV GPG_COUCH_KEY     390EF70BB1EA12B2773962950EE62FB37A00258D
--> Using cache ad3a2134d39a73a793dea21fcd2d178cc4a9fc80d35e7fc952871db417c91c52
--> ad3a2134d39
STEP 7/20: RUN set -eux;     apt-get update;     apt-get install -y curl;     export GNUPGHOME="$(mktemp -d)";     curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc;     gpg --batch --import keys.asc;     gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg;     command -v gpgconf && gpgconf --kill all || :;     rm -rf "$GNUPGHOME";     apt-key list;     apt purge -y --autoremove curl;     rm -rf /var/lib/apt/lists/*
--> Using cache b6575178b4ca6bb472eab38e3b517993a5010c8fb367c1ac5cb9884ae4828412
--> b6575178b4c
STEP 8/20: ENV COUCHDB_VERSION 3.2.2-1
--> Using cache 7177e466f1a1164eaf7f381a033978e5ec1fdb8ca3da819c30a2af691d35305c
--> 7177e466f1a
STEP 9/20: RUN . /etc/os-release;     echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" |         tee /etc/apt/sources.list.d/couchdb.list >/dev/null
--> Using cache 3f504a04eb8d62bd2b1e81dec865b5ccd22cec5f2f2f8793f21a468fa18ee0a9
--> 3f504a04eb8
STEP 10/20: COPY debian-buster/couchdb_3.2.2-05628e8~buster_amd64.deb /tmp/packages/
--> Using cache 58f61fe809b5da2a974b71d729203dc269713c9297e01ab2f70a354bb946ad06
--> 58f61fe809b
STEP 11/20: COPY debian-buster/couchdb-dbgsym_3.2.2-05628e8~buster_amd64.deb /tmp/packages/
--> Using cache 756d9ecfccba323e98d6781562881f596af0590801896f59deb2b3f4d9f3790e
--> 756d9ecfccb
STEP 12/20: RUN set -eux;     apt-get update;         echo "couchdb couchdb/mode select none" | debconf-set-selections;     DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages         /tmp/packages/*     ;     rmdir /var/lib/couchdb /var/log/couchdb;     rm /opt/couchdb/data /opt/couchdb/var/log;     mkdir -p /opt/couchdb/data /opt/couchdb/var/log;     chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log;     chmod 777 /opt/couchdb/data /opt/couchdb/var/log;     rm /opt/couchdb/etc/default.d/10-filelog.ini;     find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +;     find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +;     find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +;     chmod -f 0777 /opt/couchdb/etc/local.d;     rm -rf /var/lib/apt/lists/*;
+ apt-get update
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8184 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [208 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.6 kB]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb bullseye InRelease [5155 B]
Get:8 https://apache.jfrog.io/artifactory/couchdb-deb bullseye/main amd64 Packages [5242 B]
Fetched 8625 kB in 1s (6869 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.2.2-05628e8~buster_amd64.deb /tmp/packages/couchdb_3.2.2-05628e8~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 couchdb : Depends: libmozjs-60-0 but it is not installable
           Depends: libicu63 (>= 63.1-1~) but it is not installable
E: Unable to correct problems, you have held broken packages.
Error: error building at STEP "RUN set -eux;     apt-get update;         echo "couchdb couchdb/mode select none" | debconf-set-selections;     DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages         /tmp/packages/*     ;     rmdir /var/lib/couchdb /var/log/couchdb;     rm /opt/couchdb/data /opt/couchdb/var/log;     mkdir -p /opt/couchdb/data /opt/couchdb/var/log;     chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log;     chmod 777 /opt/couchdb/data /opt/couchdb/var/log;     rm /opt/couchdb/etc/default.d/10-filelog.ini;     find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +;     find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +;     find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +;     chmod -f 0777 /opt/couchdb/etc/local.d;     rm -rf /var/lib/apt/lists/*;": error while running runtime: exit status 100
...

we don't have images for PRs, sorry

feel free to try a newer spidermonkey, 60, 68, 78, 86, 91 all work: https://github.com/apache/couchdb/blob/main/src/couch/rebar.config.script#L62-L71 build CouchDB with ./configure --spidermonkey-version 91

update, I updated the PR with a patch that actually works :D

Okay, thanks, looks like I need to go with libmozjs-78, if I want to use debian bullseye repos:

$ podman run --rm -it debian:bullseye-slim
root@e0167e5dead8:/# apt-get update
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8184 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [209 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.6 kB]
Fetched 8615 kB in 1s (7371 kB/s)                           
Reading package lists... Done
root@e0167e5dead8:/# apt-cache search libmozjs
libmozjs-78-0 - SpiderMonkey JavaScript library
libmozjs-78-dev - SpiderMonkey JavaScript library - development headers

Therefore, I recompiled by doing the following steps (reusing the build container images, just recompile CouchDB), unfortunatly, make release fails in that case (please find the logs below). I tried again with defaults ./configure && make release which seem to work (not attaching those logs again to keep the output a bit more readable).

$ podman run --name couchdb-fix-3517 -it localhost/apache/couchdbci-debian:buster-erlang-23.3.4.15
jenkins@401db7b11852:/$ git clone --depth=1 -b fix/3517-part7 https://github.com/apache/couchdb.git
fatal: could not create work tree dir 'couchdb': Permission denied
jenkins@401db7b11852:/$ cd ~
jenkins@401db7b11852:~$ git clone --depth=1 -b fix/3517-part7 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1508, done.
remote: Counting objects: 100% (1508/1508), done.
remote: Compressing objects: 100% (1309/1309), done.
remote: Total 1508 (delta 289), reused 625 (delta 138), pack-reused 0
Receiving objects: 100% (1508/1508), 4.95 MiB | 18.51 MiB/s, done.
Resolving deltas: 100% (289/289), done.
jenkins@401db7b11852:~$ cd couchdb/
jenkins@401db7b11852:~/couchdb$ git log
commit f6d262aa2bfe5d060cc8798f7d8d00b71f61cfe7 (grafted, HEAD -> fix/3517-part7, origin/fix/3517-part7)
Author: Jan Lehnardt <jan@apache.org>
Date:   Tue Dec 6 12:50:15 2022 +0100

    fix(3517): super-simplistic fix to avoid costly AST transforms when they are not needed
jenkins@401db7b11852:~/couchdb$ ./configure --spidermonkey-version 78
==> configuring couchdb in rel/couchdb.config
Cloning into '/home/jenkins/couchdb/src/rebar'...
remote: Enumerating objects: 8885, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 8885 (delta 4), reused 9 (delta 4), pack-reused 8875
Receiving objects: 100% (8885/8885), 2.16 MiB | 19.44 MiB/s, done.
Resolving deltas: 100% (5705/5705), done.
make: Entering directory '/home/jenkins/couchdb/src/rebar'
./bootstrap
No beam files found.
Recompile: src/rebar
Recompile: src/rebar_abnfc_compiler
Recompile: src/rebar_app_utils
Recompile: src/rebar_appups
Recompile: src/rebar_asn1_compiler
Recompile: src/rebar_base_compiler
Recompile: src/rebar_cleaner
Recompile: src/rebar_config
Recompile: src/rebar_core
Recompile: src/rebar_cover_utils
Recompile: src/rebar_ct
src/rebar_ct.erl:291: Warning: crypto:rand_uniform/2 is deprecated; use rand:uniform/1 instead
Recompile: src/rebar_deps
Recompile: src/rebar_dia_compiler
Recompile: src/rebar_dialyzer
Recompile: src/rebar_edoc
Recompile: src/rebar_erlc_compiler
Recompile: src/rebar_erlydtl_compiler
Recompile: src/rebar_escripter
Recompile: src/rebar_eunit
src/rebar_eunit.erl:282: Warning: crypto:rand_uniform/2 is deprecated; use rand:uniform/1 instead
Recompile: src/rebar_file_utils
Recompile: src/rebar_getopt
Recompile: src/rebar_lfe_compiler
Recompile: src/rebar_log
Recompile: src/rebar_metacmds
Recompile: src/rebar_mustache
Recompile: src/rebar_neotoma_compiler
Recompile: src/rebar_otp_app
Recompile: src/rebar_otp_appup
Recompile: src/rebar_port_compiler
Recompile: src/rebar_proto_compiler
Recompile: src/rebar_proto_gpb_compiler
Recompile: src/rebar_protobuffs_compiler
Recompile: src/rebar_qc
Recompile: src/rebar_rand_compat
Recompile: src/rebar_rel_utils
Recompile: src/rebar_reltool
Recompile: src/rebar_require_vsn
Recompile: src/rebar_shell
Recompile: src/rebar_subdirs
Recompile: src/rebar_templater
Recompile: src/rebar_upgrade
Recompile: src/rebar_utils
Recompile: src/rebar_xref
Recompile: src/rmemo
==> rebar (compile)
==> rebar (escriptize)
Congratulations! You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps.
make: Leaving directory '/home/jenkins/couchdb/src/rebar'
make: Entering directory '/home/jenkins/couchdb/src/rebar'
make: Leaving directory '/home/jenkins/couchdb/src/rebar'
Cloning into '/home/jenkins/couchdb/src/rebar3'...
remote: Enumerating objects: 515, done.
remote: Counting objects: 100% (515/515), done.
remote: Compressing objects: 100% (445/445), done.
remote: Total 515 (delta 73), reused 300 (delta 38), pack-reused 0
Receiving objects: 100% (515/515), 862.32 KiB | 8.89 MiB/s, done.
Resolving deltas: 100% (73/73), done.
Evaluating config script "/home/jenkins/couchdb/src/rebar3/_build/default/lib/rebar/src/rebar.app.src.script"
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling getopt
===> Compiling providers
===> Compiling cf
===> Compiling erlware_commons
===> Compiling bbmustache
===> Compiling relx
===> Compiling eunit_formatters
===> Compiling cth_readable
===> Compiling certifi
===> Compiling ssl_verify_fun
===> Compiling rebar
===> Verifying dependencies...
===> Cleaning out certifi...
===> Cleaning out cf...
===> Cleaning out erlware_commons...
===> Cleaning out getopt...
===> Cleaning out providers...
===> Cleaning out rebar...
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling getopt
===> Compiling providers
===> Compiling cf
===> Compiling erlware_commons
===> Compiling bbmustache
===> Compiling relx
===> Compiling eunit_formatters
===> Compiling cth_readable
===> Compiling certifi
===> Compiling ssl_verify_fun
===> Compiling rebar
===> Building escript for rebar...
Cloning into '/home/jenkins/couchdb/src/erlfmt'...
remote: Enumerating objects: 91, done.
remote: Counting objects: 100% (91/91), done.
remote: Compressing objects: 100% (83/83), done.
remote: Total 91 (delta 6), reused 38 (delta 2), pack-reused 0
Unpacking objects: 100% (91/91), done.
===> Verifying dependencies...
===> Fetching getopt v1.0.2
===> Analyzing applications...
===> Compiling getopt
===> Analyzing applications...
^Z
[1]+  Stopped                 ./configure --spidermonkey-version 78
jenkins@401db7b11852:~/couchdb$ fg
./configure --spidermonkey-version 78
===> Compiling erlfmt
===> Building escript for erlfmt...
===> Verifying dependencies...
===> Cleaning out erlfmt...
==> updating dependencies
WARN:  Expected /home/jenkins/couchdb/src/fauxton to be a raw dependency directory, but no directory found.
==> couch_epi (get-deps)
==> config (get-deps)
==> couch_log (get-deps)
==> khash (get-deps)
WARN:  Missing plugins: [pc]
==> b64url (get-deps)
==> ets_lru (get-deps)
==> chttpd (get-deps)
==> couch (get-deps)
==> couch_event (get-deps)
==> mem3 (get-deps)
==> couch_index (get-deps)
==> couch_mrview (get-deps)
==> couch_replicator (get-deps)
==> couch_plugins (get-deps)
==> couch_pse_tests (get-deps)
==> couch_stats (get-deps)
==> couch_peruser (get-deps)
==> couch_tests (get-deps)
==> couch_dist (get-deps)
==> custodian (get-deps)
==> ddoc_cache (get-deps)
==> dreyfus (get-deps)
==> fabric (get-deps)
==> global_changes (get-deps)
==> ioq (get-deps)
==> jwtf (get-deps)
==> ken (get-deps)
==> mango (get-deps)
==> rexi (get-deps)
==> setup (get-deps)
==> smoosh (get-deps)
==> weatherreport (get-deps)
==> couch_prometheus (get-deps)
==> rel (get-deps)
==> couchdb (get-deps)
WARN:  Expected /home/jenkins/couchdb/src/fauxton to be a raw dependency directory, but no directory found.
Pulling snappy from {git,"https://github.com/apache/couchdb-snappy.git",
                         {tag,"CouchDB-1.0.8"}}
Cloning into 'snappy'...
Pulling fauxton from {git,"https://github.com/apache/couchdb-fauxton",
                          {tag,"v1.2.8"}}
Cloning into 'fauxton'...
Pulling folsom from {git,"https://github.com/apache/couchdb-folsom.git",
                         {tag,"CouchDB-0.8.4"}}
Cloning into 'folsom'...
Pulling hyper from {git,"https://github.com/apache/couchdb-hyper.git",
                        {tag,"CouchDB-2.2.0-7"}}
Cloning into 'hyper'...
Pulling ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
                          {tag,"CouchDB-4.4.2-5"}}
Cloning into 'ibrowse'...
Pulling jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
                        {tag,"1.1.1"}}
Cloning into 'jiffy'...
Pulling mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
                           {tag,"v3.1.1"}}
Cloning into 'mochiweb'...
Pulling meck from {git,"https://github.com/apache/couchdb-meck.git",
                       {tag,"0.9.2"}}
Cloning into 'meck'...
Pulling recon from {git,"https://github.com/apache/couchdb-recon.git",
                        {tag,"2.5.2"}}
Cloning into 'recon'...
Pulling proper from {git,"https://github.com/proper-testing/proper",
                         {tag,"v1.4"}}
Cloning into 'proper'...
==> snappy (get-deps)
==> fauxton (get-deps)
==> meck (get-deps)
==> folsom (get-deps)
Pulling bear from {git,"https://github.com/apache/couchdb-bear.git",
                       "008f48aff819126e281d5ccae80a258bf9bf9c30"}
Cloning into 'bear'...
==> bear (get-deps)
==> hyper (get-deps)
==> ibrowse (get-deps)
==> jiffy (get-deps)
==> mochiweb (get-deps)
==> recon (get-deps)
WARN:  Missing plugins: [covertool]
==> proper (get-deps)
==> couchdb (update-deps)
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
                          {tag,"CouchDB-1.0.8"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
                           {tag,"v1.2.8"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
                          {tag,"CouchDB-0.8.4"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
                         {tag,"CouchDB-2.2.0-7"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
                           {tag,"CouchDB-4.4.2-5"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
                         {tag,"1.1.1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
                            {tag,"v3.1.1"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
                        {tag,"0.9.2"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
                         {tag,"2.5.2"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
                          {tag,"v1.4"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
                        "008f48aff819126e281d5ccae80a258bf9bf9c30"}
WARN:  Missing plugins: [covertool]
You have configured Apache CouchDB, time to relax. Relax.
jenkins@401db7b11852:~/couchdb$ 
jenkins@401db7b11852:~/couchdb$ 
jenkins@401db7b11852:~/couchdb$ 
jenkins@401db7b11852:~/couchdb$ 
jenkins@401db7b11852:~/couchdb$ 
jenkins@401db7b11852:~/couchdb$ make release
fatal: No names found, cannot describe anything.
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
     } catch(std::bad_alloc e) {
                            ^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13:   required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3:   required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78:   required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
     if (offset - 1u < op_ptr_ - op_base_) {
         ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
Compiled src/meck_proc.erl
==> folsom (compile)
Compiled src/folsom_sup.erl
Compiled src/folsom_utils.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_const.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary.erl
Compiled src/hyper_binary_rle.erl
Compiled src/hyper.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_app.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse.erl
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochiweb_html.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochijson2.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_lib.erl
Compiled src/recon_alloc.erl
Compiled src/recon.erl
WARN:  Missing plugins: [covertool]
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_transformer.erl
Compiled src/proper_symb.erl
Compiled src/proper_types.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_sa.erl
Compiled src/proper_queue.erl
Compiled src/proper_shrink.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_gen.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_fsm.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
Compiled src/proper_erlang_abstract_code.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi.erl
==> config (compile)
Compiled src/config_sup.erl
Compiled src/config_util.erl
Compiled src/config_notifier.erl
Compiled src/config_writer.erl
Compiled src/config_listener.erl
Compiled src/config_listener_mon.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
Compiled src/couch_log_trunc_io.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
WARN:  Missing plugins: [pc]
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_util.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_uuids.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_stream.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_util.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_password_hasher.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_flags.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_doc.erl
Compiled src/couch_debug.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_btree.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_db.erl
Compiled src/couch_base32.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_app.erl
Compiled src/couch.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch_att.erl
Compiling /home/jenkins/couchdb/src/couch/priv/couch_js/86/main.cpp
/home/jenkins/couchdb/src/couch/priv/couch_js/86/main.cpp:24:10: fatal error: jsapi.h: No such file or directory
 #include <jsapi.h>
          ^~~~~~~~~
compilation terminated.
ERROR: compile failed while processing /home/jenkins/couchdb/src/couch: rebar_abort
make: *** [Makefile:123: couch] Error 1
jenkins@401db7b11852:~/couchdb$ echo $?
2

I could try rocky-8/ubi8 instead, not sure if it works there or has maybe a different default spidermonkey version available. But --spidermonkey-version 78 seem to be not working on bullseye.

I am sorry for the delay @janl, I noticed that I mixed buster and bullseye releases which obviously does not work. I now downloaded the binaries from the main build from https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/detail/main/420/artifacts and attached it in a docker image (based on https://github.com/apache/couchdb-docker/blob/1759383589c61f7ba2089cefaad4cc36e2785390/3.2.2/Dockerfile).

I adjusted the test script to reference the right images:

# couchdb 2
COUCHDB2_PORT="3002"
COUCHDB2_IMAGE="docker.io/library/couchdb:2.3.1"
COUCHDB2_CONTAINER_NAME="couchdb2"

# couchdb 3
COUCHDB3_PORT="3003"
COUCHDB3_IMAGE="localhost/couchdb:3.2.2-patch-jan-fix-3517-2022_12_16"
COUCHDB3_CONTAINER_NAME="couchdb3"

Afterwards I ran the test script setup again.

$ ./couchdb-test.sh setup
...
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"8035f2eaf1adc282c96275843955c42e","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10

real    0m7.256s
user    0m0.013s
sys 0m0.009s
Round: 2/10

real    0m5.850s
user    0m0.014s
sys 0m0.010s
Round: 3/10

real    0m6.392s
user    0m0.009s
sys 0m0.015s
Round: 4/10

real    0m7.077s
user    0m0.012s
sys 0m0.015s
Round: 5/10

real    0m6.592s
user    0m0.012s
sys 0m0.012s
Round: 6/10

real    0m5.650s
user    0m0.012s
sys 0m0.012s
Round: 7/10

real    0m5.689s
user    0m0.013s
sys 0m0.012s
Round: 8/10

real    0m6.063s
user    0m0.008s
sys 0m0.015s
Round: 9/10

real    0m7.190s
user    0m0.013s
sys 0m0.012s
Round: 10/10

real    0m7.651s
user    0m0.016s
sys 0m0.010s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.2.2","git_sha":"23a2d85e5","uuid":"f317326f8bab2fd1395d065faefe9e21","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10

real    0m17.533s
user    0m0.014s
sys 0m0.013s
Round: 2/10

real    0m19.850s
user    0m0.013s
sys 0m0.013s
Round: 3/10

real    0m17.629s
user    0m0.014s
sys 0m0.012s
Round: 4/10

real    0m16.767s
user    0m0.012s
sys 0m0.014s
Round: 5/10

real    0m20.259s
user    0m0.015s
sys 0m0.010s
Round: 6/10

real    0m16.414s
user    0m0.014s
sys 0m0.011s
Round: 7/10

real    0m16.187s
user    0m0.010s
sys 0m0.016s
Round: 8/10

real    0m18.886s
user    0m0.012s
sys 0m0.014s
Round: 9/10

real    0m17.893s
user    0m0.014s
sys 0m0.013s
Round: 10/10

real    0m16.993s
user    0m0.013s
sys 0m0.012s

This shows, that CouchDB v3 is still slower compared to v2, but way faster than initially, thanks for the fix :slightly_smiling_face:. In the meantime, I migrated my views away from custom reduce functions, so the slower performance is no longer that big of an issue for me personally, but others might still be relying on them and are benefiting from your change.

Update: our QuickJS port has support for SM185-style functions and does not suffer from this. So if you need custom reduces, try out QuickJS: https://docs.couchdb.org/en/stable/best-practices/jsdevel.html#using-quickjs // bit more detail: https://neighbourhood.ie/blog/2024/09/23/couchdb-3-4-0-feature-quickjs

apache / couchdb