Open sergey-safarov opened 6 months ago
That happens when a view shard is opened but has a view signature that's not current. It comes from https://github.com/apache/couchdb/blob/29db2df901207d67e1975ca11a091bcb7f46abcc/src/couch_mrview/src/couch_mrview_index.erl#L141
The expected signature matches file path:
binary:encode_hex(<<250,122,26,158,93,183,246,135,53,41,41,158,169,41,218,173>>).
<<"FA7A1A9E5DB7F6873529299EA929DAAD">>
The other one is:
binary:encode_hex(<<87,225,55,84,101,143,50,221,45,61,42,165,141,102,202,177>>).
<<"57E13754658F32DD2D3D2AA58D66CAB1">>
I don't recall seeing this error too often. Is there any chance your view shard files were moved, copied, restored from backup from an much older couch instance, or mounted on a volume shared across multiple nodes?
Is it easy to reproduce? Just happened once or it's a regular occurrence?
Then it triggers CPU usage and is not responsible CouchDB 3 nodes cluster.
That's expected as the next action after the log is to reset the view shard and rebuild. So in other words, after the view rebuilds, it should be back to normal.
I would also ask that question: have the files been moved or renamed outside of couchdb's control? This error is not normal, and is a protection mechanism (effectively an assertion). another possibility is you have multiple nodes pointing at the same shared volume, and trashing each others state.
Is there any chance your view shard files were moved, copied, restored from backup from an much older couch instance, or mounted on a volume shared across multiple nodes?
I do not think we make something from the above.
We starting docker container and do not move CouchDB files.
Also, we do not use shared volumes.
We make couchdb backup using bulk
requests but do not restore CouchdDB database at this time period.
Is it easy to reproduce? Just happened once or it's a regular occurrence? It is the first time.
after the view rebuilds, it should be back to normal.
This does not happen for 17 minutes and we have recreated the docker container.
Just checked via AWS console, the CouchDB has "Multi-Attach enabled: no". So use volume mounted to only one node.
after the view rebuilds, it should be back to normal.
Also, an issue with the view
file happens on one node in the cluster. But this does not explain why CPU consumed on other nodes.
So view rebuild should happen on one node and two other nodes should be not affected.
I am not sure exactly why more CPU would be consumed on the other nodes. It could be that view shards there hadn't caught up as much as the shard which was reset so they started to get built. When this view was reset perhaps there were active requests waiting to receive rows (responses) from it and so waiting view clients might have been piling up.
You can check for the number of waiting view clients with: https://docs.couchdb.org/en/stable/api/ddoc/common.html#get--db-_design-ddoc-_info
To see if it's any view builds taking place, try using https://docs.couchdb.org/en/stable/api/server/common.html#active-tasks
We have reproduced the same issue on the server without docker.
It appears that it is possible for the design doc signature to vary according to something we have not yet determined. The client's cluster that we (Neighbourhoodie) are investigating gives the following results for one of the design docs affected by a bad index signature:
$ cdb '/' | jq '{ version, git_sha }'
{
"version": "3.3.3",
"git_sha": "40afbcfc7"
}
$ cdb '/{db}/_design/{id}' | md5sum
ef932f29b40bc1795360ea095051d782 -
$ cdb '/{db}/_design/{id}/_info' | jq '.view_index.signature'
"c15a2c0300aefa07abdea125eba80a98"
We copied this design doc and saved into a local dev cluster running the same CouchDB version/commit, and it gave the same md5sum and signature.
However, when putting this same design doc into the CouchDB service installed by Homebrew, it gives the same md5sum but a different signature:
$ cdb '/{db}/_design/{id}/_info' | jq '.view_index.signature'
"20ae103a2d660c16f4cbbc43703ede09"
This indicates that it’s possible for the same design document to produce a different signature on different systems, possibly on different nodes of the same cluster depending on what the cause is.
Environment of each test:
Customer production system: Linux, CouchDB v3.3.3, SpiderMonkey 1.8.5, Erlang 24.3.4.15, ICU library 60.3, collator 153.80, algorithm 10
Local dev cluster: macOS 13.6.4, CouchDB v3.3.3, SpiderMonkey 91, Erlang 25.3.2.9, ICU library 74.2, collator 153.121, algorithm 15.1
Homebrew install: macOS 13.6.4, CouchDB v3.3.3, SpiderMonkey 91, Erlang 26.2.1, ICU library 73.2, collator 153.120, algorithm 15
@jcoglan I suspect if term_to_binary output is different between architectures or erlang versions somehow.
View signatures are computed in: https://github.com/apache/couchdb/blob/33bfa1328ed7f20de2983c9394fb473cf4ca5ecf/src/couch_mrview/src/couch_mrview_util.erl#L280-L281
See if you can an add a log statement and dump the SigInfo
term there and try running the term through the term_to_binary in an erlang prompt on all erlang/architecture versions and see it produces the same result.
Erlang 26 had introduced the deterministic
term_to_binary option so maybe we should start using that (though it only guarantees to be stable for the same Erlang/OTP release).
The other thing that's changed is UTF8 encoding of atoms. That's probably what the problem is here.
See some discussion from last year about it: https://github.com/apache/couchdb/issues/4467#issuecomment-1464005748
(node1@127.0.0.1)17> Views.
[{mrview,0,0,0,[],
[{<<"consumed">>,<<"_sum">>}],
<<"function (doc) { if (doc.pvt_type != 'allotment_consumption' || doc.pvt_deleted) ret"...>>,
nil,[]},
{mrview,1,0,0,
[<<"consumed_by_callid">>],
[],
<<"function (doc) { if (doc.pvt_type != 'allotment_consumption' || doc.pvt_deleted)"...>>,
nil,[]}]
(node1@127.0.0.1)18> Language.
<<"javascript">>
(node1@127.0.0.1)19> DesignOpts.
[]
(node1@127.0.0.1)20> Libs.
[]
(node1@127.0.0.1)21> couch_util:to_hex_bin(couch_hash:md5_hash(term_to_binary({Views, Language, DesignOpts, Libs}))).
<<"c15a2c0300aefa07abdea125eba80a98">>
(node1@127.0.0.1)22> couch_util:to_hex_bin(couch_hash:md5_hash(term_to_binary({Views, Language, DesignOpts, Libs}, [{minor_version, 2}]))).
<<"20ae103a2d660c16f4cbbc43703ede09">>
couchdb 3.3.3 does not use term_to_bin everywhere, so its output will vary based on whether it runs inside OTP 26 or earler.
fixed since 3.3.3 with commit https://github.com/apache/couchdb/commit/453c698a51b36b68e316a7f772d9120bd492670f
noting that couchdb 3.3.3 explicitly rejects OTP 26;
make
==> snappy (compile)
ERROR: OTP release 26 does not match required regex 23|24|25
rebar.config.script: {require_otp_vsn, "23|24|25"},
Good find @rnewson it's the {minor_version, 1}
and atom encoding. Any terms with atoms produced running with Erlang 26 before that would definitely have a different hash.
On 26
> term_to_binary(foo, [{minor_version,1}]).
<<131,100,0,3,102,111,111>>
> term_to_binary(foo).
<<131,119,3,102,111,111>>
But I wonder if there is something else besides atoms there that would cause in-determinism, I'd hope not because that would be terrible. We don't use maps there, ref and pids but binary reference chunks might sneak in...?
@nickva term_to_binary is fragile and we should not be using it for view signatures (at least). The original decision was done I'm sure out of expediency and/or (benign) ignorance. forcing {minor_version, 1} is likely a solid fix for years to come, but the only truly safe path out is to define the view signature algorithm explicitly.
We have reproduced the same issue on the server without docker.
@sergey-safarov that's good news. Can it turned into a script or it's fairly simple to describe steps?
For me, it isn't easy to reproduce. It randomly happens. The CouchDB server works several weeks without this error and then it can happen.
I searched through our logs and found a few of "has the wrong signature" errors as well. In our case they all happened on nodes were being decommissioned and database shards were migrating to new nodes. Wonder if there is a higher chance of it happening if there is a network partition or shard map changes when a the design document updates at the same time...
noting that couchdb 3.3.3 explicitly rejects OTP 26;
I seem to have been naughty when making the 3.3.3 Mac binaries, I swear I did this for a good reason, but I don’t recall it at the moment. I think the 25-jit failed on arm Mac, but I’m not sure: https://github.com/janl/build-couchdb-mac/blob/master/build.sh#L72-L73
None of this is relevant to the issue in the ticket, it just clarifies what @jcoglan reported
@jcoglan and I have come up with a repro for the 'wrong signature' event: https://gist.github.com/jcoglan/0a5feb4af2a496ce10c9b80cf02ea28f. Our theory is that when an index is first queried following a v2->v3 upgrade, https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L121 will rename the file and return the old signature. however, maybe_update_index_file/1 just renames the file, it does not change its content. https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L127 matches and the normal index update path is followed, but if the index is empty then no new content and no new header is written, so the old signature remains in the file. the next time the view is queried, maybe_update_index_file/1 will do nothing (the old file does not exist) and return ok, so we hit this clause where wrong signature appears https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L139
It just reproduced the issue.
The issue happened during the CouchDB upgrade from 2.3.1 to 3.3.1.
I have started daemon. Daemon failed because the user permission for the data folder was wrong. While updating the the CouchDB daemon was stopped on the old server, detached volume from the old server, attached to the new server, and started CouchDB on the new server. On user changed uid
for couchdb
user and daemons failed because does not have permission to data
folder content.
I was making chown
for the data
folder content and then couchdb
service started properly and the issue reproduced.
Now I can see in the logs
couchdb@db-0.example.com <0.3795.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/462e71ffa4b5d3b38b811d08ad72396c.view has the wrong signature: expected: <<70,46,113,255,164,181,211,179,139,129,29,8,173,114,57,108>> but got <<2,65,77,140,1,52,42,68,196,86,58,98,97,23,145,106>>
couchdb@db-0.example.com <0.3922.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/dd51edd8663a1cc90af2f8215cd797be.view has the wrong signature: expected: <<221,81,237,216,102,58,28,201,10,242,248,33,92,215,151,190>> but got <<174,153,108,105,115,213,43,72,131,221,10,174,163,141,228,115>>
couchdb@db-0.example.com <0.3934.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/27fcce55829ac4f09ab0f195d43ad529.view has the wrong signature: expected: <<39,252,206,85,130,154,196,240,154,176,241,149,212,58,213,41>> but got <<241,187,13,21,148,238,32,71,82,114,244,157,164,178,210,214>>
couchdb@db-0.example.com <0.4095.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/b9a49bb8b24a083e07c436f17397fb54.view has the wrong signature: expected: <<185,164,155,184,178,74,8,62,7,196,54,241,115,151,251,84>> but got <<255,60,196,213,171,156,94,131,41,80,206,111,60,138,78,212>>
couchdb@db-0.example.com <0.2473.4> -------- ./data/.shards/c0000000-dfffffff/port_requests.1595155200_design/mrview/d2d3acac282c8c10e0edc3102682ae07.view has the wrong signature: expected: <<210,211,172,172,40,44,140,16,224,237,195,16,38,130,174,7>> but got <<91,44,84,255,172,124,87,248,145,234,222,14,180,50,2,102>>
couchdb@db-0.example.com <0.5771.4> -------- ./data/.shards/00000000-1fffffff/port_requests.1595155200_design/mrview/d2d3acac282c8c10e0edc3102682ae07.view has the wrong signature: expected: <<210,211,172,172,40,44,140,16,224,237,195,16,38,130,174,7>> but got <<91,44,84,255,172,124,87,248,145,234,222,14,180,50,2,102>>
jan's theory makes sense to me. the 2.x -> 3.x upgrade code is flawed, it only tries the 2.0 sig if the file was found in the old location, and assumes we'll write a new header for all views on upgrade (any couchdb crash or restart before that would cause things to go wrong subsequently)
suggest a different approach at https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L121 or so, we can try to match the v2 sig if the v3 sig fails. that is, decouple the moving of the file from old to new location from whether the file itself is v2 vs v3.
Description
On one node (db-2.example.com) we catch error logs
Then it triggers CPU usage and is not responsible CouchDB 3 nodes cluster.
In logs present messages like fabric_worker_timeout get_db_info
fabric_worker_timeout open_doc
fabric_worker_timeout open_doc
Steps to Reproduce
Not known.
Expected Behaviour
Error in the one view should not stop functionality of 3 nodes cluster.
Your Environment
Additional Context
Used
apache/couchdb:3.3.2
docker container.