search

apache / couchdb

Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability
https://couchdb.apache.org/
Apache License 2.0
6.17k stars 1.02k forks source link

view has the wrong signature #4994

Open sergey-safarov opened 6 months ago

sergey-safarov commented 6 months ago

Description

On one node (db-2.example.com) we catch error logs

[error] 2024-02-26T19:53:42.735715Z couchdb@db-2.example.com <0.21142.1038> -------- ./data/.shards/e0000000-ffffffff/account/a1/15/78067a1db9f83fa4da53f6bacc16.1663539188_design/mrview/fa7a1a9e5db7f6873529299ea929daad.view has the wrong signature: expected: <<250,122,26,158,93,183,246,135,53,41,41,158,169,41,218,173>> but got <<87,225,55,84,101,143,50,221,45,61,42,165,141,102,202,177>> [error] 2024-02-26T19:54:07.738902Z couchdb@db-2.example.com <0.22173.1076> -------- ./data/.shards/40000000-5fffffff/account/a1/15/78067a1db9f83fa4da53f6bacc16.1663539188_design/mrview/fa7a1a9e5db7f6873529299ea929daad.view has the wrong signature: expected: <<250,122,26,158,93,183,246,135,53,41,41,158,169,41,218,173>> but got <<87,225,55,84,101,143,50,221,45,61,42,165,141,102,202,177>>

Then it triggers CPU usage and is not responsible CouchDB 3 nodes cluster.

In logs present messages like fabric_worker_timeout get_db_info

[error] 2024-02-26T20:00:39.603653Z couchdb@db-2.example.com <0.26790.1086> 182b91a95a fabric_worker_timeout get_db_info,'couchdb@db-0.example.com',<<"shards/20000000-3fffffff/account/56/3b/1e36302175513343e6a13f6a1372-202308.1690848013">>

fabric_worker_timeout open_doc

[error] 2024-02-26T20:00:40.479851Z couchdb@db-2.example.com <0.2074.1115> -------- fabric_worker_timeout open_doc,'couchdb@db-2.example.com',<<"shards/40000000-5fffffff/account/e3/2d/d743a02e6ee4e28f02755e070d33.1685539862">>

fabric_worker_timeout open_doc

[error] 2024-02-26T20:00:45.480715Z couchdb@db-2.example.com <0.26670.936> a6eca4ce74 fabric_worker_timeout open_doc,'couchdb@db-1.example.com',<<"shards/40000000-5fffffff/account/e3/2d/d743a02e6ee4e28f02755e070d33.1685539862">>

Steps to Reproduce

Not known.

Expected Behaviour

Error in the one view should not stop functionality of 3 nodes cluster.

Your Environment

Additional Context

Used apache/couchdb:3.3.2 docker container.

nickva commented 6 months ago

That happens when a view shard is opened but has a view signature that's not current. It comes from https://github.com/apache/couchdb/blob/29db2df901207d67e1975ca11a091bcb7f46abcc/src/couch_mrview/src/couch_mrview_index.erl#L141

The expected signature matches file path:

binary:encode_hex(<<250,122,26,158,93,183,246,135,53,41,41,158,169,41,218,173>>).
<<"FA7A1A9E5DB7F6873529299EA929DAAD">>

The other one is:

binary:encode_hex(<<87,225,55,84,101,143,50,221,45,61,42,165,141,102,202,177>>).
<<"57E13754658F32DD2D3D2AA58D66CAB1">>

I don't recall seeing this error too often. Is there any chance your view shard files were moved, copied, restored from backup from an much older couch instance, or mounted on a volume shared across multiple nodes?

Is it easy to reproduce? Just happened once or it's a regular occurrence?

Then it triggers CPU usage and is not responsible CouchDB 3 nodes cluster.

That's expected as the next action after the log is to reset the view shard and rebuild. So in other words, after the view rebuilds, it should be back to normal.

rnewson commented 6 months ago

I would also ask that question: have the files been moved or renamed outside of couchdb's control? This error is not normal, and is a protection mechanism (effectively an assertion). another possibility is you have multiple nodes pointing at the same shared volume, and trashing each others state.

sergey-safarov commented 6 months ago

Is there any chance your view shard files were moved, copied, restored from backup from an much older couch instance, or mounted on a volume shared across multiple nodes?

I do not think we make something from the above. We starting docker container and do not move CouchDB files. Also, we do not use shared volumes. We make couchdb backup using bulk requests but do not restore CouchdDB database at this time period.

Is it easy to reproduce? Just happened once or it's a regular occurrence? It is the first time.

after the view rebuilds, it should be back to normal.

This does not happen for 17 minutes and we have recreated the docker container.

sergey-safarov commented 6 months ago

Just checked via AWS console, the CouchDB has "Multi-Attach enabled: no". So use volume mounted to only one node.

sergey-safarov commented 6 months ago

after the view rebuilds, it should be back to normal.

Also, an issue with the view file happens on one node in the cluster. But this does not explain why CPU consumed on other nodes. So view rebuild should happen on one node and two other nodes should be not affected.

nickva commented 6 months ago

I am not sure exactly why more CPU would be consumed on the other nodes. It could be that view shards there hadn't caught up as much as the shard which was reset so they started to get built. When this view was reset perhaps there were active requests waiting to receive rows (responses) from it and so waiting view clients might have been piling up.

You can check for the number of waiting view clients with: https://docs.couchdb.org/en/stable/api/ddoc/common.html#get--db-_design-ddoc-_info

To see if it's any view builds taking place, try using https://docs.couchdb.org/en/stable/api/server/common.html#active-tasks

sergey-safarov commented 5 months ago

We have reproduced the same issue on the server without docker.

jcoglan commented 5 months ago

It appears that it is possible for the design doc signature to vary according to something we have not yet determined. The client's cluster that we (Neighbourhoodie) are investigating gives the following results for one of the design docs affected by a bad index signature:

$ cdb '/' | jq '{ version, git_sha }'
{
  "version": "3.3.3",
  "git_sha": "40afbcfc7"
}

$ cdb '/{db}/_design/{id}' | md5sum
ef932f29b40bc1795360ea095051d782  -

$ cdb '/{db}/_design/{id}/_info' | jq '.view_index.signature'
"c15a2c0300aefa07abdea125eba80a98"

We copied this design doc and saved into a local dev cluster running the same CouchDB version/commit, and it gave the same md5sum and signature.

However, when putting this same design doc into the CouchDB service installed by Homebrew, it gives the same md5sum but a different signature:

$ cdb '/{db}/_design/{id}/_info' | jq '.view_index.signature'
"20ae103a2d660c16f4cbbc43703ede09"

This indicates that it’s possible for the same design document to produce a different signature on different systems, possibly on different nodes of the same cluster depending on what the cause is.

Environment of each test:

nickva commented 5 months ago

@jcoglan I suspect if term_to_binary output is different between architectures or erlang versions somehow.

View signatures are computed in: https://github.com/apache/couchdb/blob/33bfa1328ed7f20de2983c9394fb473cf4ca5ecf/src/couch_mrview/src/couch_mrview_util.erl#L280-L281

See if you can an add a log statement and dump the SigInfo term there and try running the term through the term_to_binary in an erlang prompt on all erlang/architecture versions and see it produces the same result.

Erlang 26 had introduced the deterministic term_to_binary option so maybe we should start using that (though it only guarantees to be stable for the same Erlang/OTP release).

The other thing that's changed is UTF8 encoding of atoms. That's probably what the problem is here.

See some discussion from last year about it: https://github.com/apache/couchdb/issues/4467#issuecomment-1464005748

rnewson commented 5 months ago 
(node1@127.0.0.1)17> Views.
[{mrview,0,0,0,[],
         [{<<"consumed">>,<<"_sum">>}],
         <<"function (doc) { if (doc.pvt_type != 'allotment_consumption' || doc.pvt_deleted) ret"...>>,
         nil,[]},
 {mrview,1,0,0,
         [<<"consumed_by_callid">>],
         [],
         <<"function (doc) { if (doc.pvt_type != 'allotment_consumption' || doc.pvt_deleted)"...>>,
         nil,[]}]
(node1@127.0.0.1)18> Language.
<<"javascript">>
(node1@127.0.0.1)19> DesignOpts.
[]
(node1@127.0.0.1)20> Libs.
[]
(node1@127.0.0.1)21> couch_util:to_hex_bin(couch_hash:md5_hash(term_to_binary({Views, Language, DesignOpts, Libs}))).
<<"c15a2c0300aefa07abdea125eba80a98">>
(node1@127.0.0.1)22> couch_util:to_hex_bin(couch_hash:md5_hash(term_to_binary({Views, Language, DesignOpts, Libs}, [{minor_version, 2}]))).
<<"20ae103a2d660c16f4cbbc43703ede09">>

couchdb 3.3.3 does not use term_to_bin everywhere, so its output will vary based on whether it runs inside OTP 26 or earler.

rnewson commented 5 months ago

fixed since 3.3.3 with commit https://github.com/apache/couchdb/commit/453c698a51b36b68e316a7f772d9120bd492670f

rnewson commented 5 months ago

noting that couchdb 3.3.3 explicitly rejects OTP 26;

make
==> snappy (compile)
ERROR: OTP release 26 does not match required regex 23|24|25
rnewson commented 5 months ago 
rebar.config.script:    {require_otp_vsn, "23|24|25"},
nickva commented 5 months ago

Good find @rnewson it's the {minor_version, 1} and atom encoding. Any terms with atoms produced running with Erlang 26 before that would definitely have a different hash.

On 26

> term_to_binary(foo, [{minor_version,1}]).
<<131,100,0,3,102,111,111>>

> term_to_binary(foo).
<<131,119,3,102,111,111>>

But I wonder if there is something else besides atoms there that would cause in-determinism, I'd hope not because that would be terrible. We don't use maps there, ref and pids but binary reference chunks might sneak in...?

rnewson commented 5 months ago

@nickva term_to_binary is fragile and we should not be using it for view signatures (at least). The original decision was done I'm sure out of expediency and/or (benign) ignorance. forcing {minor_version, 1} is likely a solid fix for years to come, but the only truly safe path out is to define the view signature algorithm explicitly.

nickva commented 5 months ago

We have reproduced the same issue on the server without docker.

@sergey-safarov that's good news. Can it turned into a script or it's fairly simple to describe steps?

sergey-safarov commented 5 months ago

For me, it isn't easy to reproduce. It randomly happens. The CouchDB server works several weeks without this error and then it can happen.

nickva commented 5 months ago

I searched through our logs and found a few of "has the wrong signature" errors as well. In our case they all happened on nodes were being decommissioned and database shards were migrating to new nodes. Wonder if there is a higher chance of it happening if there is a network partition or shard map changes when a the design document updates at the same time...

janl commented 5 months ago

noting that couchdb 3.3.3 explicitly rejects OTP 26;

I seem to have been naughty when making the 3.3.3 Mac binaries, I swear I did this for a good reason, but I don’t recall it at the moment. I think the 25-jit failed on arm Mac, but I’m not sure: https://github.com/janl/build-couchdb-mac/blob/master/build.sh#L72-L73

None of this is relevant to the issue in the ticket, it just clarifies what @jcoglan reported

janl commented 5 months ago

@jcoglan and I have come up with a repro for the 'wrong signature' event: https://gist.github.com/jcoglan/0a5feb4af2a496ce10c9b80cf02ea28f. Our theory is that when an index is first queried following a v2->v3 upgrade, https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L121 will rename the file and return the old signature. however, maybe_update_index_file/1 just renames the file, it does not change its content. https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L127 matches and the normal index update path is followed, but if the index is empty then no new content and no new header is written, so the old signature remains in the file. the next time the view is queried, maybe_update_index_file/1 will do nothing (the old file does not exist) and return ok, so we hit this clause where wrong signature appears https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L139

sergey-safarov commented 1 month ago

It just reproduced the issue. The issue happened during the CouchDB upgrade from 2.3.1 to 3.3.1. I have started daemon. Daemon failed because the user permission for the data folder was wrong. While updating the the CouchDB daemon was stopped on the old server, detached volume from the old server, attached to the new server, and started CouchDB on the new server. On user changed uid for couchdb user and daemons failed because does not have permission to data folder content. I was making chown for the data folder content and then couchdb service started properly and the issue reproduced. Now I can see in the logs

couchdb@db-0.example.com <0.3795.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/462e71ffa4b5d3b38b811d08ad72396c.view has the wrong signature: expected: <<70,46,113,255,164,181,211,179,139,129,29,8,173,114,57,108>> but got <<2,65,77,140,1,52,42,68,196,86,58,98,97,23,145,106>>
couchdb@db-0.example.com <0.3922.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/dd51edd8663a1cc90af2f8215cd797be.view has the wrong signature: expected: <<221,81,237,216,102,58,28,201,10,242,248,33,92,215,151,190>> but got <<174,153,108,105,115,213,43,72,131,221,10,174,163,141,228,115>>
couchdb@db-0.example.com <0.3934.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/27fcce55829ac4f09ab0f195d43ad529.view has the wrong signature: expected: <<39,252,206,85,130,154,196,240,154,176,241,149,212,58,213,41>> but got <<241,187,13,21,148,238,32,71,82,114,244,157,164,178,210,214>>
couchdb@db-0.example.com <0.4095.3> -------- ./data/.shards/e0000000-ffffffff/account/44/3c/44a301b106b24f0fc1fd061d3e26.1668526196_design/mrview/b9a49bb8b24a083e07c436f17397fb54.view has the wrong signature: expected: <<185,164,155,184,178,74,8,62,7,196,54,241,115,151,251,84>> but got <<255,60,196,213,171,156,94,131,41,80,206,111,60,138,78,212>>
couchdb@db-0.example.com <0.2473.4> -------- ./data/.shards/c0000000-dfffffff/port_requests.1595155200_design/mrview/d2d3acac282c8c10e0edc3102682ae07.view has the wrong signature: expected: <<210,211,172,172,40,44,140,16,224,237,195,16,38,130,174,7>> but got <<91,44,84,255,172,124,87,248,145,234,222,14,180,50,2,102>>
couchdb@db-0.example.com <0.5771.4> -------- ./data/.shards/00000000-1fffffff/port_requests.1595155200_design/mrview/d2d3acac282c8c10e0edc3102682ae07.view has the wrong signature: expected: <<210,211,172,172,40,44,140,16,224,237,195,16,38,130,174,7>> but got <<91,44,84,255,172,124,87,248,145,234,222,14,180,50,2,102>>
rnewson commented 1 month ago

jan's theory makes sense to me. the 2.x -> 3.x upgrade code is flawed, it only tries the 2.0 sig if the file was found in the old location, and assumes we'll write a new header for all views on upgrade (any couchdb crash or restart before that would cause things to go wrong subsequently)

suggest a different approach at https://github.com/apache/couchdb/blob/3.3.3/src/couch_mrview/src/couch_mrview_index.erl#L121 or so, we can try to match the v2 sig if the v3 sig fails. that is, decouple the moving of the file from old to new location from whether the file itself is v2 vs v3.