AtnNn
commented
7 years ago @timmaxw @danielmewes Do you know what might cause this?
Open RubenKelevra opened 7 years ago
AtnNn
commented
7 years ago @timmaxw @danielmewes Do you know what might cause this?
timmaxw
commented
7 years ago Background information:
This backtrace is appearing on the Raft leader for a table. Each Raft follower for a table sends contract_ack_ts to the Raft leader for that table. These contract_ack_ts are tagged with the pair<server_id_t, contract_id_t> where the first one is the server sending the ack and the second is the ID of the contract being acked. (contract_ack_t is an alias for uuid_u which is why it appears as uuid_u in the backtrace.)
More specifically, there is a minidir_read_manager_t on the Raft leader and a minidir_write_manager_t on each Raft follower. Each Raft follower gives the minidir_write_manager_t a watchable_map_t<pair<...>,contract_ack_t>; the minidir_write_manager_t transmits the initial value and subsequent updates to the minidir_read_manager_t. The minidir_read_manager_t then aggregates them into a new watchable_map_t<pair<...>,contract_ack_t>, and continuously applies the updates it receives. If a Raft follower disconnects, its entries should disappear from the minidir_write_manager_t's map. So the idea is that the watchable_map_t coming out of the minidir_read_manager_t on the Raft leader will always be the union of the watchable_map_ts going into the minidir_write_manager_ts on the Raft followers that are in network contact (modulo network lag).
Suppose two of the Raft followers have the same key in the watchable_map_t that they pass to the minidir_write_manager_t. Then when the minidir_read_manager_t tries to unify the maps it will see the conflicting key and produce the backtrace we're seeing here. But that shouldn't happen, because the key is a pair<server_id_t,...>, and the two servers have different server IDs. (We've occasionally had problems in the past with two servers getting the same server ID, but in that case we'd see an error immediately at startup, so I don't think that's going on here.)
What I think is actually happening is that the entry from the server buri is conflicting with an earlier version of itself. Specifically, I suspect the following sequence of events:
buri (a Raft follower) is connected to this server (the Raft leader) and so it has an entry in the map like normalburi shuts down for some reason. The connection to this server drops.minidir_read_manager_t::on_connection_change() here. The correct behavior in this case is to remove the entries from the watchable_map_t because the server that sent them is no longer connected. However, instead of removing the entries immediately we spawn a coroutine to do it.buri starts up again and reconnects to this server. Because buri restarted completely, it has a new peer_id_t, so the minidir_read_manager_t treats it as a separate connection. However, it still has the same server_id_t as the old connection (which hasn't been cleaned up yet) and so we see this conflict.@RubenKelevra can you tell us whether buri restarted immediately before this error occurred?
If my hypothesis is correct, then we should fix on_connection_change() by making it clean up the entries from the watchable_map_t immediately instead of spawning a coroutine to do it. The current reason to spawn a coroutine is that the entries get cleaned up in the destructor of the peer_data_t, but there might be some instances of minidir_read_manager_t::on_update() that are blocking on the fifo_enforcer_sink_t, and they are holding pointers to the peer_data_t, so we can't delete the peer_data_t until those coroutines exit. I propose the following patch:
bool is_closing; on peer_data_t.on_connection_change(), immediately iterate over all the link_data_ts in peer_data->link_map and call link_data->map.clear() on each one to destroy the watchable_map_t::entry_t objects. Also, immediately set peer_data->is_closing = true, but don't actually delete the peer_data yet; we'll still spawn a coroutine to actually delete the peer_data.on_update(), in this conditional, replace peer_keepalive.get_drain_signal()->is_pulsed() with peer_keepalive.get_drain_signal()->is_pulsed() || peer_data->is_closing so that on_update() doesn't create a new entry_t after we called link_data->map.clear().timmaxw
commented
7 years ago (Note for people following along by email: I edited the above comment.)
RubenKelevra
commented
7 years ago A different server of the same cluster crashed three times within several days with the same message - maybe my setup is triggering a racecondition which happens if the connections are a but flunky...? :)
-- Reboot --
Jul 15 08:48:21 vidar systemd[1]: Started RethinkDB database server for instance 'freifunk'.
Jul 15 08:48:22 vidar rethinkdb[1588]: Warning: The following options are not used by this invocation of rethinkdb and will be ignored: server-name
Jul 15 08:48:23 vidar rethinkdb[1588]: Running rethinkdb 2.3.5 (GCC 7.1.1)...
Jul 15 08:48:23 vidar rethinkdb[1588]: Running on Linux 4.11.9-1-ARCH x86_64
Jul 15 08:48:23 vidar rethinkdb[1588]: Loading data from directory /var/lib/rethinkdb/default
Jul 15 08:48:23 vidar rethinkdb[1588]: Listening for intracluster connections on port 29015
Jul 15 08:48:58 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 15 08:48:59 vidar rethinkdb[1588]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 15 08:48:59 vidar rethinkdb[1588]: Connected to server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 15 08:48:59 vidar rethinkdb[1588]: Listening for client driver connections on port 28015
Jul 15 08:48:59 vidar rethinkdb[1588]: Administrative HTTP connections are disabled.
Jul 15 08:48:59 vidar rethinkdb[1588]: Listening on cluster addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 15 08:48:59 vidar rethinkdb[1588]: Listening on driver addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 15 08:48:59 vidar rethinkdb[1588]: Listening on http addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 15 08:48:59 vidar rethinkdb[1588]: To fully expose RethinkDB on the network, bind to all addresses by adding `bind=all' to the config file (/etc/rethinkdb/instances.d/freifunk.conf).
Jul 15 08:48:59 vidar rethinkdb[1588]: Server ready, "vidar" 13a61804-84be-4089-886e-19d490858ef2
Jul 15 08:49:13 vidar rethinkdb[1588]: warn: We were unable to connect to the following peer, or the --join address does not match the peer's canonical address: 10.206.0.7:29015
Jul 15 09:01:54 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/be883d27-58b3-49fb-86dc-43e08377ff44
Jul 15 09:38:51 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/69a9f39c-6e55-488e-961a-93951ff13e45
Jul 15 09:39:23 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/15f17839-c4f7-4157-a194-56a950e1b059
Jul 15 09:39:23 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/3cf14e1d-4631-4d46-b71c-078e71b835a1
Jul 15 09:39:24 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/c4c5910d-421c-4bfb-8285-dcec9bc9f281
Jul 15 09:39:24 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/96aab87f-4bb8-41ea-8a36-8664f632c90e
Jul 15 09:39:24 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/80d0ae28-79ab-4a7b-92ab-63aba3b509a6
Jul 15 09:39:24 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/62581149-7130-41be-9f32-a3f75738425d
Jul 15 09:39:25 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/27bd4b31-7fe7-4f5a-9cd4-5c5a390b5796
Jul 15 09:39:26 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/07894a7f-54e6-4908-902c-cce1e51debeb
Jul 15 09:39:26 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/7be6f213-cc82-47be-b972-23ef82934283
Jul 15 09:39:26 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/db308341-8f58-4fa9-9947-ca0aa244d687
Jul 15 09:39:26 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/87b89ee0-bbc2-4be8-b886-6de73b4d68ba
Jul 15 09:45:03 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/826e19f7-90df-4fb7-8f53-ba739786e872
Jul 15 09:58:52 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/4623f183-07ee-4779-a2be-bf28e0a1b498
Jul 15 09:58:53 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/462536e4-3391-47d2-b272-5dcff4e9cfa8
Jul 15 09:58:53 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/ffab4cbd-12dd-4b58-9a4b-eedde1cd65d5
Jul 15 09:58:53 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/5ba7db21-34e6-40b9-ae30-51241fbb1736
Jul 15 09:58:54 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/42392196-6f28-4068-b9ab-61cecc68330e
Jul 15 09:58:54 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/35352f72-8454-417a-9b07-2cbda35b64ca
Jul 15 09:58:54 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/882c73b2-fec1-4a83-b6c0-29024dbe9f3f
Jul 15 09:58:54 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/fe998433-977c-4fef-ba0c-28cfcfaee0e0
Jul 15 09:58:55 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/fd942d69-e224-42cf-9dbf-9e4abbb8a8d0
Jul 15 09:58:55 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/aae15672-dbdc-4b9b-aecc-8cf8288eb22d
Jul 15 09:58:56 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/4ebc9ace-39c7-433c-b56e-9cb39d2ff70f
Jul 15 09:58:56 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/e365b669-2f11-4e5b-971d-fe7800dedf17
Jul 15 09:58:56 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/53adcb4a-f03c-4935-86ba-fae466f53aa6
Jul 15 09:58:56 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/9a102314-8a12-4035-8847-df805d117760
Jul 15 09:58:56 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/942192ae-3295-4d91-9d3f-c6670276ee26
Jul 15 09:58:56 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/b34bcb35-5c03-4ee2-8dad-5f65a1f83493
Jul 15 09:58:57 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/737393a8-93cf-41c5-af30-5ca7eff90ec0
Jul 15 09:58:57 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/d604794b-4608-45e6-8e3e-788f4b8c0ba1
Jul 15 09:58:57 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/e2e98b84-5362-48f8-8bae-f8a5a8ab66f4
Jul 15 09:58:57 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/70e3db69-5d3a-40b7-8012-1d9762acb294
Jul 15 09:58:58 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/d06cca25-1d13-4cf1-84e5-46d10274eb26
Jul 15 09:58:58 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/6d698979-37db-44cd-b138-c0ae29294a67
Jul 15 09:58:59 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/9c6d018d-7144-4f67-8f14-ca6b156ea406
Jul 15 09:58:59 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/822f5fba-5d33-4494-a96f-885fad6267bf
Jul 15 09:58:59 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/05929549-4ae3-474f-86cc-b29cc9b3749e
Jul 15 09:59:02 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/9c8b725a-faa5-4485-b1ea-119d383f408d
Jul 15 09:59:03 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/6a5b34e4-8d1c-4945-9f60-2377f1a569a6
Jul 15 09:59:03 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/f69079ef-138d-4a1a-951b-8e6ddaab9b71
Jul 15 09:59:03 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/3f225c81-f7a9-4aa4-a239-d74669d89f25
Jul 15 09:59:04 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/7e807593-405f-4d91-9a12-15a07af9da14
Jul 15 09:59:04 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/710de3d3-6037-4740-be11-c32522358de3
Jul 15 09:59:04 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/c0efabae-cb59-4751-8f29-1765bcf31d87
Jul 15 09:59:05 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/0c45e95e-cd9d-4cb2-9033-34e61234b0fc
Jul 15 09:59:06 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/2195ee0c-a5b1-4eff-a14f-27742adba79d
Jul 15 09:59:06 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/1d57e6e5-93c8-4b97-912b-ec68adb28b1f
Jul 15 09:59:06 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/0c642756-b88a-4d84-bbd3-58d3ae993605
Jul 15 09:59:09 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/9f1c37c9-1b87-4c41-b0d3-c338e4c142d2
Jul 15 09:59:11 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/b4cfef7d-9a59-4b2b-9f6e-813d786a3786
Jul 15 09:59:13 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/c87ed5d6-d352-4be4-8172-0fa94bc1a116
Jul 15 09:59:15 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/6e75fb57-2836-400f-be6b-74630606bb3e
Jul 15 09:59:17 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/0fa4bbdf-e56e-4615-8ba6-93e37bb5588d
Jul 15 09:59:19 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/0ca3a0e8-e24d-4b6b-8b36-bce0c7254463
Jul 15 09:59:21 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/c367ae0e-498b-47a8-80e4-aa35193554f5
Jul 15 09:59:23 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/0854afda-aec6-4da0-baf6-b93f02c25391
Jul 15 09:59:25 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/d60a5e0e-a242-4aa0-ba35-0e2c1513bc2b
Jul 15 09:59:27 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/0022675e-5366-4ea5-80b0-f4204f8aa51d
Jul 15 09:59:29 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/60967540-03c8-4d0b-be75-a429211cf400
Jul 15 09:59:31 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/4c469bfc-f215-435a-9758-d4c3e569ac85
Jul 15 09:59:33 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/11bd7136-b455-496b-81ac-dd4b0feb150a
Jul 15 09:59:35 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/a308244b-61a3-468b-8dac-e5fe4a013d85
Jul 15 09:59:37 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/b85bd0e3-ca44-4268-9524-7a35dea51d00
Jul 15 09:59:39 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/3de19a33-e8c4-442d-bba8-b538e34ae4ab
Jul 15 09:59:41 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/9a09567f-c4ec-4b21-a691-a4e50724cf46
Jul 15 09:59:43 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/03fe778d-b00c-44f8-b249-db4b2804d805
Jul 15 10:00:30 vidar rethinkdb[1588]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 15 10:00:31 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 15 10:00:33 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 15 18:00:19 vidar rethinkdb[1588]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:54887
Jul 15 18:00:19 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 15 18:00:36 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 15 21:52:35 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/a605e566-ef55-4a25-8c5a-ab9a26b0cf73
Jul 15 21:52:36 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/bd1ff98f-cf48-4fa8-a3c1-fef57a5aa136
Jul 15 21:52:37 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/d454cb38-db5c-446c-84b9-c24e7569580b
Jul 15 22:18:32 vidar rethinkdb[1588]: Removing file /var/lib/rethinkdb/default/70a746f7-1d5e-474b-b96d-6a6e886a2e88
Jul 16 01:00:32 vidar rethinkdb[1588]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:60737
Jul 16 01:00:32 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:01:08 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:01:35 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:01:53 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:03:40 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:03:40 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:04:51 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:05:09 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:05:41 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:05:59 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:08:19 vidar rethinkdb[1588]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:08:24 vidar rethinkdb[1588]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 01:08:25 vidar rethinkdb[1588]: Disconnected from server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 16 01:08:34 vidar rethinkdb[1588]: Version: rethinkdb 2.3.5 (GCC 7.1.1)
Jul 16 01:08:34 vidar rethinkdb[1588]: error: Error in ./src/concurrency/watchable_map.tcc at line 98:
Jul 16 01:08:34 vidar rethinkdb[1588]: error: Guarantee failed: [pair.second] key for entry_t already exists
Jul 16 01:08:34 vidar rethinkdb[1588]: error: Backtrace:
Jul 16 01:08:34 vidar rethinkdb[1588]: error: Sun Jul 16 01:08:34 2017
Jul 16 01:08:34 vidar rethinkdb[1588]:
Jul 16 01:08:34 vidar rethinkdb[1588]: 1 [0x7abf16]: backtrace_t::backtrace_t() at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: 2 [0x7ac3ca]: format_backtrace[abi:cxx11](bool) at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: 3 [0xb01047]: report_fatal_error(char const*, int, char const*, ...) at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: 4 [0xd37d0a]: watchable_map_var_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::entry_t::entry_t(watchable_map_var_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>*, std::pair<server_id_t, uuid_u> const&, contract_execution_bcard_t const&) at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: 5 [0xda414d]: minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::on_update(signal_t*, peer_id_t const&, uuid_u const&, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> > const&, boost::optional<contract_execution_bcard_t> const&) at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: 6 [0xd8140c]: std::_Function_handler<void (signal_t*, peer_id_t, uuid_u, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> >, boost::optional<contract_execution_bcard_t>), std::_Bind<void (minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::*(minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>))(signal_t*, peer_id_t const&, uuid_u const&, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> > const&, boost::optional<contract_execution_bcard_t> const&)> >::_M_invoke(std::_Any_data const&, signal_t*&&, peer_id_t&&, uuid_u&&, fifo_enforcer_write_token_t&&, bool&&, boost::optional<std::pair<server_id_t, uuid_u> >&&, boost::optional<contract_execution_bcard_t>&&) at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: 7 [0xd9e579]: mailbox_t<void (peer_id_t, uuid_u, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> >, boost::optional<contract_execution_bcard_t>)>::read_impl_t::read(read_stream_t*, signal_t*) at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: 8 [0x797ffb]: /usr/bin/rethinkdb() [0x797ffb] at 0x797ffb ()
Jul 16 01:08:34 vidar rethinkdb[1588]: 9 [0x7ea1ac]: coro_t::run() at ??:?
Jul 16 01:08:34 vidar rethinkdb[1588]: error: Exiting.
Jul 16 01:08:39 vidar systemd[1]: rethinkdb@freifunk.service: Main process exited, code=killed, status=5/TRAP
Jul 16 01:08:39 vidar systemd[1]: rethinkdb@freifunk.service: Unit entered failed state.
Jul 16 01:08:39 vidar systemd[1]: rethinkdb@freifunk.service: Failed with result 'signal'.
Jul 16 07:46:44 vidar systemd[1]: Started RethinkDB database server for instance 'freifunk'.
Jul 16 07:46:44 vidar rethinkdb[14984]: Warning: The following options are not used by this invocation of rethinkdb and will be ignored: server-name
Jul 16 07:46:44 vidar rethinkdb[14984]: error: The pid-file specified already exists. This might mean that an instance is already running.
Jul 16 07:46:44 vidar systemd[1]: rethinkdb@freifunk.service: Main process exited, code=exited, status=1/FAILURE
Jul 16 07:46:44 vidar systemd[1]: rethinkdb@freifunk.service: Unit entered failed state.
Jul 16 07:46:44 vidar systemd[1]: rethinkdb@freifunk.service: Failed with result 'exit-code'.
Jul 16 07:48:14 vidar systemd[1]: Started RethinkDB database server for instance 'freifunk'.
Jul 16 07:48:14 vidar rethinkdb[17220]: Warning: The following options are not used by this invocation of rethinkdb and will be ignored: server-name
Jul 16 07:48:15 vidar rethinkdb[17220]: error: The pid-file specified already exists. This might mean that an instance is already running.
Jul 16 07:48:15 vidar systemd[1]: rethinkdb@freifunk.service: Main process exited, code=exited, status=1/FAILURE
Jul 16 07:48:15 vidar systemd[1]: rethinkdb@freifunk.service: Unit entered failed state.
Jul 16 07:48:15 vidar systemd[1]: rethinkdb@freifunk.service: Failed with result 'exit-code'.
Jul 16 07:49:36 vidar systemd[1]: Started RethinkDB database server for instance 'freifunk'.
Jul 16 07:49:36 vidar rethinkdb[19464]: Warning: The following options are not used by this invocation of rethinkdb and will be ignored: server-name
Jul 16 07:49:36 vidar rethinkdb[19464]: Running rethinkdb 2.3.5 (GCC 7.1.1)...
Jul 16 07:49:36 vidar rethinkdb[19464]: Running on Linux 4.11.9-1-ARCH x86_64
Jul 16 07:49:36 vidar rethinkdb[19464]: Loading data from directory /var/lib/rethinkdb/default
Jul 16 07:49:36 vidar rethinkdb[19464]: Listening for intracluster connections on port 29015
Jul 16 07:49:36 vidar rethinkdb[19464]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 16 07:49:37 vidar rethinkdb[19464]: Connected to server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 16 07:49:37 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 07:49:41 vidar rethinkdb[19464]: Listening for client driver connections on port 28015
Jul 16 07:49:41 vidar rethinkdb[19464]: Administrative HTTP connections are disabled.
Jul 16 07:49:41 vidar rethinkdb[19464]: Listening on cluster addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 16 07:49:41 vidar rethinkdb[19464]: Listening on driver addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 16 07:49:41 vidar rethinkdb[19464]: Listening on http addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 16 07:49:41 vidar rethinkdb[19464]: To fully expose RethinkDB on the network, bind to all addresses by adding `bind=all' to the config file (/etc/rethinkdb/instances.d/freifunk.conf).
Jul 16 07:49:41 vidar rethinkdb[19464]: Server ready, "vidar" 13a61804-84be-4089-886e-19d490858ef2
Jul 16 07:49:41 vidar rethinkdb[19464]: warn: We were unable to connect to the following peer, or the --join address does not match the peer's canonical address: 10.206.0.7:29015
Jul 16 20:01:56 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 16 20:01:56 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 16 20:02:26 vidar rethinkdb[19464]: warn: Received invalid clustering header from 10.206.0.3:29015, closing connection -- something might be connecting to the wrong port.
Jul 17 01:59:17 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 02:00:31 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:17851
Jul 17 02:00:31 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 02:01:22 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 02:31:16 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:17861
Jul 17 02:31:16 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 02:31:29 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:30:19 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 17 03:30:19 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:30:56 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:31:55 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:32:05 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:32:45 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:32:45 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:45:43 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 17 03:45:43 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 03:45:53 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 04:15:49 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 17 04:15:49 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 04:16:01 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 04:30:43 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:21339
Jul 17 04:30:44 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 04:30:49 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 05:00:18 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:21349
Jul 17 05:00:18 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 05:01:14 vidar rethinkdb[19464]: warn: Received invalid clustering header from 10.206.0.3:29015, closing connection -- something might be connecting to the wrong port.
Jul 17 05:01:15 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 05:30:45 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:21949
Jul 17 05:30:45 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 05:31:58 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 06:45:22 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 17 06:45:22 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 06:45:44 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 06:46:46 vidar rethinkdb[19464]: Disconnected from server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 17 06:47:00 vidar rethinkdb[19464]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 17 07:00:38 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29073
Jul 17 07:00:38 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 07:00:49 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 07:01:53 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:30063
Jul 17 07:01:53 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 07:02:13 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 08:48:03 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:30075
Jul 17 08:48:03 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 08:48:06 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 10:00:53 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 10:00:54 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 10:30:22 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:35771
Jul 17 10:30:22 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 10:30:44 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 11:01:19 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 11:01:20 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 11:15:26 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 17 11:15:27 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 11:15:45 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 12:00:44 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 12:00:52 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 18:30:35 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 18:30:35 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 18:43:11 vidar rethinkdb[19464]: Removing file /var/lib/rethinkdb/default/87ecacd9-eda7-4059-9efe-57d36a31fe29
Jul 17 18:43:12 vidar rethinkdb[19464]: Removing file /var/lib/rethinkdb/default/32ff9a40-dbda-4745-813d-bcfde9253047
Jul 17 20:00:18 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 17 20:00:18 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 20:00:46 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 17 20:38:11 vidar rethinkdb[19464]: Removing file /var/lib/rethinkdb/default/9a60cec7-5828-4ec7-b09f-324340da641c
Jul 17 20:39:07 vidar rethinkdb[19464]: Removing file /var/lib/rethinkdb/default/e19d049d-8d90-48e0-a43b-653ad2924055
Jul 17 20:41:36 vidar rethinkdb[19464]: Removing file /var/lib/rethinkdb/default/00a759b3-8cd7-45e7-babc-ffeab55e0424
Jul 17 20:44:20 vidar rethinkdb[19464]: Removing file /var/lib/rethinkdb/default/35fab1a8-9238-4c9f-a2c3-5f867cf027ea
Jul 18 00:01:49 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 00:01:50 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 01:00:26 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 18 01:00:26 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 01:01:09 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 01:06:15 vidar rethinkdb[19464]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:46323
Jul 18 01:06:15 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 01:06:25 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 01:11:21 vidar rethinkdb[19464]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 01:11:53 vidar rethinkdb[19464]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 18 01:12:25 vidar rethinkdb[19464]: Version: rethinkdb 2.3.5 (GCC 7.1.1)
Jul 18 01:12:25 vidar rethinkdb[19464]: error: Error in ./src/concurrency/watchable_map.tcc at line 98:
Jul 18 01:12:25 vidar rethinkdb[19464]: error: Guarantee failed: [pair.second] key for entry_t already exists
Jul 18 01:12:25 vidar rethinkdb[19464]: error: Backtrace:
Jul 18 01:12:25 vidar rethinkdb[19464]: error: Tue Jul 18 01:12:25 2017
Jul 18 01:12:25 vidar rethinkdb[19464]:
Jul 18 01:12:25 vidar rethinkdb[19464]: 1 [0x7abf16]: backtrace_t::backtrace_t() at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: 2 [0x7ac3ca]: format_backtrace[abi:cxx11](bool) at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: 3 [0xb01047]: report_fatal_error(char const*, int, char const*, ...) at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: 4 [0xd37d0a]: watchable_map_var_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::entry_t::entry_t(watchable_map_var_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>*, std::pair<server_id_t, uuid_u> const&, contract_execution_bcard_t const&) at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: 5 [0xda414d]: minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::on_update(signal_t*, peer_id_t const&, uuid_u const&, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> > const&, boost::optional<contract_execution_bcard_t> const&) at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: 6 [0xd8140c]: std::_Function_handler<void (signal_t*, peer_id_t, uuid_u, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> >, boost::optional<contract_execution_bcard_t>), std::_Bind<void (minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::*(minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>))(signal_t*, peer_id_t const&, uuid_u const&, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> > const&, boost::optional<contract_execution_bcard_t> const&)> >::_M_invoke(std::_Any_data const&, signal_t*&&, peer_id_t&&, uuid_u&&, fifo_enforcer_write_token_t&&, bool&&, boost::optional<std::pair<server_id_t, uuid_u> >&&, boost::optional<contract_execution_bcard_t>&&) at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: 7 [0xd9e579]: mailbox_t<void (peer_id_t, uuid_u, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> >, boost::optional<contract_execution_bcard_t>)>::read_impl_t::read(read_stream_t*, signal_t*) at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: 8 [0x797ffb]: /usr/bin/rethinkdb() [0x797ffb] at 0x797ffb ()
Jul 18 01:12:25 vidar rethinkdb[19464]: 9 [0x7ea1ac]: coro_t::run() at ??:?
Jul 18 01:12:25 vidar rethinkdb[19464]: error: Exiting.
Jul 18 01:12:42 vidar systemd[1]: rethinkdb@freifunk.service: Main process exited, code=killed, status=5/TRAP
Jul 18 01:12:42 vidar systemd[1]: rethinkdb@freifunk.service: Unit entered failed state.
Jul 18 01:12:42 vidar systemd[1]: rethinkdb@freifunk.service: Failed with result 'signal'.
-- Reboot --
Jul 19 01:14:08 vidar systemd[1]: Started RethinkDB database server for instance 'freifunk'.
Jul 19 01:14:10 vidar rethinkdb[1631]: Warning: The following options are not used by this invocation of rethinkdb and will be ignored: server-name
Jul 19 01:14:10 vidar rethinkdb[1631]: Running rethinkdb 2.3.6 (GCC 7.1.1)...
Jul 19 01:14:10 vidar rethinkdb[1631]: Running on Linux 4.11.9-1-ARCH x86_64
Jul 19 01:14:10 vidar rethinkdb[1631]: Loading data from directory /var/lib/rethinkdb/default
Jul 19 01:14:10 vidar rethinkdb[1631]: Listening for intracluster connections on port 29015
Jul 19 01:14:46 vidar rethinkdb[1631]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 19 01:14:59 vidar rethinkdb[1631]: warn: We were unable to connect to the following peers, or the --join address does not match the peer's canonical address: 10.206.0.2:29015, 10.206.0.3:29015, 10.206.0.7:29015
Jul 19 01:15:18 vidar rethinkdb[1631]: Listening for client driver connections on port 28015
Jul 19 01:15:18 vidar rethinkdb[1631]: Administrative HTTP connections are disabled.
Jul 19 01:15:18 vidar rethinkdb[1631]: Listening on cluster addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 19 01:15:18 vidar rethinkdb[1631]: Listening on driver addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 19 01:15:18 vidar rethinkdb[1631]: Listening on http addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 19 01:15:18 vidar rethinkdb[1631]: To fully expose RethinkDB on the network, bind to all addresses by adding `bind=all' to the config file (/etc/rethinkdb/instances.d/freifunk.conf).
Jul 19 01:15:18 vidar rethinkdb[1631]: Server ready, "vidar" 13a61804-84be-4089-886e-19d490858ef2
Jul 19 01:18:25 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:25:54 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:28:28 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:30:20 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:47387
Jul 19 01:30:20 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:32:03 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:32:34 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:39:40 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:41:45 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:47429
Jul 19 01:41:45 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:42:04 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:46:42 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:47881
Jul 19 01:46:42 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:46:52 vidar rethinkdb[1631]: warn: Received invalid clustering header from 10.206.0.3:29015, closing connection -- something might be connecting to the wrong port.
Jul 19 01:47:10 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:53:35 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 01:56:27 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 02:00:25 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:47935
Jul 19 02:00:25 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 02:00:47 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 18:45:17 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:48139
Jul 19 18:45:17 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 19 18:45:30 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 20 00:00:26 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 20 00:00:26 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 20 00:00:36 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 20 02:00:16 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 20 02:00:17 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 20 02:00:34 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 00:00:25 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:48837
Jul 21 00:00:26 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 00:00:33 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 09:10:47 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 21 09:10:48 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 09:14:37 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.5:29015
Jul 21 09:14:37 vidar rethinkdb[1631]: Disconnected from server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 21 09:15:11 vidar rethinkdb[1631]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:25581
Jul 21 09:19:55 vidar rethinkdb[1631]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 21 09:23:18 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 09:45:58 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 09:46:10 vidar rethinkdb[1631]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 09:49:47 vidar rethinkdb[1631]: Connected to server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 21 10:00:38 vidar systemd[1]: Stopping RethinkDB database server for instance 'freifunk'...
Jul 21 10:00:38 vidar rethinkdb[1631]: Server got SIGTERM from pid 1, uid 0; shutting down...
Jul 21 10:00:38 vidar rethinkdb[1631]: Shutting down client connections...
Jul 21 10:00:38 vidar rethinkdb[1631]: All client connections closed.
Jul 21 10:00:38 vidar rethinkdb[1631]: Shutting down storage engine... (This may take a while if you had a lot of unflushed data in the writeback cache.)
Jul 21 10:00:40 vidar rethinkdb[1631]: Storage engine shut down.
Jul 21 10:00:40 vidar rethinkdb[1631]: Disconnected from server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 21 10:00:40 vidar rethinkdb[1631]: Disconnected from server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 21 10:00:40 vidar rethinkdb[1631]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 10:00:40 vidar systemd[1]: Stopped RethinkDB database server for instance 'freifunk'.
Jul 21 10:00:50 vidar systemd[1]: Started RethinkDB database server for instance 'freifunk'.
Jul 21 10:00:50 vidar rethinkdb[17432]: Warning: The following options are not used by this invocation of rethinkdb and will be ignored: server-name
Jul 21 10:00:51 vidar rethinkdb[17432]: Running rethinkdb 2.3.6 (GCC 7.1.1)...
Jul 21 10:00:51 vidar rethinkdb[17432]: Running on Linux 4.11.9-1-ARCH x86_64
Jul 21 10:00:51 vidar rethinkdb[17432]: Loading data from directory /var/lib/rethinkdb/default
Jul 21 10:00:51 vidar rethinkdb[17432]: Listening for intracluster connections on port 29015
Jul 21 10:00:51 vidar rethinkdb[17432]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 21 10:00:51 vidar rethinkdb[17432]: Connected to server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 21 10:00:51 vidar rethinkdb[17432]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 10:00:56 vidar rethinkdb[17432]: warn: We were unable to connect to the following peer, or the --join address does not match the peer's canonical address: 10.206.0.7:29015
Jul 21 10:01:01 vidar rethinkdb[17432]: Listening for client driver connections on port 28015
Jul 21 10:01:01 vidar rethinkdb[17432]: Administrative HTTP connections are disabled.
Jul 21 10:01:01 vidar rethinkdb[17432]: Listening on cluster addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 21 10:01:01 vidar rethinkdb[17432]: Listening on driver addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 21 10:01:01 vidar rethinkdb[17432]: Listening on http addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 21 10:01:01 vidar rethinkdb[17432]: To fully expose RethinkDB on the network, bind to all addresses by adding `bind=all' to the config file (/etc/rethinkdb/instances.d/freifunk.conf).
Jul 21 10:01:01 vidar rethinkdb[17432]: Server ready, "vidar" 13a61804-84be-4089-886e-19d490858ef2
Jul 21 12:38:18 vidar systemd[1]: Stopping RethinkDB database server for instance 'freifunk'...
Jul 21 12:38:18 vidar rethinkdb[17432]: Server got SIGTERM from pid 1, uid 0; shutting down...
Jul 21 12:38:18 vidar rethinkdb[17432]: Shutting down client connections...
Jul 21 12:38:18 vidar rethinkdb[17432]: All client connections closed.
Jul 21 12:38:18 vidar rethinkdb[17432]: Shutting down storage engine... (This may take a while if you had a lot of unflushed data in the writeback cache.)
Jul 21 12:38:20 vidar rethinkdb[17432]: Storage engine shut down.
Jul 21 12:38:20 vidar rethinkdb[17432]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:38:20 vidar rethinkdb[17432]: Disconnected from server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 21 12:38:20 vidar rethinkdb[17432]: Disconnected from server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 21 12:38:20 vidar systemd[1]: Stopped RethinkDB database server for instance 'freifunk'.
-- Reboot --
Jul 21 12:38:35 vidar systemd[1]: Started RethinkDB database server for instance 'freifunk'.
Jul 21 12:38:37 vidar rethinkdb[1581]: Warning: The following options are not used by this invocation of rethinkdb and will be ignored: server-name
Jul 21 12:38:37 vidar rethinkdb[1581]: Running rethinkdb 2.3.6 (GCC 7.1.1)...
Jul 21 12:38:37 vidar rethinkdb[1581]: Running on Linux 4.11.9-1-ARCH x86_64
Jul 21 12:38:37 vidar rethinkdb[1581]: Loading data from directory /var/lib/rethinkdb/default
Jul 21 12:38:38 vidar rethinkdb[1581]: Listening for intracluster connections on port 29015
Jul 21 12:39:12 vidar rethinkdb[1581]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 21 12:39:12 vidar rethinkdb[1581]: Connected to server "buri" 1906d3ca-3f19-4a96-afbf-5a89dd8e0bb7
Jul 21 12:39:13 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:39:27 vidar rethinkdb[1581]: warn: We were unable to connect to the following peer, or the --join address does not match the peer's canonical address: 10.206.0.7:29015
Jul 21 12:39:54 vidar rethinkdb[1581]: Listening for client driver connections on port 28015
Jul 21 12:39:55 vidar rethinkdb[1581]: Administrative HTTP connections are disabled.
Jul 21 12:39:55 vidar rethinkdb[1581]: Listening on cluster addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 21 12:39:55 vidar rethinkdb[1581]: Listening on driver addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 21 12:39:55 vidar rethinkdb[1581]: Listening on http addresses: 127.0.0.1, 10.206.0.6, ::1
Jul 21 12:39:55 vidar rethinkdb[1581]: To fully expose RethinkDB on the network, bind to all addresses by adding `bind=all' to the config file (/etc/rethinkdb/instances.d/freifunk.conf).
Jul 21 12:39:55 vidar rethinkdb[1581]: Server ready, "vidar" 13a61804-84be-4089-886e-19d490858ef2
Jul 21 12:42:15 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:42:16 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:43:20 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:43:22 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:44:18 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.5:29015
Jul 21 12:44:19 vidar rethinkdb[1581]: Disconnected from server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 21 12:46:46 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:46:54 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:56:40 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:29015
Jul 21 12:56:41 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 21 12:56:56 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 00:00:19 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:36169
Jul 22 00:00:19 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 00:00:56 vidar rethinkdb[1581]: warn: Received invalid clustering header from 10.206.0.3:29015, closing connection -- something might be connecting to the wrong port.
Jul 22 00:00:56 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 01:15:17 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:41739
Jul 22 01:15:17 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 01:16:00 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 01:30:38 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:42387
Jul 22 01:30:38 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 01:30:42 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 10:04:15 vidar rethinkdb[1581]: Connected to server "schenut" 02d95bec-2087-4cf4-be4b-c120fa277f59
Jul 22 14:37:09 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:42523
Jul 22 14:37:09 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 22 14:43:13 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 00:00:22 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:47711
Jul 23 00:00:22 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 00:01:21 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:03:37 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:05:09 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:06:12 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:06:39 vidar rethinkdb[1581]: warn: Received invalid clustering header from 10.206.0.3:29015, closing connection -- something might be connecting to the wrong port.
Jul 23 01:08:01 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:09:27 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:09:34 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:11:40 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:12:09 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:15:39 vidar rethinkdb[1581]: error: Heartbeat timeout, killing connection to peer 10.206.0.3:48083
Jul 23 01:15:39 vidar rethinkdb[1581]: Disconnected from server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:15:46 vidar rethinkdb[1581]: Connected to server "fulla_p3i" b9c8a626-f7de-4939-ab32-1647af53a6aa
Jul 23 01:16:42 vidar rethinkdb[1581]: Version: rethinkdb 2.3.6 (GCC 7.1.1)
Jul 23 01:16:42 vidar rethinkdb[1581]: error: Error in ./src/concurrency/watchable_map.tcc at line 98:
Jul 23 01:16:42 vidar rethinkdb[1581]: error: Guarantee failed: [pair.second] key for entry_t already exists
Jul 23 01:16:42 vidar rethinkdb[1581]: error: Backtrace:
Jul 23 01:16:42 vidar rethinkdb[1581]: error: Sun Jul 23 01:16:42 2017
Jul 23 01:16:42 vidar rethinkdb[1581]:
Jul 23 01:16:42 vidar rethinkdb[1581]: 1 [0x55f11207480a]: backtrace_t::backtrace_t() at 0x55f11207480a (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 2 [0x55f112074ccb]: format_backtrace[abi:cxx11](bool) at 0x55f112074ccb (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 3 [0x55f1123e86c3]: report_fatal_error(char const*, int, char const*, ...) at 0x55f1123e86c3 (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 4 [0x55f11262e827]: watchable_map_var_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::entry_t::entry_t(watchable_map_var_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>*, std::pair<server_id_t, uuid_u> const&, contract_execution_bcard_t const&) at 0x55f11262e827 (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 5 [0x55f11269e32a]: minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::on_update(signal_t*, peer_id_t const&, uuid_u const&, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> > const&, boost::optional<contract_execution_bcard_t> const&) at 0x55f11269e32a (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 6 [0x55f11267a28c]: std::_Function_handler<void (signal_t*, peer_id_t, uuid_u, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> >, boost::optional<contract_execution_bcard_t>), std::_Bind<void (minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>::*(minidir_read_manager_t<std::pair<server_id_t, uuid_u>, contract_execution_bcard_t>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>, std::_Placeholder<4>, std::_Placeholder<5>, std::_Placeholder<6>, std::_Placeholder<7>))(signal_t*, peer_id_t const&, uuid_u const&, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> > const&, boost::optional<contract_execution_bcard_t> const&)> >::_M_invoke(std::_Any_data const&, signal_t*&&, peer_id_t&&, uuid_u&&, fifo_enforcer_write_token_t&&, bool&&, boost::optional<std::pair<server_id_t, uuid_u> >&&, boost::optional<contract_execution_bcard_t>&&) at 0x55f11267a28c (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 7 [0x55f112698437]: mailbox_t<void (peer_id_t, uuid_u, fifo_enforcer_write_token_t, bool, boost::optional<std::pair<server_id_t, uuid_u> >, boost::optional<contract_execution_bcard_t>)>::read_impl_t::read(read_stream_t*, signal_t*) at 0x55f112698437 (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 8 [0x55f11205fa65]: /usr/bin/rethinkdb(+0x496a65) [0x55f11205fa65] at 0x55f11205fa65 (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: 9 [0x55f1120b654d]: coro_t::run() at 0x55f1120b654d (/usr/bin/rethinkdb)
Jul 23 01:16:42 vidar rethinkdb[1581]: error: Exiting.
Jul 23 01:16:58 vidar systemd[1]: rethinkdb@freifunk.service: Main process exited, code=killed, status=5/TRAP
Jul 23 01:16:58 vidar systemd[1]: rethinkdb@freifunk.service: Unit entered failed state.
Jul 23 01:16:58 vidar systemd[1]: rethinkdb@freifunk.service: Failed with result 'signal'.@timmaxw
@RubenKelevra can you tell us whether buri restarted immediately before this error occurred?
Sure, this is the uptime-graph of buri
If you need more exact timestamps, I can dig a bit into the logs. :)
robertjpayne
commented
7 years ago Is this a dupe of #4872?
One of my rethinkdb-cluster members just crashed: