search

Cryptonomic / Conseil

Query API and indexer for Tezos and other decentralized platforms.
Apache License 2.0
89 stars 22 forks source link

ConsistentForkDetector identify block then violate foreign key constraint "bake_rights_block_fkey" #955

Open jun0tpyrc opened 3 years ago

jun0tpyrc commented 3 years ago 
05:16:44.384 [-akka.actor.default-dispatcher-2]  WARN  t.c.c.i.t.f.ConsistentForkDetector   - Looking for the block where the local indexer diverged from the chain node due to a fork. I will make 5 attempt(s).
05:16:44.411 [-akka.actor.default-dispatcher-2]  WARN  t.c.c.i.t.f.ConsistentForkDetector   - Looking for the block where the local indexer diverged from the chain node due to a fork. I will make 5 attempt(s).
05:16:44.438 [akka.actor.default-dispatcher-15]  WARN  t.c.c.i.t.f.ConsistentForkDetector   - Looking for the block where the local indexer diverged from the chain node due to a fork. I will make 5 attempt(s).
05:16:44.481 [akka.actor.default-dispatcher-14]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.519 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.553 [-akka.actor.default-dispatcher-4]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.590 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.618 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.652 [-akka.actor.default-dispatcher-3]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.684 [-akka.actor.default-dispatcher-3]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.719 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.748 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.778 [-akka.actor.default-dispatcher-3]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.810 [-akka.actor.default-dispatcher-3]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.855 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.881 [-akka.actor.default-dispatcher-3]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.906 [-akka.actor.default-dispatcher-3]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.932 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.958 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:44.986 [-akka.actor.default-dispatcher-3]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:45.012 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:45.035 [akka.actor.default-dispatcher-15]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 308550
05:16:49.173 [main]                              INFO  t.cryptonomic.conseil.indexer.Lorre$ - Doing clean-up.
05:16:49.176 [main]                              INFO  com.zaxxer.hikari.HikariDataSource   - lorre.db - Shutdown initiated...
05:16:49.229 [main]                              INFO  com.zaxxer.hikari.HikariDataSource   - lorre.db - Shutdown completed.
05:16:49.357 [main]                              INFO  t.cryptonomic.conseil.indexer.Lorre$ - All things closed.
Exception in thread "main" org.postgresql.util.PSQLException: ERROR: insert or update on table "baking_rights" violates foreign key constraint "bake_rights_block_fkey"
  Detail: Key (block_hash, fork_id)=(BLvpVpRV44nkry7nFV5CWtqp7BmqHnhT1QYdctjbGAPgYop5XiK, 406a7d97-8630-47cf-bc6d-d075146c25eb) is not present in table "blocks".
jun0tpyrc commented 3 years ago

it looks that conseil might do some partial insert which makes things fail to run again if a batch fails to finish

the workaround i use for making things continue

SET search_path TO "$user",public,tezos; # include the chain you use
ALTER TABLE baking_rights DROP CONSTRAINT bake_rights_block_fkey;

(In our case it occurred on Delphinet & Mainnet for tezos too)

vishakh commented 3 years ago

This is due to an issue with our fork detection logic. We are actively working on a fix. In the meanwhile, you can use the hotfix/feature-flag-fork-detection branch to work around this issue.

Sorry for the inconvenience.

ghost commented 3 years ago

hi , I seem to have a similar issue but without the SQL error. I only have this in the log:

...
11:48:53.211 [-akka.actor.default-dispatcher-2]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 1305050
11:48:53.233 [-akka.actor.default-dispatcher-2]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 1305050
11:48:53.253 [-akka.actor.default-dispatcher-5]  INFO  t.c.c.i.t.f.ConsistentForkDetector   - The fork block level was identified at 1305050
...