Fix handling of sidechains and gaps

[michaelsproul]

At the moment blockprint's DB is not handling sidechains very elegantly.

The database is designed so that a block can be inserted without knowing its parent. This is nice because it provides the ability to stay online processing new blocks, even if some old blocks have been missed due to temporary downtime. There's a background process in background_tasks.py which is meant to query the API for gaps to fill in, and patch them up.

The problem is that the logic for determining gaps returns some gaps that are impossible for the background task to heal. A gap is currently defined as a slot interval between a block with a parent missing from the DB (end_slot) and the last known block prior to that missing parent (start_slot):

https://github.com/sigp/blockprint/blob/c7f570daab242a1b4ba41e17f6253a744a6270ce/build_db.py#L156-L163

https://github.com/sigp/blockprint/blob/c7f570daab242a1b4ba41e17f6253a744a6270ce/build_db.py#L181-L191

E.g. the current output from https://api.blockprint.sigp.io/sync/gaps is:

[
  {
    "start": 8253079,
    "end": 8253086
  },
  {
    "start": 8253031,
    "end": 8253045
  },
  {
    "start": 8253127,
    "end": 8253151
  },
  {
    "start": 8254502,
    "end": 8254511
  },
  {
    "start": 8277096,
    "end": 8277097
  },
  {
    "start": 8299646,
    "end": 8299647
  },
  {
    "start": 8299650,
    "end": 8299651
  }
]

Looking at the first gap, we see that the block with missing parent that triggered this must be one at slot 8253087, which is reorged out (beaconcha.in doesn't even know about it): https://beaconcha.in/slot/8253087. Our Lighthouse nodes saw it though:

Jan 21 18:17:49.412 DEBG Cloned snapshot for late block/skipped slot, block_delay: Some(2.040814714s), parent_root: 0xd429dc371766b1d71fdad731879aafe7c2df990b402fbc5704b29144009cce8f, parent_slot: 8253079, slot: 8253087, service: beacon

Now the interesting thing here is the parent slot, https://beaconcha.in/slot/8253079. It's also empty! In order to heal the gap, we would need to load this parent block at 8253079, which we can't do because it has also been pruned.

In summary, blockprint's gap healing is broken for sidechains of length > 1. I can think of two ways to fix it:

Give the background task the ability to either delete or mark orphaned blocks in the database when: the slot of the missing parent has been finalized as a skipped slot. If we just mark them as orphaned, then we get to keep them in the DB (moar data) but won't block the gap healing process on them. On the other hand, marking them orphaned would require a new database column and a little DB migration (not too bad, given the small number of live blockprint instances).

[michaelsproul]

Just tried this query as a hack to see if I could get the DB to heal:

WITH parentless AS (
  SELECT slot, proposer_index FROM blocks b1
  WHERE
    (SELECT slot FROM blocks WHERE slot = b1.parent_slot) IS NULL
    AND slot <> 1
)
DELETE FROM blocks
WHERE EXISTS (SELECT 1 FROM parentless WHERE parentless.slot = blocks.slot);

It deletes all blocks that lack parents, which could fix the DB if there are just a few length-2 sidechains. We'll see.

[michaelsproul]

Seems to have worked.