Flag --prune-blobs false prunes blobs older than 4096 epochs

[nzenchik]

Description

I'm running lighthouse with --prune-blobs set to false, but blobs older than 4096 epochs are pruned. Here is args I used:

consensus-lighthouse:
    container_name: consensus-lighthouse
    image: sigp/lighthouse:v5.1.3
    volumes:
      - /data/lighthouse-consensus:/root/.lighthouse
      - /data/erigon/jwtsecret:/jwtsecret
    ports:
      - 9909:9909/tcp
      - 9909:9909/udp
      - 5052:5052/tcp
    command: >
      lighthouse bn
      --network mainnet
      --checkpoint-sync-url https://mainnet.checkpoint.sigp.io
      --execution-endpoint http://erigon:8551
      --execution-jwt /jwtsecret
      --disable-deposit-contract-sync
      --listen-address 0.0.0.0
      --port 9909
      --prune-blobs false
      --blob-prune-margin-epochs 1048576
      --http
      --http-address 0.0.0.0
      --disable-backfill-rate-limiting
      --genesis-backfill

I've also tried to add --blob-prune-margin-epochs 1048576 but this doesn't help - blobs older 4096 are also prunned

Here is curl "http://localhost:5052/lighthouse/database/info" | jq result

{
  "schema_version": 19,
  "config": {
    "slots_per_restore_point": 8192,
    "slots_per_restore_point_set_explicitly": false,
    "block_cache_size": 5,
    "historic_state_cache_size": 1,
    "compact_on_init": false,
    "compact_on_prune": true,
    "prune_payloads": true,
    "prune_blobs": false,
    "epochs_per_blob_prune": 1,
    "blob_prune_margin_epochs": 1048576
  },
  "split": {
    "slot": "9092224",
    "state_root": "0xda3a95a06ba5ed3dfa7179b52bd914cca2ffe7caeadd69f5917d3d248b9373fd",
    "block_root": "0xa4c72c0edaa362149573b34fdf57a807a24737f53e8bbd9575402a89e134101d"
  },
  "anchor": {
    "anchor_slot": "9091872",
    "oldest_block_slot": "8856256",
    "oldest_block_parent": "0xf8c591ffdf5f3c10e5349c90dc711e208ac9604def60bcc8e01199c17cf83f96",
    "state_upper_limit": "18446744073709551615",
    "state_lower_limit": "0"
  },
  "blob_info": {
    "oldest_blob_slot": "8961127",
    "blobs_db": true
  }
}

Version

sigp/lighthouse:v5.1.3

Expected Behaviour

With --prune-blobs false no blobs should be pruned

[chong-he]

How long have you run the node with --prune-blobs false ? If you check the database endpoint now, does the oldest_blob_slot remain the same or increased?

[nzenchik]

oldest_blob_slot is increasing

{
  "schema_version": 19,
  "config": {
    "slots_per_restore_point": 8192,
    "slots_per_restore_point_set_explicitly": false,
    "block_cache_size": 5,
    "historic_state_cache_size": 1,
    "compact_on_init": false,
    "compact_on_prune": true,
    "prune_payloads": true,
    "prune_blobs": true,
    "epochs_per_blob_prune": 1,
    "blob_prune_margin_epochs": 1048576
  },
  "split": {
    "slot": "9095744",
    "state_root": "0x8f882276bcf85599c77c98711e64ee12feaa20cc941c6425231831f14c5ed23a",
    "block_root": "0x640932847f988c3dd0f152fcc0ac75271cf05820d6d974baeab767508486b8a9"
  },
  "anchor": {
    "anchor_slot": "9092288",
    "oldest_block_slot": "6921536",
    "oldest_block_parent": "0xcff1fa1ef07ca256e4bb7ac3be7cd70febd02c628662ca8238dfdd7108a04873",
    "state_upper_limit": "18446744073709551615",
    "state_lower_limit": "0"
  },
  "blob_info": {
    "oldest_blob_slot": "8961506",
    "blobs_db": true
  }
}

[chong-he]

oldest_blob_slot is increasing

{
  "schema_version": 19,
  "config": {
    "slots_per_restore_point": 8192,
    "slots_per_restore_point_set_explicitly": false,
    "block_cache_size": 5,
    "historic_state_cache_size": 1,
    "compact_on_init": false,
    "compact_on_prune": true,
    "prune_payloads": true,
    "prune_blobs": true,
    "epochs_per_blob_prune": 1,
    "blob_prune_margin_epochs": 1048576
  },
  "split": {
    "slot": "9095744",
    "state_root": "0x8f882276bcf85599c77c98711e64ee12feaa20cc941c6425231831f14c5ed23a",
    "block_root": "0x640932847f988c3dd0f152fcc0ac75271cf05820d6d974baeab767508486b8a9"
  },
  "anchor": {
    "anchor_slot": "9092288",
    "oldest_block_slot": "6921536",
    "oldest_block_parent": "0xcff1fa1ef07ca256e4bb7ac3be7cd70febd02c628662ca8238dfdd7108a04873",
    "state_upper_limit": "18446744073709551615",
    "state_lower_limit": "0"
  },
  "blob_info": {
    "oldest_blob_slot": "8961506",
    "blobs_db": true
  }
}

This is showing "prune_blobs": true,, hence it will prune blobs. Your first post does show it is false, maybe there is some configuration mixed up?

[nzenchik]

Yes, I'm trying different flags on the same server to solve the issue. In this case "blob_prune_margin_epochs" is quite high for blobs to be pruned and once I removed prune_blobs I haven't done any data clean up.

[nzenchik]

I've checked and now I see that oldest_blob_slot is not increasing. Am I correct thats if I use checkpoint sync from scratch, it can backfill only blobs for 4096 latest epochs and from that moment blobs won't be pruned if I have prune_blobs: false? And the only way to backfill all blobs data is to sync from genesis?

[chong-he]

I've checked and now I see that oldest_blob_slot is not increasing. Am I correct thats if I use checkpoint sync from scratch, it can backfill only blobs for 4096 latest epochs and from that moment blobs won't be pruned if I have prune_blobs: false? And the only way to backfill all blobs data is to sync from genesis?

Yes that's correct. It will only backfill 4096 epochs from where you synced.

Older blobs data can't be backfilled yet, We have an issue open for this: https://github.com/sigp/lighthouse/issues/5391