near_indexer::streamer::fetchers: Block either has never been observed on the node or has been garbage collected

[kulame]

i got a error

near_indexer::streamer::fetchers: Block either has never been observed on the node or has been garbage collected: BLOCK: 3bnkrmNGp1VpgDDvPJ6dN7udiMBMjF6ZXqJDYKYcYoEf

and deadloop on fetch_latest_block

how can i fix this issue?

[khorolets]

Hey there!

Could you provide a bit more details so we can help you.

1. Which network you are running the indexer for?
2. What command do you use to run the indexer?
3. Could you provide some logs a bit before the error and full error to got the context?

Thank you!

[kulame]

1 testnet 2 i write a indexer to do 3 my config.json

{
  "genesis_file": "genesis.json",
  "genesis_records_file": null,
  "validator_key_file": "validator_key.json",
  "node_key_file": "node_key.json",
  "rpc": {
    "addr": "0.0.0.0:3030",
    "prometheus_addr": null,
    "cors_allowed_origins": [
      "*"
    ],
    "polling_config": {
      "polling_interval": {
        "secs": 0,
        "nanos": 500000000
      },
      "polling_timeout": {
        "secs": 10,
        "nanos": 0
      }
    },
    "limits_config": {
      "json_payload_max_size": 10485760
    }
  },
  "telemetry": {
    "endpoints": [
      "https://explorer.testnet.near.org/api/nodes",
      "https://explorer.testnet.near.org/api/nodes"
    ]
  },
  "network": {
    "addr": "0.0.0.0:24567",
    "external_address": "",
    "boot_nodes": "ed25519:4k9csx6zMiXy4waUvRMPTkEtAS2RFKLVScocR5HwN53P@34.73.25.182:24567,ed25519:4keFArc3M4SE1debUQWi3F1jiuFZSWThgVuA2Ja2p3Jv@34.94.158.10:24567,ed25519:D2t1KTLJuwKDhbcD9tMXcXaydMNykA99Cedz7SkJkdj2@35.234.138.23:24567,ed25519:CAzhtaUPrxCuwJoFzceebiThD9wBofzqqEMCiupZ4M3E@34.94.177.51:24567",
    "max_num_peers": 40,
    "minimum_outbound_peers": 5,
    "ideal_connections_lo": 30,
    "ideal_connections_hi": 35,
    "peer_recent_time_window": {
      "secs": 600,
      "nanos": 0
    },
    "safe_set_size": 20,
    "archival_peer_connections_lower_bound": 10,
    "handshake_timeout": {
      "secs": 20,
      "nanos": 0
    },
    "reconnect_delay": {
      "secs": 60,
      "nanos": 0
    },
    "skip_sync_wait": false,
    "ban_window": {
      "secs": 10800,
      "nanos": 0
    },
    "blacklist": [],
    "ttl_account_id_router": {
      "secs": 3600,
      "nanos": 0
    },
    "peer_stats_period": {
      "secs": 5,
      "nanos": 0
    }
  },
  "consensus": {
    "min_num_peers": 3,
    "block_production_tracking_delay": {
      "secs": 0,
      "nanos": 100000000
    },
    "min_block_production_delay": {
      "secs": 0,
      "nanos": 600000000
    },
    "max_block_production_delay": {
      "secs": 2,
      "nanos": 0
    },
    "max_block_wait_delay": {
      "secs": 6,
      "nanos": 0
    },
    "reduce_wait_for_missing_block": {
      "secs": 0,
      "nanos": 100000000
    },
    "produce_empty_blocks": true,
    "block_fetch_horizon": 50,
    "state_fetch_horizon": 5,
    "block_header_fetch_horizon": 50,
    "catchup_step_period": {
      "secs": 0,
      "nanos": 100000000
    },
    "chunk_request_retry_period": {
      "secs": 0,
      "nanos": 200000000
    },
    "header_sync_initial_timeout": {
      "secs": 10,
      "nanos": 0
    },
    "header_sync_progress_timeout": {
      "secs": 2,
      "nanos": 0
    },
    "header_sync_stall_ban_timeout": {
      "secs": 120,
      "nanos": 0
    },
    "state_sync_timeout": {
      "secs": 60,
      "nanos": 0
    },
    "header_sync_expected_height_per_second": 10,
    "sync_check_period": {
      "secs": 10,
      "nanos": 0
    },
    "sync_step_period": {
      "secs": 0,
      "nanos": 10000000
    },
    "doomslug_step_period": {
      "secs": 0,
      "nanos": 100000000
    }
  },
  "tracked_accounts": [],
  "tracked_shards": [
    0
  ],
  "archive": false,
  "log_summary_style": "colored",
  "gc_blocks_limit": 2,
  "view_client_threads": 4,
  "epoch_sync_enabled": true,
  "view_client_throttle_period": {
    "secs": 30,
    "nanos": 0
  },
  "trie_viewer_state_size_limit": 50000
}

[khorolets]

@kulame Please provide logs with error, so I can have a look and think what went wrong

[frol]

@kulame your node is not an archival node ("archive": false), and it seems you might try to query the data that is older than ~2 days (5 NEAR epochs). Either run archival node (and download the archival backup), or index the recent blocks.

[khorolets]

@kulame feel free to reopen the issue if @frol's advice doesn't help. Meanwhile, I'm closing the issue.