search

streamingfast / firehose-ethereum

Ethereum on StreamingFast
Apache License 2.0
80 stars 24 forks source link

Bad block breaks geth #44

Closed paymog closed 2 years ago

paymog commented 2 years ago

I've started running firehose and I just ran into the following log lines which seems to have broken geth - it's now reporting that a Disk Quota has been been exceeded:

firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:24.478946191-04:00","logger":"reader.geth","message":"beacon chain gapped                      head=15,869,849 newHead=15,869,855","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:24.478990291-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:24.477] Failed to clean stale beacon headers     err=\"filled header below beacon header tail: 6158489 < 15869849\"","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:24.693057822-04:00","logger":"reader.geth","message":"unindexed transactions                   blocks=439 txs=55767 tail=6,158,362 elapsed=227.838ms","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:24.880649279-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:25.332971157-04:00","logger":"reader.geth","message":"syncing beacon headers                   downloaded=1540 left=6,158,488 eta=550h18m24.979s","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:25.382861547-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:25.518788928-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:25.518] Expired request does not exist           peer=e053914017a506772cce70913eac44cc9b0e4378adc138f6f11b7c943fe5acbf","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:25.884820224-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:25.955989251-04:00","logger":"reader.geth","message":"downloader queue stats                   receiptTasks=0 blockTasks=34168 itemSize=27.24KiB throttle=8192","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.036932293-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.036] Unhandled trie error: missing trie node 4cdf3958a41d42a048cb37085d0e2450f3a92074a99d446ab4dd7b48cfd190ab (path 03070b030e0c04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.212740567-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.212] Unhandled trie error: missing trie node 164b4f0699da01839785857cce9e614559afc9a569c10b3e82254db1abb7d16b (path 09090b0f0d04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.259208341-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.259] Unhandled trie error: missing trie node 164b4f0699da01839785857cce9e614559afc9a569c10b3e82254db1abb7d16b (path 09090b0f0d04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.268408351-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.268] Unhandled trie error: missing trie node 7217e48235c2c5bed47a4bc25418e952db40c64433424ea1f99b44669000b375 (path 09090e00010900) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.28229091-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.282] Unhandled trie error: missing trie node 7217e48235c2c5bed47a4bc25418e952db40c64433424ea1f99b44669000b375 (path 09090e00010900) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.287387822-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.283] Unhandled trie error: missing trie node ca4cbcd5f6e09f53c448cc68a9b452ce669a25a9eed0f22a79a4e07d4a07bae3 (path 07010e090b0f08) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.287431338-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.284] Unhandled trie error: missing trie node ca4cbcd5f6e09f53c448cc68a9b452ce669a25a9eed0f22a79a4e07d4a07bae3 (path 07010e090b0f08) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30427221-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.304] Unhandled trie error: missing trie node b60717f155b92939dc3387492394afd09912678a26923af5d2c4e5e315c913f2 (path 04070100050e03) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30760353-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.306] Unhandled trie error: missing trie node b60717f155b92939dc3387492394afd09912678a26923af5d2c4e5e315c913f2 (path 04070100050e03) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30780068-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.307] Unhandled trie error: missing trie node 0936e560feba21cfd5ca6686193ede797b4c4a1818436703888696343c38b086 (path 0c09040909020b) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30991422-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.309] Unhandled trie error: missing trie node 0936e560feba21cfd5ca6686193ede797b4c4a1818436703888696343c38b086 (path 0c09040909020b) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.311182544-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.311] Unhandled trie error: missing trie node da95e202db5dadb272aba88d218c26038af5e437514fa80e3f116f60d50cc4f5 (path 0c0a00030108) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.312121839-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.312] Unhandled trie error: missing trie node da95e202db5dadb272aba88d218c26038af5e437514fa80e3f116f60d50cc4f5 (path 0c0a00030108) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:26.393611317-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.40731312-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.407] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.408046113-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.407] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.408799923-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.408] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose FAILED trx "a9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927" at block 6158490 (hash unavailable, probably forked): insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515 2472
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410209421-04:00","logger":"reader.geth","message":"skip duplicated bad block                number=6,158,490  hash=b6e729..3720d4","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.410320327-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.410] ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410359297-04:00","logger":"reader.geth","message":"########## BAD BLOCK #########","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410375414-04:00","logger":"reader.geth","message":"chain config: Chain ID:  1 (mainnet)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410387081-04:00","logger":"reader.geth","message":"consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410399144-04:00","logger":"reader.geth","message":"pre-Merge hard forks:","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41041168-04:00","logger":"reader.geth","message":" - Homestead:                   1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410424045-04:00","logger":"reader.geth","message":" - DAO Fork:                    1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410451966-04:00","logger":"reader.geth","message":" - Tangerine Whistle (EIP 150): 2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410467232-04:00","logger":"reader.geth","message":" - Spurious Dragon/1 (EIP 155): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41047902-04:00","logger":"reader.geth","message":" - Spurious Dragon/2 (EIP 158): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410490538-04:00","logger":"reader.geth","message":" - Byzantium:                   4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410502395-04:00","logger":"reader.geth","message":" - Constantinople:              7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410515128-04:00","logger":"reader.geth","message":" - Petersburg:                  7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410528678-04:00","logger":"reader.geth","message":" - Istanbul:                    9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410545419-04:00","logger":"reader.geth","message":" - Muir Glacier:                9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41057709-04:00","logger":"reader.geth","message":" - Berlin:                      12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410593931-04:00","logger":"reader.geth","message":" - London:                      12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410607918-04:00","logger":"reader.geth","message":" - Arrow Glacier:               13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.4106357-04:00","logger":"reader.geth","message":" - Gray Glacier:                15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41065043-04:00","logger":"reader.geth","message":"merge configured:","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410661111-04:00","logger":"reader.geth","message":" - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41067195-04:00","logger":"reader.geth","message":" - Network known to be merged: true","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410682275-04:00","logger":"reader.geth","message":" - Total terminal difficulty:  58750000000000000000000","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410693105-04:00","logger":"reader.geth","message":" - Merge netsplit block:       <nil>","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410704011-04:00","logger":"reader.geth","message":"number: 6158490","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41072913-04:00","logger":"reader.geth","message":"hash: 0xb6e729d033a32968a21450a3408c8b5ab4566e0c5e1bd112017519d21b3720d4","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410747068-04:00","logger":"reader.geth","message":"error: could not apply tx 113 [0xa9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927]: insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410762094-04:00","logger":"reader.geth","message":"##############################","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410772044-04:00","logger":"reader.geth","message":" ","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.410808153-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.410] Beacon backfilling failed                err=\"retrieved hash chain is invalid: could not apply tx 113 [0xa9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927]: insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515\"","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.874415245-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"merged-blocks/0006158300.dbin.zst","base_filename":"0006158300","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:26.896155225-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:27.399447259-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:27.901626774-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}

Is this a known issue? Could it be that my geth instance isn't syncing properly? Might the bad block be a red herring for the disk quota error?

maoueh commented 2 years ago

Not sure why Geth is not happy, everything flagged "logger":"reader.geth" is coming from Geth directly. Bad block indicates something really bad. The insufficient space might have cause this indeed.

Fix the space issue, rewind head earlier and try again. Not really a firehose-ethereum issue directly.

paymog commented 2 years ago

weird, restarting firehose seems to have solved the issue, at least temporarily. Will keep an eye on this and see if it comes back.

I've tried geth attaching to the new instance, and it seems I'm now at block 6158574:

> root@firehose-mainnet-7457c6d7f7-t6m26:/# geth attach /data/firehose/reader/ipc
INFO [10-31|15:43:18.537] Initializing firehose
INFO [10-31|15:43:18.556] Firehose initialized                     enabled=false sync_instrumentation_enabled=true mining_enabled=false block_progress_enabled=false compaction_disabled=false archive_blocks_to_keep=0 genesis_provenance="Geth Default" firehose_version=2.0 geth_version=1.10.25-fh2 chain_variant=geth
Welcome to the Geth JavaScript console!

instance: Geth/v1.10.25-fh2-67d4fa90/linux-amd64/go1.18.5
at block: 6158571 (Thu Aug 16 2018 11:46:38 GMT-0400 (EDT))
 datadir: /data/firehose/reader/data
 modules: admin:1.0 debug:1.0 engine:1.0 eth:1.0 ethash:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

To exit, press ctrl-d or type exit
> eth.syncing
{
  currentBlock: 6158574,
  healedBytecodeBytes: 0,
  healedBytecodes: 0,
  healedTrienodeBytes: 0,
  healedTrienodes: 0,
  healingBytecode: 0,
  healingTrienodes: 0,
  highestBlock: 15870156,
  startingBlock: 6158488,
  syncedAccountBytes: 0,
  syncedAccounts: 0,
  syncedBytecodeBytes: 0,
  syncedBytecodes: 0,
  syncedStorage: 0,
  syncedStorageBytes: 0
}
paymog commented 2 years ago

yeah, looks like the disk space was an issue and the bad block was a symptom of that. I had causality reversed.