Open osizadmin opened 3 years ago
I am seeing the same error after a SIGSEGV crash on the previous run. The only difference is that I'm doing a full sync.
Here is the log of the crash:
INFO [03-01|16:33:09.714] Starting Geth on Ethereum mainnet...
INFO [03-01|16:33:09.717] Maximum peer count ETH=50 LES=0 total=50
INFO [03-01|16:33:09.718] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [03-01|16:33:09.722] Set global gas cap cap=25000000
INFO [03-01|16:33:09.723] Allocated trie memory caches clean=1023.00MiB dirty=1024.00MiB
INFO [03-01|16:33:09.723] Allocated cache and file handles database=/home/user/.ethereum/geth/chaindata cache=2.00GiB handles=262144
WARN [03-01|16:33:10.560] Truncating dangling indexes database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.563] Truncating freezer table database=/home/user/.ethereum/geth/chaindata/ancient table=receipts items=2593457 limit=2593452
WARN [03-01|16:33:10.563] Truncating freezer table database=/home/user/.ethereum/geth/chaindata/ancient table=diffs items=2593457 limit=2593452
WARN [03-01|16:33:10.563] Truncating freezer table database=/home/user/.ethereum/geth/chaindata/ancient table=headers items=2593457 limit=2593452
WARN [03-01|16:33:10.563] Truncating freezer table database=/home/user/.ethereum/geth/chaindata/ancient table=hashes items=2593457 limit=2593452
INFO [03-01|16:33:10.565] Opened ancient database database=/home/user/.ethereum/geth/chaindata/ancient
INFO [03-01|16:33:10.576] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantin
ople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, YOLO v2: <nil>, Engine: ethash}"
INFO [03-01|16:33:10.576] Disk storage enabled for ethash caches dir=/home/user/.ethereum/geth/ethash count=3
INFO [03-01|16:33:10.576] Disk storage enabled for ethash DAGs dir=/home/user/.ethash count=2
INFO [03-01|16:33:10.584] Initialising Ethereum protocol versions="[65 64 63]" network=1 dbversion=8
ERROR[03-01|16:33:10.647] Canonical hash missing, can't freeze number=2594277
INFO [03-01|16:33:10.743] Deep froze chain segment blocks=825 elapsed=170.839ms number=2594276 hash="a41766…89ccd5"
INFO [03-01|16:33:12.419] Loaded most recent local header number=2684683 hash="1bafa3…36aeb9" td=94044217750532710492 age=4y3mo4w
INFO [03-01|16:33:12.419] Loaded most recent local full block number=2684683 hash="1bafa3…36aeb9" td=94044217750532710492 age=4y3mo4w
INFO [03-01|16:33:12.419] Loaded most recent local fast block number=2684683 hash="1bafa3…36aeb9" td=94044217750532710492 age=4y3mo4w
WARN [03-01|16:33:12.425] Head state missing, repairing number=2684683 hash="1bafa3…36aeb9"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x561131f1263f]
goroutine 1 [running]:
github.com/ethereum/go-ethereum/core/types.(*Block).Root(...)
github.com/ethereum/go-ethereum/core/types/block.go:336
github.com/ethereum/go-ethereum/core.(*BlockChain).SetHeadBeyondRoot.func1(0x7fcbd876a1d8, 0xc0001f6240, 0xc0006566c0, 0x7fcbd876a1d8, 0xc0001f6240)
github.com/ethereum/go-ethereum/core/blockchain.go:528 +0x2df
github.com/ethereum/go-ethereum/core.(*BlockChain).SetHeadBeyondRoot(0xc0000d7900, 0x28f70b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
github.com/ethereum/go-ethereum/core/blockchain.go:606 +0x295
github.com/ethereum/go-ethereum/core.(*BlockChain).SetHead(...)
github.com/ethereum/go-ethereum/core/blockchain.go:484
github.com/ethereum/go-ethereum/core.NewBlockChain(0x561132a6e528, 0xc0001f6240, 0xc0005dc140, 0x5611330aba60, 0x561132a6c0f8, 0xc000c8a000, 0x0, 0x0, 0x0, 0x0, ...)
github.com/ethereum/go-ethereum/core/blockchain.go:310 +0x1af0
github.com/ethereum/go-ethereum/eth.New(0xc0000ba9c0, 0xc00015f200, 0x561133099010, 0x1, 0x1)
github.com/ethereum/go-ethereum/eth/backend.go:175 +0xac6
github.com/ethereum/go-ethereum/cmd/utils.RegisterEthService(0xc0000ba9c0, 0xc00015f200, 0x0, 0x1)
github.com/ethereum/go-ethereum/cmd/utils/flags.go:1701 +0xec
main.makeFullNode(0xc00011a580, 0x561132586ba6, 0x1, 0xc0000992c0)
github.com/ethereum/go-ethereum/cmd/geth/config.go:165 +0xfb
main.geth(0xc00011a580, 0x0, 0x0)
github.com/ethereum/go-ethereum/cmd/geth/main.go:354 +0xf6
gopkg.in/urfave/cli%2ev1.HandleAction(0x56113285bfa0, 0x561132a252e8, 0xc00011a580, 0xc000280e40, 0x0)
gopkg.in/urfave/cli.v1@v1.20.0/app.go:490 +0x84
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc0000bc9c0, 0xc0000320a0, 0x5, 0x5, 0x0, 0x0)
gopkg.in/urfave/cli.v1@v1.20.0/app.go:264 +0x5f5
main.main()
github.com/ethereum/go-ethereum/cmd/geth/main.go:273 +0x57
And here is the log I get whenever I try to run geth now:
INFO [03-01|16:38:17.018] Starting Geth on Ethereum mainnet...
INFO [03-01|16:38:17.019] Maximum peer count ETH=50 LES=0 total=50
INFO [03-01|16:38:17.020] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [03-01|16:38:17.021] Set global gas cap cap=25000000
INFO [03-01|16:38:17.021] Allocated trie memory caches clean=1023.00MiB dirty=1024.00MiB
INFO [03-01|16:38:17.021] Allocated cache and file handles database=/home/user/.ethereum/geth/chaindata cache=2.00GiB handles=262144
INFO [03-01|16:38:17.086] Opened ancient database database=/home/user/.ethereum/geth/chaindata/ancient
Fatal: Failed to register the Ethereum service: gap (#2594277) in the chain between ancients and leveldb
I run geth using the following command:
geth --syncmode full --cache 4096
My .ethereum directory is currently 15GB, but I could upload the whole thing somewhere if that makes it easier for you guys to debug this.
@osizadmin can you provide some logs? Did you have a crash before the problem you describe?
@TheEdgeOfRage the crash should be fixed in the recent release - but you will need to resync. Thanks for your log this was most helpful
Hello there! We have something close to yours:
Command:
command: --gpo.blocks 10 --gpo.percentile 90 --ropsten --cache 2000 --syncmode "full" --nousb --http.api debug,eth,web3,personal,net --http --http.addr "0.0.0.0" --http.vhosts="*" --ws --ws.addr "0.0.0.0" --ws.origins="*" --port 30303 --ws.port 8546 --http.port 8545
Version:
OS: Ubuntu 20.04 LTS (GNU/Linux 5.4.0-28-generic x86_64)
Geth: v1.10.1
geth_1 | INFO [03-29|15:19:11.838] Maximum peer count ETH=50 LES=0 total=50
geth_1 | INFO [03-29|15:19:11.838] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
geth_1 | WARN [03-29|15:19:11.838] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable
geth_1 | INFO [03-29|15:19:11.838] Set global gas cap cap=25000000
geth_1 | INFO [03-29|15:19:11.839] Allocated trie memory caches clean=300.00MiB dirty=500.00MiB
geth_1 | INFO [03-29|15:19:11.839] Allocated cache and file handles database=/root/.ethereum/ropsten/geth/chaindata cache=1000.00MiB handles=524288
geth_1 | INFO [03-29|15:19:12.841] Opened ancient database database=/root/.ethereum/ropsten/geth/chaindata/ancient
geth_1 | Fatal: Failed to register the Ethereum service: gap (#9719292) in the chain between ancients and leveldb
geth_1 | Fatal: Failed to register the Ethereum service: gap (#9719292) in the chain between ancients and leveldb```
This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have more relevant information or answers to our questions so that we can investigate further.
Hey, Can someone help on the same issue? I am having the same error. I am running geth with --syncmode=full
Fatal: Failed to register the Ethereum service: gap (#12254272) in the chain between ancients and leveldb INFO [05-09|05:17:31.432] Starting Geth on Ethereum mainnet... INFO [05-09|05:17:31.433] Maximum peer count ETH=50 LES=0 total=50 INFO [05-09|05:17:31.433] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" WARN [05-09|05:17:31.433] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable INFO [05-09|05:17:31.434] Set global gas cap cap=25,000,000 INFO [05-09|05:17:31.434] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB INFO [05-09|05:17:31.434] Allocated cache and file handles database=/home/ubuntu/data/geth/chaindata cache=2.00GiB handles=524,288 INFO [05-09|05:17:33.255] Opened ancient database database=/home/ubuntu/data/geth/chaindata/ancient readonly=false Fatal: Failed to register the Ethereum service: gap (#12254272) in the chain between ancients and leveldb
Same issue with geth version v.1.10.2
/home/cryptodaemon/bin/geth --datadir=/home/cryptodaemon/.ethereum --rpc --rpcapi personal,eth,web3 --rpcaddr localhost --rpcport 8545 --maxpeers=100 --cache 2048 --nousb --syncmode "fast" --verbosity 5
INFO [05-31|10:59:59.995] Starting Geth on Ethereum mainnet...
INFO [05-31|10:59:59.997] Maximum peer count ETH=100 LES=0 total=100
WARN [05-31|10:59:59.997] The flag --rpc is deprecated and will be removed June 2021, please use --http
WARN [05-31|10:59:59.997] The flag --rpcaddr is deprecated and will be removed June 2021, please use --http.addr
WARN [05-31|10:59:59.997] The flag --rpcport is deprecated and will be removed June 2021, please use --http.port
WARN [05-31|10:59:59.997] The flag --rpcapi is deprecated and will be removed June 2021, please use --http.api
INFO [05-31|10:59:59.997] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [05-31|10:59:59.997] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable
TRACE[05-31|10:59:59.997] Started watching keystore folder path=/home/cryptodaemon/.ethereum/keystore
DEBUG[05-31|10:59:59.997] FS scan times list="338.446µs" set="97.653µs" diff="35.429µs"
TRACE[05-31|10:59:59.998] Handled keystore changes time=1.187307ms
DEBUG[05-31|10:59:59.999] Sanitizing Go's GC trigger percent=50
INFO [05-31|10:59:59.999] Set global gas cap cap=25000000
INFO [05-31|10:59:59.999] Allocated trie memory caches clean=307.00MiB dirty=512.00MiB
INFO [05-31|10:59:59.999] Allocated cache and file handles database=/home/cryptodaemon/.ethereum/geth/chaindata cache=1024.00MiB handles=524288
DEBUG[05-31|11:00:02.309] Chain freezer table opened database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=hashes items=12437071 size=379.55MiB
DEBUG[05-31|11:00:02.309] Chain freezer table opened database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=bodies items=12437071 size=253.89MiB
DEBUG[05-31|11:00:02.310] Chain freezer table opened database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=receipts items=12437071 size=1.61GiB
DEBUG[05-31|11:00:02.310] Chain freezer table opened database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=diffs items=12437071 size=123.48MiB
DEBUG[05-31|11:00:02.310] Chain freezer table opened database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=headers items=12437071 size=1.60GiB
INFO [05-31|11:00:02.310] Opened ancient database database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient readonly=false
Fatal: Failed to register the Ethereum service: gap (#12437071) in the chain between ancients and leveldb
Should be fixed by @fjl upcoming ancients refactor, will reopen and close once that's merged
This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have more relevant information or answers to our questions so that we can investigate further.
Bad Bot
thanks @MariusVanDerWijden is there a way I can fix it right now by syncing from the error block I am seeing? Like I don't want to sync fully again. I may want to sync from the block I am seeing has error
How can this be avoided?
i have same issue. is there any way to quick fix it? or rescan from that block?
i have same issue. is there any way to quick fix it? or rescan from that block?
did anybody try to delete that corrupted .ldb file?
This root cause of this issue should be fixed in Geth 1.10.4. Please try it.
@fjl do we need to resync after upgrade? Is there a shorter way so don't have to resync since start. I tried starting the client again after upgrade and it gives me same error.
INFO [06-17|19:39:34.486] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [06-17|19:39:34.486] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable
WARN [06-17|19:39:34.486] Disable transaction unindexing for archive node
INFO [06-17|19:39:34.487] Enabling recording of key preimages since archive mode is used
INFO [06-17|19:39:34.487] Set global gas cap cap=50,000,000
INFO [06-17|19:39:34.487] Allocated trie memory caches clean=1.20GiB dirty=0.00B
INFO [06-17|19:39:34.487] Allocated cache and file handles database=/home/ubuntu/data/geth/chaindata cache=2.00GiB handles=524,288
INFO [06-17|19:39:36.698] Opened ancient database database=/home/ubuntu/data/geth/chaindata/ancient readonly=false
Fatal: Failed to register the Ethereum service: gap (#12254272) in the chain between ancients and leveldb
Yes, you need to resync to fix this issue. Sorry about that. But the sync should be pretty fast with v1.10.4.
Yes, you need to resync to fix this issue. Sorry about that. But the sync should be pretty fast with v1.10.4.
Got this issue after updating to 1.10.4 I'm running a validator, but it's okay cause I'm using Infura as backup. How do I resync geth? Delete the database and restart geth?
The file explorer freezes when I try to select all files in the chaindata folder. The rm -rfv command acts like it did something, but the files are still there My chain data is in var/lib/goethereum/geth/chaindata (I think geth expects it to be in a different location) Is there a way to specify the path with geth removedb? geth removedb returns this...
anthony@ethstaker:~$ geth removedb INFO [06-17|21:12:34.742] Maximum peer count ETH=50 LES=0 total=50 INFO [06-17|21:12:34.742] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" INFO [06-17|21:12:34.742] Set global gas cap cap=50,000,000 INFO [06-17|21:12:34.742] Full node state database missing path=/home/anthony/.ethereum/geth/chaindata INFO [06-17|21:12:34.742] Full node ancient database missing path=/home/anthony/.ethereum/geth/chaindata/ancient INFO [06-17|21:12:34.742] Light node database missing path=/home/anthony/.ethereum/geth/lightchaindata
I ended up deleting the entire chaindata folder. I didn't want to do it before because I didn't if it had special permissions or not.
Also, I wasn't using sudo geth removedb before. But using sudo didn't help anyway.
The chaindata directory doesn't use any special permissions. You can try using geth --datadir ... removedb
.
Thanks for the response. All is working now.
To correct something I said earlier, I originally said this happened after I upgraded to geth v1.10.4. This is incorrect. I only noticed the issue after I upgraded.
Upon checking infura stats, my validator has been using infura since May 27th... while using the previous version of geth.
I'm not gonna go into an in-depth analysis about what I think happened to my machine, because I don't think it's very useful to anyone (and parts of my analysis doens't really line up). But a week prior to May 27th, my internet went and I was troubleshooting why. (if it was my fault or my ISP fault). A chaindata file could've been corrupted while I was rebooting and moving the machine to diagnose the issue. Though this doesn't really make sense. Because internet was restored on May 22nd and my validator was running fine (with geth, not infura) from the 22nd to the 27th.
But, at the end of the day, it doesn't matter to anyone what caused my issue. All that matters is that is is not an issue with geth v1.10.4 and that deleting the chaindata folder and letting geth resync is the solution.
Thanks all and take care.
This is happening with v1.10.4
Fatal: Failed to register the Ethereum service: gap (#12674612) in the chain between ancients and leveldb
Fatal: Failed to register the Ethereum service: gap (#12674612) in the chain between ancients and leveldb
to clarify: this is post upgrade to v1.10.4, not an empty ledger.
Got this yet again after updating to geth 1.10.6
Fatal: Failed to register the Ethereum service: gap (#12793685) in the chain between ancients and leveldb
Why is geth still filled with so many errors? can't finish syncing without 10 errors in the process
Just got this after gracefully stopping and restarting the 1.10.8
I'm getting a similar error. Syncing to the mainnet on Rocketpool.
eth1_1 | INFO [10-13|14:24:43.809] Maximum peer count ETH=50 LES=0 total=50 eth1_1 | INFO [10-13|14:24:43.809] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" eth1_1 | INFO [10-13|14:24:43.810] Set global gas cap cap=50,000,000 eth1_1 | INFO [10-13|14:24:43.819] Allocated trie memory caches clean=307.00MiB dirty=512.00MiB eth1_1 | INFO [10-13|14:24:43.819] Allocated cache and file handles database=/ethclient/geth/geth/chaindata cache=1024.00MiB handles=524,288 eth1_1 | INFO [10-13|14:24:43.966] Opened ancient database database=/ethclient/geth/geth/chaindata/ancient readonly=false eth1_1 | Fatal: Failed to register the Ethereum service: gap (#4988667) in the chain between ancients and leveldb eth1_1 | Fatal: Failed to register the Ethereum service: gap (#4988667) in the chain between ancients and leveldb
Same on 1.10.12-unstable-476fb565-20211105, but to be fair I probably shouldn't be using sshfs
Still same issue after v.1.10.4. Right now we have 1.10.8
Still not fixed
Mar 08 15:36:15 sa9 geth[2193755]: Fatal: Failed to register the Ethereum service: gap (#14256897) in the chain between ancients and leveldb
Geth
Version: 1.10.13-stable
Git Commit: 7a0c19f813e285516f4b525305fd73b625d2dec8
Git Commit Date: 20211124
Architecture: amd64
Go Version: go1.17.2
Operating System: linux
Version v.1.10.19 problem still here, after 1 week of geth start and resync.
Fatal: Failed to register the Ethereum service: gap (#166416) in the chain between ancients and leveldb
I'm getting this error:
Fatal: Failed to register the Ethereum service: failed to retrieve genesis from ancient EOF
Verison 1.10.23-stable
Happens pretty frequently when trying to sync geth. Happens when my USB fails, which is connecting my SSD to my machine, and I reconnect/reboot.
Getting this on a Geth v1.10.23 after an unclean shutdown 🤬
hitting this also , v1.10.23
can confirm still happening with v.10.26
This could possibly be related to a bug in the 'freezer repair' procedure. For anyone encountering this issue, please let us know if your geth printed any of the following log messages:
Truncating dangling indexes
Truncating dangling head
Fatal: Failed to register the Ethereum service: gap (#986975) in the chain between ancients and leveldb Fatal: Failed to register the Ethereum service: gap (#986975) in the chain between ancients and leveldb
Fatal: Failed to register the Ethereum service: gap (#5436187) in the chain between ancients and leveldb Fatal: Failed to register the Ethereum service: gap (#5436187) in the chain between ancients and leveldb
release geth v1.10.26 stable
Just restart my full geth node
I'm getting this error:
Fatal: Failed to register the Ethereum service: failed to retrieve genesis from ancient EOF
Verison 1.10.23-stable
Happens pretty frequently when trying to sync geth. Happens when my USB fails, which is connecting my SSD to my machine, and I reconnect/reboot.
@edsonayllon the failed to retrieve genesis
error should stop the boot-up phase, since it detects that you are missing data that should be present. You say that it happens when USB fails, and thus the ancients cannot be read.
I don't understand what the bug report is. Other than printing an error and refusing to start, what do you expect geth to do? I would expect the following:
System information
Geth version: 1.9.25-e7872729
Error : Fatal: Failed to register the Ethereum service: gap (#492149) in the chain between ancients and leveldb
We have faced the above error while start the geth in fast syncmode.