Seed node failed to start with statesync while state and block pruning are enabled

[smuu]

Summary of Bug

When starting with statsync downloading the snapshot fails after downloading all chunks without any retry. I restarted the node multiple times and I see the same behavior everytime.

Statesync related logs: https://pastebin.com/auaFdz9i

Happy to provide more context & information when needed.

Version

v1.6.0

Steps to Reproduce

environment

CELESTIA_HOME=/home/celestia

start command:

start --home $(CELESTIA_HOME) --rpc.laddr tcp://0.0.0.0:26657 --api.enable --grpc.enable

config.toml

priv_validator_key_file = "keys/priv_validator_key.json"
node_key_file = "keys/node_key.json"
[p2p]
persistent_peers = "4b19506a0432de219ee2db883c53756c1b3cbdfb@88.217.142.187:26656,890940193b592e7b53b882b62bf85d7a25190192@51.222.244.105:26656,a623f039a36391662a3b514d1fa17fbaa2e25319@149.50.96.24:16656,7d5767dfe229d6b82246165c1c353a56a6da1983@54.177.201.19:26656,de7a0953f9d5878604c077a54925167c120a97eb@141.95.65.115:26656,c89507c926f41901784f54b26e1ccc49ac9ecfda@31.214.144.83:20056,f103c4809c8263b311463422195e6ad8526911d6@74.118.136.167:26656,0cb62ebbce54e2d5e72ced753a879d3c08c61583@162.19.171.80:26656,dd603ca5470c0f195dea38d1f938b9e43a729a47@162.19.170.154:26656,6ba9f87efe6beb23bb7af2204eac769767d8ec78@78.129.165.127:23656,491c37d5de210600463c5599a3ffd272c8690014@57.128.20.184:23656,7f067f54987cccfe489fbe302ff1bf6c56aaf02f@162.19.81.54:23656,0cca24be2f0d845e67a3135fb67dff6e71750e60@148.251.151.51:16400,3af3e3556515eb68bfe4ec5ad134eb9ec56b521d@51.81.166.95:16400,994574dc373fc7552d5fb2616ca9d40654760a91@141.164.42.9:16400,d364357f55e37a038d9dbd143448ea792bb8edcf@15.235.115.156:16400,624257ef1a8b6e2d39ac13ed3e3e16963fbb54ea@149.102.156.193:26656,d5fafee95503bb76719f5c2849f3fe0be46e758c@217.160.39.214:26656,45779d388b18ad48d98c6ad54f35e95fe2a3b35a@165.22.30.175:26656,4ebfc1fac7a646105956ee563ea0d760e3099269@celestia-peer1.easy2stake.com:26756,87af580078c80c630625db2360b3dc19483d29cd@185.182.194.163:26656,005b439eca9a27b736156c0081a863a40b3be517@peers.celestia-node.pn.prod.fcstech.de:26656,5992372287bbb173bd291e299a26495057310165@57.128.74.73:26656,29c8a82a0be59a2c6a5d6fb2ad0a2e1b4d09de0f@181.188.232.25:26656,72dd61ecab64873e45c77f902fec73069dfb33f0@103.219.171.65:26656,0a0da77a47cc1a58febe39c73b3bc96758455de6@142.165.207.45:26656,3c18baae029c2d39347ef77ac777fe6fd38bc1e6@176.9.48.38:20056,1d824f2cf4df9c9da98c120b78be3f4319a1d7ff@46.4.23.42:26656,b1187a51400cef8b49a8482088eac57327989fe8@135.181.75.114:26656,deefd5d3a4d497be6772295d10174f021d344ac5@57.128.75.161:26656,094e037734702dc0803a58a10b50fc3bb46c7e73@51.210.34.156:26656,24a869f01efa19104dbce61ae6697952a9e9a5d2@139.84.232.124:26656,1b92a439555a19ee6edea85b184abb6873757dae@51.89.173.96:43656,8240e8a13594d40b6839f183795c551503309d3c@57.128.87.18:26656,3b11e4123a9db23f72fb5828b3f5b101217ee43d@116.202.208.214:2600,c1c92e14d641a6368627da662b96ff77135e4754@185.182.193.20:26656,4ebe56795fc5da1c0dc5f7acd5760d5c8fd4d250@5.9.87.216:33357,bb72dcd76b96214c4e647a1ebfe47c9e541d5325@185.182.186.202:26656,be5ea8954fa452ad464ae8a7067f01bf5a0da556@147.135.144.53:26656,95c00bb9f24e907891e9a98c4ad506af093a7944@87.106.237.243:26656,4ad28c8a0b04bac4aac441b105c815c449e2182f@65.108.61.189:26656,c7dac72f630d2b3c3354ed664bcd6704a8f9e627@51.79.229.177:26656,ce2873ef3893f0da50eb27892761e9c06e704944@159.203.143.94:26656,1b92a439555a19ee6edea85b184abb6873757dae@51.89.173.96:26656,ea9d858371ba86ae8f9c5b292c105329029c9efd@5.199.174.19:26656,5bc86308e2b3730ce611abf622c8be752780f934@5.199.172.53:26656,f5bce79feec9068154e7f50fe8db7a8191a58663@82.220.38.204:10056,629c36a551ee1a3c8bb3ecee6e00a9977a60e53e@65.108.46.248:56656,d0c530290267e1c539f27061c9446d5042eaf1ca@161.97.180.20:26656,b3f473393a6469c0da65e4a7ebaa3ff3182df705@37.120.245.32:26656,a41b4bc451b1b71d537aa1018226e08d7fa7e44e@5.255.77.44:26656,2f05b07cde80c5b08ee0159658f7977c7765f966@148.113.17.55:26756,572cb08735d4572fe62b2fc8b9555c479d8e162f@65.108.137.217:26656,22ea694db139e744c46c74c9b5f22851266630af@51.159.80.121:2090,5001de72be39622c9dc34f2117eccc3f3fca8a7a@celestia-mainnet-01.twinstake.io:26756,9a82b9bc0ab0f15692952aa829c8ba7d2f55315f@78.46.93.125:26666,e979a4f3528a4fac1c1ab93af14d26b978eead8a@162.19.72.172:26656,b751ffe713e3e5d80e8f45c3bd5e640ee9a91cb9@10.129.127.95:26656,fef05ab5f892d8c85c13e08f49935f1b0077efd0@95.163.230.31:26656,1d607e1859212eb8cd4130a1df074d201fb5186c@168.119.64.26:26656,0015a3241645e7029804f5e363cdc1822eb0797b@mainnet.celestia.stakingcabin.com:26656,4727239dfe0173ef862f55c2fa26e9a6a057a572@celestia-mainnet.stakingwizard.com:26656,5b0d1b8cb6f893426168300f69a82750d4be8729@193.35.57.185:11656,23b88ebcfb2177dbd2d8b2920c363a25e038e69a@89.58.61.213:2000,00133e62873e724700421d41d4f2f6d24b849bf4@185.111.159.231:2000,58e2c4d930c235aeee1e5bd4b906aa373f0ad4f5@65.21.232.33:2000,a86cfe5a22e73ff0c9ceec388e9b52bf8355efdd@85.239.233.57:2000,b833d0cac06e9831550f3273de21792cec23211d@78.46.21.248:26656,76455257d6ae848d17b6e324b1b278c01f4b769d@celestia-peer.freshstaking.com:34656,62b11589e64b4cef9f7202a2fd012937b1522745@celestia-mainnet.spidey.services:26603,b8c78f8edb68486ed07d7c34fc694be4fc37cd34@144.217.71.152:26656"
seed_mode = true
max_num_inbound_peers = 80
max_num_outbound_peers = 80
[instrumentation]
prometheus = true
[tx_index]
indexer = "kv"
[rpc]
max_subscriptions_per_client = 300
[statesync]
enable = true
rpc_servers = "http://consensus-full-rpc:26657,http://consensus-full-rpc:26657"
trust_height = 1182160
trust_hash = "2B259F441546BA25F62643D3C79AA1DF21B3405F8F9DE1FCB1604FF9164EA6F5"

app.toml

minimum-gas-prices = "0.002utia"
min-retain-blocks = 1
pruning = "everything"
[state-sync]
snapshot-interval = 0

[smuu]

After around 3 hours, the node seems to have the snapshot applied and started syncing.

[evan-forbes]

unfortunatley, there is no service discovery, so connecting to a node that will server the state sync snapshot that was requested can take a long time