[testnet node stuck at block 1322393] unsupported ledger version, propagating failure even at newest dockerhub image

[jun0tpyrc]

stellar-stdout seems continue to sync new blocks

2020-10-21T02:48:19.963 GBUCO [History INFO] Verifying ledgers [1322394,1327680)
2020-10-21T02:48:19.963 GBUCO [History ERROR] Catchup material failed verification - unsupported ledger version, propagating failure [VerifyLedgerChainWork.cpp:438]
2020-10-21T02:48:19.963 GBUCO [History ERROR] Upgrade this stellar-core installation to newest version [VerifyLedgerChainWork.cpp:441]
2020-10-21T02:48:19.963 GBUCO [History INFO] Verifying ledgers [1322394,1327680)
2020-10-21T02:48:19.963 GBUCO [History INFO] Catching up to ledger 1327679: Failed: verify-ledger-chain
2020-10-21T02:48:19.963 GBUCO [History INFO] Catching up to ledger 1327679: Succeeded: batch-download-ledger-00142dbf-0014427f : 332/332 children completed
2020-10-21T02:48:20.042 GBUCO [Herder INFO] Quorum information for 1327737 : {"agree":3,"delayed":0,"disagree":0,"fail_at":2,"hash":"59d361","lag_ms":21,"ledger":1327737,"missing":0,"phase":"EXTERNALIZE"}

2020-10-21T02:48:20.044 GBUCO [Ledger INFO] Got consensus: [seq=1327739, prev=2a48b2, txs=2, ops=89, sv: [ SIGNED@sdf_testnet_1 txH: 5076df, ct: 1603248499, upgrades: [ ] ]]
2020-10-21T02:48:20.044 GBUCO [Ledger INFO] Close of ledger 1327739 buffered
2020-10-21T02:48:21.737 GBUCO [Overlay INFO] Connected to 35.202.159.46:11625
2020-10-21T02:48:21.928 GBUCO [Overlay INFO] Peer 35.202.159.46:11625 dropped us; reason: ERR_LOAD (peer rejected)
2020-10-21T02:48:24.771 GBUCO [Overlay INFO] Connected to 35.202.159.46:11625
2020-10-21T02:48:24.963 GBUCO [Overlay INFO] Peer 35.202.159.46:11625 dropped us; reason: ERR_LOAD (peer rejected)
2020-10-21T02:48:25.961 GBUCO [Herder INFO] Quorum information for 1327738 : {"agree":3,"delayed":0,"disagree":0,"fail_at":2,"hash":"59d361","lag_ms":21,"ledger":1327738,"missing":0,"phase":"EXTERNALIZE"}

2020-10-21T02:48:25.963 GBUCO [Ledger INFO] Got consensus: [seq=1327740, prev=5342b6, txs=3, ops=81, sv: [ SIGNED@sdf_testnet_3 txH: 48f9e0, ct: 1603248504, upgrades: [ ] ]]
2020-10-21T02:48:25.963 GBUCO [Ledger INFO] Close of ledger 1327740 buffered
2020-10-21T02:48:31.166 GBUCO [Herder INFO] Quorum information for 1327739 : {"agree":3,"delayed":0,"disagree":0,"fail_at":2,"hash":"59d361","lag_ms":24,"ledger":1327739,"missing":0,"phase":"EXTERNALIZE"}

stellar-core-stderr (empty)

horizon-stdout

time="2020-10-21T02:50:58.732Z" level=info msg="request failed due to error" error="loading transaction records: executing transaction records query: sql: no rows in result set" pid=21 req=bstg-xlm-1/BFKppYLlUe-015283 stack="[main.go:43 transaction.go:115 handler.go:193 handler.go:264 middleware.go:242 server.go:2041 chain.go:31 mux.go:425 server.go:2041 http.go:73 server.go:2041 cors.go:190 server.go:2041 compress.go:190 server.go:2041 middleware.go:215 server.go:2041 middleware.go:113 server.go:2041 middleware.go:85 server.go:2041 xff.go:64 server.go:2041 middleware.go:47 server.go:2041 request_id.go:76 server.go:2041 middleware.go:37 server.go:2041 strip.go:25 server.go:2041 mux.go:82]"
time="2020-10-21T02:50:58.732Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=322 client_name=js-stellar-sdk client_version=4.0.2 duration=0.000635214 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.32.42 ip_port="10.196.32.42:43596" method=GET path="/accounts/GCL7QAKNIBGFRFA3RBC3FBQOQXQ2KFNR6N647NJNJOOHNUU6ULBLG6NO/transactions?cursor=3017858770542592&c=0.7960719862759782" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015283 route="/accounts/{account_id:\\w+}/transactions" status=404 streaming=false
time="2020-10-21T02:50:58.738Z" level=info msg="Starting request" app_name=undefined app_version=undefined client_name=js-stellar-sdk client_version=4.0.2 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.33.8 ip_port="10.196.33.8:33666" method=GET path="/accounts/GDF676YSEN4CQ6K6HY3K3QLSHLYU275XB5ZT5MC6JKPXR32P5YKAYVZX?c=0.583803628737664" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015284 streaming=false
time="2020-10-21T02:50:58.741Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=2461 client_name=js-stellar-sdk client_version=4.0.2 duration=0.003055872 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.33.8 ip_port="10.196.33.8:33666" method=GET path="/accounts/GDF676YSEN4CQ6K6HY3K3QLSHLYU275XB5ZT5MC6JKPXR32P5YKAYVZX?c=0.583803628737664" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015284 route="/accounts/{account_id}/" status=200 streaming=false
time="2020-10-21T02:50:58.746Z" level=info msg="Starting request" app_name=undefined app_version=undefined client_name=js-stellar-sdk client_version=4.0.2 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.32.42 ip_port="10.196.32.42:43662" method=GET path="/accounts/GDF676YSEN4CQ6K6HY3K3QLSHLYU275XB5ZT5MC6JKPXR32P5YKAYVZX/transactions?cursor=3018064928968704&c=0.45321568483692287" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015285 streaming=false
time="2020-10-21T02:50:58.746Z" level=info msg="request failed due to error" error="loading transaction records: executing transaction records query: sql: no rows in result set" pid=21 req=bstg-xlm-1/BFKppYLlUe-015285 stack="[main.go:43 transaction.go:115 handler.go:193 handler.go:264 middleware.go:242 server.go:2041 chain.go:31 mux.go:425 server.go:2041 http.go:73 server.go:2041 cors.go:190 server.go:2041 compress.go:190 server.go:2041 middleware.go:215 server.go:2041 middleware.go:113 server.go:2041 middleware.go:85 server.go:2041 xff.go:64 server.go:2041 middleware.go:47 server.go:2041 request_id.go:76 server.go:2041 middleware.go:37 server.go:2041 strip.go:25 server.go:2041 mux.go:82]"
time="2020-10-21T02:50:58.746Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=322 client_name=js-stellar-sdk client_version=4.0.2 duration=0.000597502 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.32.42 ip_port="10.196.32.42:43662" method=GET path="/accounts/GDF676YSEN4CQ6K6HY3K3QLSHLYU275XB5ZT5MC6JKPXR32P5YKAYVZX/transactions?cursor=3018064928968704&c=0.45321568483692287" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015285 route="/accounts/{account_id:\\w+}/transactions" status=404 streaming=false
time="2020-10-21T02:50:58.753Z" level=info msg="Starting request" app_name=undefined app_version=undefined client_name=js-stellar-sdk client_version=4.0.2 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.33.8 ip_port="10.196.33.8:33666" method=GET path="/accounts/GAZSPN35UODWGQ3E3VYKVMP7T3OPBEQZJE3QJLXPF6VKWAZP7TYWCKQX?c=0.09843990698251082" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015286 streaming=false
time="2020-10-21T02:50:58.758Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=2481 client_name=js-stellar-sdk client_version=4.0.2 duration=0.005671598 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.33.8 ip_port="10.196.33.8:33666" method=GET path="/accounts/GAZSPN35UODWGQ3E3VYKVMP7T3OPBEQZJE3QJLXPF6VKWAZP7TYWCKQX?c=0.09843990698251082" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015286 route="/accounts/{account_id}/" status=200 streaming=false
time="2020-10-21T02:50:58.766Z" level=info msg="Starting request" app_name=undefined app_version=undefined client_name=js-stellar-sdk client_version=4.0.2 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.32.42 ip_port="10.196.32.42:43662" method=GET path="/accounts/GAZSPN35UODWGQ3E3VYKVMP7T3OPBEQZJE3QJLXPF6VKWAZP7TYWCKQX/transactions?cursor=5649913513811968&c=0.05051725100915716" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015287 streaming=false
time="2020-10-21T02:50:58.768Z" level=info msg="Finished request" app_name=undefined app_version=undefined bytes=757 client_name=js-stellar-sdk client_version=4.0.2 duration=0.002223337 forwarded_ip=172.28.72.241 host="xlm-node.app-bstg.local:8000" ip=10.196.32.42 ip_port="10.196.32.42:43662" method=GET path="/accounts/GAZSPN35UODWGQ3E3VYKVMP7T3OPBEQZJE3QJLXPF6VKWAZP7TYWCKQX/transactions?cursor=5649913513811968&c=0.05051725100915716" pid=21 referer=undefined req=bstg-xlm-1/BFKppYLlUe-015287 route="/accounts/{account_id:\\w+}/transactions" status=200 streaming=false
time="2020-10-21T02:50:58.925Z" level=info msg="Ingestion system state machine transition" current_state="resume(latestSuccessfullyProcessedLedger=1322393)" next_state="resume(latestSuccessfullyProcessedLedger=1322393)" pid=21 service=expingest
time="2020-10-21T02:50:58.955Z" level=info msg="Waiting for ledger to be available in stellar-core" core_sequence=1322393 ingest_sequence=1322394 pid=21 service=expingest
time="2020-10-21T02:50:59.956Z" level=info msg="Ingestion system state machine transition" current_state="resume(latestSuccessfullyProcessedLedger=1322393)" next_state="resume(latestSuccessfullyProcessedLedger=1322393)" pid=21 service=expingest

[jun0tpyrc]

the line in horizon-stdout

loading transaction records: executing transaction records query: sql: no rows in result set might be related?

or this might be file / data corruption?

[jun0tpyrc]

2020-10-21T05:14:46.596 GADW2 [History INFO] Verifying ledgers [1322394,1329408)
2020-10-21T05:14:46.596 GADW2 [History ERROR] Catchup material failed verification - unsupported ledger version, propagating failure [VerifyLedgerChainWork.cpp:438]
2020-10-21T05:14:46.596 GADW2 [History ERROR] Upgrade this stellar-core installation to newest version [VerifyLedgerChainWork.cpp:441]
2020-10-21T05:14:46.596 GADW2 [History INFO] Verifying ledgers [1322394,1329408)
2020-10-21T05:14:46.596 GADW2 [History INFO] Catching up to ledger 1329407: Failed: verify-ledger-chain
2020-10-21T05:14:46.596 GADW2 [History INFO] Catching up to ledger 1329407: Succeeded: batch-download-ledger-00142dbf-0014493f : 440/440 children completed
2020-10-21T05:14:46.890 GADW2 [Herder INFO] Quorum information for 1329410 : {"agree":3,"delayed":0,"disagree":0,"fail_at":2,"hash":"59d361","lag_ms":13,"ledger":1329410,"missing":0,"phase":"EXTERNALIZE"}

even updated to newest image in docker hub (13 hours ago ) stellar/quickstart:latest DIGEST:sha256:b48e00fb86095830028e62a47abdb97daa35186735198d5f2d127509714b1c70

[jun0tpyrc]

ok seems https://www.stellar.org/blog/announcing-stellar-protocol-15 is the reason, For that reason, can we have a quickstart image at docker hub soon? thanks

[MonsieurNicolas]

The quickstart image already supports testnet, you need to use the Docker.testing docker file.

I think it's a situation where the instructions are not super clear?

[bert2002]

Any chance to get the apt repository updated?

[MonsieurNicolas]

Right now the only apt package available is in the unstable repo, that's what https://github.com/stellar/docker-stellar-core-horizon/blob/master/Dockerfile.testing is using.

The promotion to stable (that can be deployed to production nodes) is scheduled for next week.

[YuWT-CN]

using stellar/quickstart:testing is ok.

[jun0tpyrc]

verified image now working to catch up testnet , thanks

[bert2002]

Using the rc1 build of https://apt.stellar.org/pool/testing/s/stellar-core/stellar-core_15.0.0rc1-318~4e7d3fd~v2_amd64.deb and seems to work too.