ChainSafe / gossamer

🕸️ Go Implementation of the Polkadot Host
https://chainsafe.github.io/gossamer
GNU Lesser General Public License v3.0
427 stars 110 forks source link

Tests: Flaky tests failing in our CI #4035

Closed dimartiro closed 3 weeks ago

dimartiro commented 3 weeks ago

Issue summary

We have a flaky test failing in our CI

Run go test -timeout=45m -tags integration github.com/ChainSafe/gossamer/dot
go: downloading github.com/go-playground/validator/v10 v10.21.0
202[4](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:5)-06-06T18:44:05Z INFO     🕸️ initialising node with name Gossamer, id gssmr, base path /tmp/TestWriteGenesisSpecFile_Integration330[5](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:6)430329/001 and chain-spec /home/runner/actions-runner/_work/gossamer/gossamer/chain/westend-dev/westend-dev-spec-raw.json...    pkg=dot
2024-06-06T18:44:06Z INFO     🔨 finalised block #0 (0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 0, set id 0  pkg=state
2024-0[6](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:7)-06T18:44:06Z INFO     block state hash genesis hash: 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8   pkg=state
2024/06/06 18:44:06 [JOB 1] WAL file /tmp/TestWriteGenesisSpecFile_Integration3305430329/001/db/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:06 [JOB 1] WAL file /tmp/TestWriteGenesisSpecFile_Integration3305430329/001/db/000004.log with log number 000004 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:06 [JOB 1] WAL file /tmp/TestWriteGenesisSpecFile_Integration3305430329/001/db/000005.log with log number 000005 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:06 [JOB 1] WAL file /tmp/TestWriteGenesisSpecFile_Integration3305430329/001/db/000006.log with log number 000006 stopped reading at offset: 115991[7](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:8); replayed 159 keys in 15 batches
2024-06-06T18:44:06Z INFO     node initialised with name Gossamer, id gssmr, base path /tmp/TestWriteGenesisSpecFile_Integration3305430329/001, chain-spec /home/runner/actions-runner/_work/gossamer/gossamer/chain/westend-dev/westend-dev-spec-raw.json, block 0 and genesis hash 0x276bfa91f70[8](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:9)59348285599321ea96afd3ae681f0be47d36196bac8075ea32e8   pkg=dot
2024-06-06T18:44:06Z INFO     🕸️ initialising node with name Gossamer, id gssmr, base path /tmp/TestBuildFromDB_Integration2234440180/001 and chain-spec /home/runner/actions-runner/_work/gossamer/gossamer/chain/westend-dev/westend-dev-spec-raw.json... pkg=dot
2024-06-06T18:44:07Z INFO     🔨 finalised block #0 (0x276bfa[9](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:10)1f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 0, set id 0   pkg=state
2024-06-06T18:44:07Z INFO     block state hash genesis hash: 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8 pkg=state
2024/06/06 18:44:07 [JOB 1] WAL file /tmp/TestBuildFromDB_Integration2234440180/001/db/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:07 [JOB 1] WAL file /tmp/TestBuildFromDB_Integration2234440180/001/db/000004.log with log number 000004 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:07 [JOB 1] WAL file /tmp/TestBuildFromDB_Integration2234440180/001/db/000005.log with log number 000005 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:07 [JOB 1] WAL file /tmp/TestBuildFromDB_Integration2234440180/001/db/000006.log with log number 000006 stopped reading at offset: 1159917; replayed 159 keys in 15 batches
2024-06-06T18:44:07Z INFO     node initialised with name Gossamer, id gssmr, base path /tmp/TestBuildFromDB_Integration2234440180/001, chain-spec /home/runner/actions-runner/_work/gossamer/gossamer/chain/westend-dev/westend-dev-spec-raw.json, block 0 and genesis hash 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8  pkg=dot
2024/06/06 18:44:07 [JOB 1] WAL file /tmp/TestBuildFromDB_Integration2234440180/001/db/000008.log with log number 000008 stopped reading at offset: 43; replayed 1 keys in 1 batches
2024-06-06T18:44:07Z INFO     created state service with head 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8, highest number 0 and genesis hash 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8  pkg=state
2024-06-06T18:44:07Z INFO     🕸️ initialising node with name Gossamer, id gssmr, base path /tmp/TestBuildFromDB965264979/001 and chain-spec /home/runner/actions-runner/_work/gossamer/gossamer/chain/westend-dev/westend-dev-spec-raw.json...  pkg=dot
2024-06-06T18:44:08Z INFO     🔨 finalised block #0 (0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 0, set id 0  pkg=state
2024-06-06T18:44:08Z INFO     block state hash genesis hash: 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8 pkg=state
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestBuildFromDB965264979/001/db/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestBuildFromDB965264979/001/db/000004.log with log number 000004 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestBuildFromDB965264979/001/db/000005.log with log number 000005 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestBuildFromDB965264979/001/db/000006.log with log number 000006 stopped reading at offset: 1159917; replayed 159 keys in 15 batches
2024-06-06T18:44:08Z INFO     node initialised with name Gossamer, id gssmr, base path /tmp/TestBuildFromDB965264979/001, chain-spec /home/runner/actions-runner/_work/gossamer/gossamer/chain/westend-dev/westend-dev-spec-raw.json, block 0 and genesis hash 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8   pkg=dot
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestBuildFromDB965264979/001/db/000008.log with log number 000008 stopped reading at offset: 43; replayed 1 keys in 1 batches
2024-06-06T18:44:08Z INFO     created state service with head 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8, highest number 0 and genesis hash 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8  pkg=state
2024-06-06T18:44:08Z INFO     🕸️ initialising node with name Gossamer, id gssmr, base path /tmp/TestImportState_Integration1132615983/001 and chain-spec /tmp/TestImportState_Integration1132615983/002/genesis.json... pkg=dot
2024-06-06T18:44:08Z INFO     🔨 finalised block #0 (0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 0, set id 0  pkg=state
2024-06-06T18:44:08Z INFO     block state hash genesis hash: 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8 pkg=state
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestImportState_Integration1132615983/001/db/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestImportState_Integration1132615983/001/db/000004.log with log number 000004 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:08 [JOB 1] WAL file /tmp/TestImportState_Integration1132615983/001/db/000005.log with log number 000005 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:09 [JOB 1] WAL file /tmp/TestImportState_Integration1132615983/001/db/000006.log with log number 000006 stopped reading at offset: 1159913; replayed 159 keys in 15 batches
2024-06-06T18:44:09Z INFO     node initialised with name Gossamer, id gssmr, base path /tmp/TestImportState_Integration1132615983/001, chain-spec /tmp/TestImportState_Integration1132615983/002/genesis.json, block 0 and genesis hash 0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8  pkg=dot
2024-06-06T18:44:09Z INFO     ImportState with header: ParentHash=0x3b45c9c22dcece75a30acc9c2968cb311e6b0557350f83b430f47559db786975 Number=1482002 StateRoot=0x09f9ca28df0560c2291aa16b56e15e07d1e1927088f51356d522722aa90ca7cb ExtrinsicsRoot=0xda26dc8c1455f8f81cae12e4fc59e23ce961b2c837f6d3f664283af906d344e0 Digest=[PreRuntimeDigest ConsensusEngineID=BABE Data=0x013c0000009659bd0f0000000070edad1c9064fff78cb18435223d8adaf5ea04c24b1a8766e3dc01eb03cc6a0c11b79793d4e31cc0990838229c44fed1669a7c7c79e1e6d0a96374d6496728069d1ef739e290497a0e3b728fa88fcbdd3a5504e0efde0242e7a806dd4fa9260c SealDigest ConsensusEngineID=BABE Data=0x9e7f28dddcf27c1e6b328d5694c368d5b2ec5dbe0e412ae1c98f88d53be4d8502fac571f3f19c9caaf281a673319241e0c5095a683ad34316204088a36a4bd86] Hash=0x587f6da1bfa71a675f[10](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:11)dfa0f63edfcf168e8ece97eb5f526aaf0e8a8e82db3f  pkg=dot
2024/06/06 18:44:09 [JOB 1] WAL file /tmp/TestImportState_Integration1132615983/001/db/000008.log with log number 000008 stopped reading at offset: 43; replayed 1 keys in 1 batches
2024-06-06T18:44:09Z INFO     importing storage trie from base path /tmp/TestImportState_Integration[11](https://github.com/ChainSafe/gossamer/actions/runs/9392235477/job/25908991712?pr=3954#step:8:12)32615983/001 with root 0x09f9ca28df0560c2291aa16b56e15e07d1e1927088f51356d522722aa90ca7cb...   pkg=state
2024-06-06T18:44:10Z INFO     finished state import pkg=state
2024/06/06 18:44:10 [JOB 1] WAL file /tmp/TestImportState_Integration1132615983/001/db/000023.log with log number 000023 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/06/06 18:44:10 000022.sst:
open /tmp/TestImportState_Integration1132615983/001/db/000022.sst: no such file or directory
directory contains 14 files, 0 unknown, 7 tables, 1 logs, 3 manifests
FAIL    github.com/ChainSafe/gossamer/dot   4.741s
FAIL

The interesting thing is that this is failing only in our CI, I cannot reproduce it locally. So it should be related with something in our CI

➜  gossamer git:(diego/trie-inserts) go test -timeout=45m -tags integration github.com/ChainSafe/gossamer/dot
ok      github.com/ChainSafe/gossamer/dot   53.872s