search

celestiaorg / test-infra

Testing infrastructure for the Celestia Network
Apache License 2.0
25 stars 10 forks source link

Local docker big block test panics at the end of a successful test #206

Open evan-forbes opened 1 year ago

evan-forbes commented 1 year ago

when running testground run composition -f compositions/local-docker/big-blocks/001-val-large-txs-4.toml --wait locally, I'm consistently seeing

Apr 10 12:53:29.207423  INFO    all outcomes are complete   {"runner": "local:docker", "run_id": "cgq0br8q56tm8i4dn51g"}
Apr 10 12:53:30.480501  INFO    container exited    {"runner": "local:docker", "run_id": "cgq0br8q56tm8i4dn51g", "id": "c9d63a26891f537483414c7bf85c9ce6bdc7f492f3005185145462bbdac06cf5", "group": "seeds", "group_index": 0, "status": 0}
Apr 10 12:53:30.505709  INFO    container exited    {"runner": "local:docker", "run_id": "cgq0br8q56tm8i4dn51g", "id": "2df74536f9e2c21c8cda2307feceae2561fb4fcb14d6cb83a28e7e9c999f66ee", "group": "validators", "group_index": 2, "status": 0}
Apr 10 12:53:30.602151  INFO    container exited    {"runner": "local:docker", "run_id": "cgq0br8q56tm8i4dn51g", "id": "cf1fee5902e35a91c0c6dcac123ccd4eac57e3f7aa020aabfa035a9fa64ccdd0", "group": "validators", "group_index": 0, "status": 0}
Apr 10 12:53:30.663706  INFO    container exited    {"runner": "local:docker", "run_id": "cgq0br8q56tm8i4dn51g", "id": "2caabbe0beff898acae05ea83d7e31f8d1533ddd0537242839c7752793b2415e", "group": "validators", "group_index": 1, "status": 0}
Apr 10 12:53:30.663770  INFO    all containers are complete {"runner": "local:docker", "run_id": "cgq0br8q56tm8i4dn51g"}
Apr 10 12:53:30.663789  INFO    deleting containers {"runner": "local:docker", "run_id": "cgq0br8q56tm8i4dn51g", "ids": ["cf1fee5902e35a91c0c6dcac123ccd4eac57e3f7aa020aabfa035a9fa64ccdd0", "2caabbe0beff898acae05ea83d7e31f8d1533ddd0537242839c7752793b2415e", "2df74536f9e2c21c8cda2307feceae2561fb4fcb14d6cb83a28e7e9c999f66ee", "c9d63a26891f537483414c7bf85c9ce6bdc7f492f3005185145462bbdac06cf5"]}
panic: send on closed channel

goroutine 87 [running]:
github.com/testground/sdk-go/sync.(*DefaultClient).responsesWorker(0xc000906620)
    /home/evan/go/pkg/mod/github.com/testground/sdk-go@v0.3.1-0.20220525111316-b6b10897b578/sync/client_conn.go:43 +0x285
created by github.com/testground/sdk-go/sync.newClient
    /home/evan/go/pkg/mod/github.com/testground/sdk-go@v0.3.1-0.20220525111316-b6b10897b578/sync/client.go:118 +0x1ea

in the testground daemon. It appears that the tests are running fine, but testground still panics from what looks like some goroutine mismanagement.

yihuang commented 3 months ago

I have similar error when playing with a trivial testground test plan in our own project.