search

hyperledger-cacti / cacti

Hyperledger Cacti is a new approach to the blockchain interoperability problem
https://wiki.hyperledger.org/display/cactus
Apache License 2.0
344 stars 286 forks source link

test(connector-corda): fix flaky v4.7 test #1473

Open petermetz opened 3 years ago

petermetz commented 3 years ago

This likely has the same solution as https://github.com/hyperledger/cactus/issues/1469

Describe the bug

We've got a flaky one at: packages/cactus-plugin-ledger-connector-corda/src/test/typescript/integration/deploy-cordapp-jars-to-nodes-v4.7.test.ts

One of the corda nodes within the container occasionally don't boot up (hangs and then the test waits for an hour before giving up).

To Reproduce

Keep running the test and eventually you'll see it.

Expected behavior

Tests are stable, not flaky.

Logs/Stack traces

The CI logs from the GHA runner as of this morning: logs_10174.zip

Cloud provider or hardware configuration:

GHA runner

Operating system name, version, build:

Ubuntu 20.04

Hyperledger Cactus release version or commit (git rev-parse --short HEAD):

1.0.0-rc.1

Hyperledger Cactus Plugins/Connectors Used

Corda

Additional context

I'll send a separate PR that skips this test to avoid flakiness in the meantime.

jagpreetsinghsasan commented 2 years ago

Fortunately I didn't face it for 3 runs.

petermetz commented 2 years ago

@jagpreetsinghsasan Yeah that's what makes it especially hard to debug, that it works around 9 times out of 10. My best idea so far to make it easier to reproduce is to simulate the constraint hardware resources of the CI environment with a custom dev container for which I have a branch that I've been carrying around (e.g. rebasing onto upstream/main) for just the purpose of trying to reproduce CI-only flaky test issues:

https://github.com/hyperledger/cactus/compare/main...petermetz:devcontainer-json-resource-limits-stash