search

orbs-network / orbs-network-go

Orbs node virtual chain core reference implementation in Go
MIT License
47 stars 12 forks source link

System froze(??) for 1 second in acceptance tests running on CircleCI #1443

Closed vistra closed 4 years ago

vistra commented 4 years ago

https://circleci.com/gh/orbs-network/orbs-network-go/22005#tests/containers/2

There were no log records between 23:13:41.668338 and 23:13:42.695971. It failed the test because getOrderedCommittee hit a timeout, which caused an error to be logged.

Update 12/11 - happened again with the same test https://circleci.com/gh/orbs-network/orbs-network-go/22322#tests/containers/1 ~500ms freeze starting at 23:11:00.834153 No heartbeats were printed to the log during that period

electricmonk commented 4 years ago

another heartbeat mechanism we might employ here is a background job at the OS level that prints current date, running in parallel to the go test process

vistra commented 4 years ago

Maybe also print cpu/memory usage with each heartbeat

On Wed, Nov 13, 2019, 15:50 Shai Yallin notifications@github.com wrote:

another heartbeat mechanism we might employ here is a background job at the OS level that prints current date, running in parallel to the go test process

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/orbs-network/orbs-network-go/issues/1443?email_source=notifications&email_token=AMH4NCKJTMAVR67SFZKGXFDQTQA2HA5CNFSM4JLMEGT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED6GNSI#issuecomment-553412297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMH4NCJ6TYMRIXJMPGSDLPDQTQA2HANCNFSM4JLMEGTQ .

electricmonk commented 4 years ago

it's gonna be a vcpu though, so it's not going to be indicative of anything really