search

fasterci / rules_gitops

Apache License 2.0
17 stars 6 forks source link

race condition in a sidecar #7

Closed apesternikov closed 6 months ago

apesternikov commented 1 year ago

@harjsing7 reported a race condition in a sidecar on pod removal.

2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02 panic: close of closed channel
2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02 
2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02 goroutine 108 [running]:
2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02 github.com/adobe/rules_gitops/testing/it_sidecar/stern.(*Tail).Start.func2()
2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02   external/com_adobe_rules_gitops/testing/it_sidecar/stern/tail.go:86 +0x45
2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02 created by github.com/adobe/rules_gitops/testing/it_sidecar/stern.(*Tail).Start
2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02   external/com_adobe_rules_gitops/testing/it_sidecar/stern/tail.go:84 +0x12b
2023-03-22T05:44:02Z | INFO | 2023/03/22 05:44:02 exit status 2

The closing of the channel can happen in two spots: https://github.com/adobe/rules_gitops/blob/eafc9c3e0eac03f9487abb22e3b735c47ff1a043/testing/it_sidecar/stern/tail.go#L86 https://github.com/adobe/rules_gitops/blob/eafc9c3e0eac03f9487abb22e3b735c47ff1a043/testing/it_sidecar/stern/main.go#L56

We need to handle this situation properly, using cancellable context would be appropriate here Also, we should add a warning in a log on pod deletion, since it is typically an indication of a problem in the test - like pod crashlooping or even early test failure (configurable)