Race condition in `make test-docker`

natebrennand commented 2 years ago

Issue Summary

When running make test-docker locally, the first test executed by Go, TestGet, failed reliably.

When inspecting the logs from docker-compose, the Prism startup output was emitted long after the go tests executed and failed. This makes me suspect that what I'm encountering is a startup issue where the Prism container isn't ready to accept requests before the go tests begin. Mitigation suggestions are here: https://docs.docker.com/compose/startup-order/

Steps to Reproduce

Run make test-docker
Run docker-compose logs -f from examples/prism/ to see log output

Exception/Log

I added a log to better inspect the error occurring:

diff --git a/examples/go-client/helper/rest/api/v2010/api_test.go b/examples/go-client/helper/rest/api/v2010/api_test.go
index e75d78d..e56d02c 100644
--- a/examples/go-client/helper/rest/api/v2010/api_test.go
+++ b/examples/go-client/helper/rest/api/v2010/api_test.go
@@ -28,6 +28,7 @@ func TestMain(m *testing.M) {

 func TestGet(t *testing.T) {
        resp, err := testApiService.PageCredentialAws(nil, "", "")
+       t.Logf("err: %s", err)

Which then yielded this error: prism-go-client-test-1 | {"Time":"2022-04-18T18:49:46.703409863Z","Action":"output","Package":"go-client/helper/rest/api/v2010","Test":"TestGet","Output":" api_test.go:31: err: Get \"http://prism_twilio:4010/v1/Credentials/AWS\": dial tcp 172.24.0.2:4010: connect: connection refused\n"}

Technical details:

twilio-go version: v0.24.0 (present main)
go version: 1.18

childish-sambino commented 2 years ago

Hrm, I understand why this could be a problem, but I'm not seeing it locally. We also run these tests daily and not seeing any issues: https://github.com/twilio/twilio-oai-generator/blob/11aa70dd14b66c96ca9aa2fc573696e10f6e1f9c/.github/workflows/test.yml#L38

Is there some other piece that I'm missing which is causing the failure for you?

natebrennand commented 2 years ago

Is there some other piece that I'm missing which is causing the failure for you?

My best guess is my local environment contributes to it? I've got a souped up M1 Max and 1gb fiber.

AIUI, this failure only occurs if examples/go-client/Dockerfile can execute those steps and start the tests before the prism container is ready to start requests. The go-client test execution might never win the race in an environment with some combination of:

slower network access that rate limits the installation of the dependencies
limited compute resources to compile the Go libraries prior to executing tests

In this past execution, you can see that this race was narrowly lost, Prism starts up ~6s before the tests start executing: https://github.com/twilio/twilio-oai-generator/runs/5561494883?check_suite_focus=true#step:6:10273 (I had to link to an older build because these logs aren't included in recent runs after the move to daemon mode for docker-compose)

childish-sambino commented 2 years ago

Makes sense, will take a look at the PR.

twilio / twilio-oai-generator