terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

ts-scripts test runner deletes cached docker image when it is needed for a second test suite #3811

Closed busma13 closed 2 weeks ago

busma13 commented 3 weeks ago

When we added caching of docker images we did not consider the case of elasticsearch-assets. This workflow runs tests on the asset as well as the api from a single job in the workflow. Each workflow job downloads docker images from cache. The test runner for asset tests then loads the docker image and deletes it from the temporary cache folder. When the test runner for api tests tries to load the docker image, it has been deleted, so it must pull the docker image. If we run too many tests we exceed our docker pull rate limit.

My first thought at a fix is to have a flag that we can pass to the test-runner, telling it whether or not it should delete the docker image after load. This would default to true. Then we just need to update the elasticsearch-assets test workflow in one spot, passing in the flag set to false.