search

onsi / ginkgo

A Modern Testing Framework for Go
http://onsi.github.io/ginkgo/
MIT License
8.28k stars 654 forks source link

Run a single suite with N instances in parallel with different `BeforeSuite` Setup. #1385

Closed anveshreddy18 closed 5 months ago

anveshreddy18 commented 5 months ago

Hello, firstly, thanks for the wonderful framework.

I would like to know if I can run the same test suite with say 3-4 instances running in parallel where the only difference between them would be the BeforeSuite setup.

This is coming from Kubernetes related integration tests requirement where I want to run the test suite in parallel in multiple namespaces with different cloud providers in each of them, so the required BeforeSuite setup is different but the specs are all the same. Is there a way to achieve this? I'm new to Ginkgo and researched fairly about this but I'm afraid I didn't find much info related to running instances of same spec in parallel.

Can I know if this is possible? Can someone explain in detail.

Thanks for your time :)

anveshreddy18 commented 5 months ago

@onsi can you PTAL at this once. Thanks :)

onsi commented 5 months ago

heyo - sorry for the delay. I've been out of pocket for a couple of weeks.

Ginkgo doesn't directly support this usecase. I can imagine some workarounds but nothing that will be super clean. I would, personally, approach this by making the cloud provider configurable (e.g. via a command line argument or an environment variable) and then invoke ginkgo multiple times for each cloud provider (e.g. in series). This gives you a building block that you can then leverage in CI. For example, back in the day the Cloud Foundry test suite could run against a variety of cloud providers. A single Ginkgo suite could target any of these given different configuration inputs. We would run them in parallel in CI - using the CI system's built in capacity for parallel builds to nicely sequester and monitor the different ginkgo suites.

This is, honestly, a bit cleaner as you end up with logs and reports that are cleanly sequestered by cloud provider. It also makes it a bit easier to write cloud-provider-specific specs.

Could something like this be added to Ginkgo? No doubt. But it's not something that's currently on the roadmap. I'd suggest going down the CI route.

anveshreddy18 commented 5 months ago

Thanks @onsi for taking the time to comment. Yes it does seem like more approachable and cleaner way to run them in parallel in CI. Thanks for the advice :)