Open liubog2008 opened 1 year ago
Hey there @liubog2008 - I think I understand what you're trying to do and why. I don't currently have a strong answer for you. It seems that a common use-case in Kubernetes is:
(Synchronized)BeforeSuite
to do some very expensive set up.BeforeEach
) amount of shared set up.Ginkgo doesn't have a strong answer for this matrix of use-cases... yet. It's currently designed around the notion that:
(Synchronized)BeforeSuite
is very expensive.Over the years we've gradually added some additional options:
Serial
decorator allows you to specify that a given spec cannot run in parallel with any other spec.Ordered
decorator allows you to specify that a group of specs have a shared setup (BeforeAll
) and must run on the same parallel process (so they run in serial with-respect-to each other, but can be in parallel with other specs).I'm considering options to allow for something more general so you can express things like "this group of tests can run in parallel with respect to each other but not with respect to other tests" but I only have some ideas at this point, not a concrete plan or proposal yet.
Until then, fwiw:
Why not call ginkgo multiple times?
I hope to get a summary report.
You can get this with a little bit of code that loads and merges multiple JSON/JUnit reports.
Yes, I'm worked in Kubernetes. The biggest problem is that I have to do something between two subset of tests.
some tests -> do something serial and expensive(e.g. upgrade some apps and the upgrade also should be tested) -> other tests
Thanks for the detail - it's helpful and further paints a picture of just how much flexibility folks seem to need when building these complex e2e suites.
I have a few more questions as I'm actively considering how to design solutions for this kind of problem space:
When the "serial and expensive" thing happens are other (unrelated) tests allowed to run in parallel? For example, say you have a set of tests related to Feature A and a set of tests related to Feature B. Feature A looks like "some tests -> do something serial and expensive -> other tests" and Feature B also looks like "some tests -> do something serial and expensive -> other test". Are both Feature A and Feature B allowed to run in parallel with each other (i.e. on different Ginkgo parallel processes)? Or does "something serial and expensive" really need to be serial across the entire cluster (i.e. Feature B would have to wait before proceeding if Feature A is doing its serial and expensive thing)?
How do you currently run in parallel? Do you have a single cluster and each test has its own namespace and so when running in parallel there is no interference?
Assuming Feature A and Feature B can run at the same time then what you are asking for can be accomplished with a single massive It
for each of A and B. Within the It
you could manage some goroutines and aggregate failures yourself. Obviously this isn't elegant and leads to less effective test reporting - but I'd love to understand the issues you see with approaching things that way so I can better understand how to design a more general solution.
Have you actually tried either of these options:
/*
option 1: this runs "some tests" in parallel, then does the upgrade once and runs "other tests" in serial
*/
Describe("Feature A", func() {
BeforeEach(func() {
// set up the application
}, OncePerOrdered)
// some tests
It("tests something", func() { ... })
It("tests something else", func() { ... })
// do something serial and expensive
Describe("When upgraded", Ordered, func() {
BeforeAll(func() {
// the expensive upgrade
})
// other tests
It("tests something after the upgrade", func() { ... })
It("tests another thing after the upgrade", func() { ... })
})
})
/*
option 2: this runs everything in parallel. It's true that the upgrade is very expensive - but with enough parallelism and resources the actual run-time is not too bad.
*/
Describe("Feature A", func() {
BeforeEach(func() {
// set up the application
}, OncePerOrdered)
// some tests
It("tests something", func() { ... })
It("tests something else", func() { ... })
// do something expensive, repeatedly, for each test
Describe("When upgraded", func() {
BeforeEach(func() {
// the expensive upgrade
})
// other tests
It("tests something after the upgrade", func() { ... })
It("tests another thing after the upgrade", func() { ... })
})
})
and evaluated their performance characteristics when running in parallel with ginkgo -p
? I understand that the operation in question is expensive. When I've been faced with situations like this I've either (a) used a single large It
documented with By
s, (b) simply paid the cost of running the upgrade for each spec but running in parallel which amortizes that cost (option 2), (c) used an Ordered
container with a BeforeAll
as shown above (option 1).
For example:
Why not use multiple suites?
Because features maybe too many
Why not call ginkgo multiple times?
I hope to get a summary report.
Why not run once with all features?
BeforeSuite
is too expensive.