kubeshop / testkube

☸️ Kubernetes-native testing framework for test execution and orchestration
https://testkube.io
Other
1.32k stars 129 forks source link

Feature Request: K6 parallel parameter in options #4952

Open Cliftonz opened 7 months ago

Cliftonz commented 7 months ago

Is your feature request related to a problem? Please describe. When I want to scale my k6 performance tests within Testkube to simulate high traffic scenarios efficiently. Currently, from me messing around in Testkube, it doesn't allow me to specify the level of parallelism directly for k6 tests, limiting my ability to mimic real-world heavy load conditions accurately.

This limitation hampers my ability to conduct thorough performance testing and optimize my applications for high-demand scenarios.

Describe the solution you'd like I would like Testkube to support an option where I can define the number of parallel executions for a k6 test directly. This feature should allow users to specify the desired level of parallelism in the test definition or via the CLI when triggering a test. For instance, if I want to run a test with 100 VUs (Virtual Users), I should be able to split this into 10 parallel executions with 10 VUs each to distribute the load and simulate more realistic traffic patterns across multiple nodes or pods in a Kubernetes environment. This approach would improve the scalability of performance testing and provide more flexibility in simulating various load conditions.

This feature already exists in the k6 operator resource definition and can easily be hooked into.

CleanShot 2024-01-28 at 20 12 21

Note in the above spec here I believe these two options (Parallelism & Script) should be able to be set for load tests.

Parallelism: How many instances of k6 you want to create. Each instance will be assigned an equal execution segment. For instance, if your test script is configured to run 200 VUs and parallelism is set to 4, as in the example above, the operator will create four k6 jobs, each running 50 VUs to achieve the desired VU count.

Separate: Toggles whether the jobs created need to be distributed across different nodes. This is useful if you're running a test with a really high VU count and want to make sure the resources of each node won't become a bottleneck. Which is directly from here: https://github.com/grafana/k6-operator#executing-tests

Describe alternatives you've considered An alternative solution I've considered is manually segmenting the test into multiple smaller tests with fewer VUs and triggering them simultaneously via separate Testkube executions however this is a pro feature to combine multiple tests.

I am willing to put in a pull request for this however I would need some guidance on where I would update this through the system.

vsukhin commented 7 months ago

thank you @Cliftonz sounds like a nice feature. for @TheBrunoLopes to proritise

Cliftonz commented 7 months ago

@vsukhin @TheBrunoLopes Is there anything I can do to help or speed this up?

Cliftonz commented 7 months ago

@vsukhin @TheBrunoLopes Just following up on this?

vsukhin commented 5 months ago

@Cliftonz we will add it to our new workflow feature, but not sure, if it's planned to be available in OSS or Pro only @jmorante-ks @TheBrunoLopes

TheBrunoLopes commented 5 months ago

Hey there, happy to announce that we are close in delivering this functionality. It will fall under the Testkube Community License (TCL) and will be available for Testkube Pro users in the next few weeks.

Cliftonz commented 5 months ago

Amazing, I am glade that you guys decided to add this to the OS side too.

TheBrunoLopes commented 5 months ago

Hey @Cliftonz, just wanted to clarify that the code under the TCL license while open, it requires a license or the installation of Testkube Pro, its free tier allows you to use it with 1 environment and 3 users

Cliftonz commented 5 months ago

@TheBrunoLopes So if someone uses it with just 1 environment on the free tier then they can still leverage the parallelism, correct?

TheBrunoLopes commented 5 months ago

If your company only has 1 Testkube Pro installation and 3 users or less. Yes

Cliftonz commented 5 months ago

I do not agree with that, if it was over multiple clusters or in multiple clusters over different regions (very advanced case) I would absolutely tell you to charge for that and could easily understand why.

But just for adding a number on the operator on the same cluster does not feel like something that should be a pro feature.

Your essentially saying hey pay us money so you do not have to create a node pool with large nodes to run your k6 tests on. That way if you pay us money you can leverage smaller nodes that can better replicate what you have in production.

Your charging for a feature that is already open source and can be replicated with a relatively small amount of work.

I like what you guys are doing, but I do not feel like this is the right direction for this feature. I think it would be better to charge for something on top of this feature rather then the feature itself.

@TheBrunoLopes @vsukhin

TheBrunoLopes commented 5 months ago

@Cliftonz maybe I expressed myself incorrectly or I understood this last message wrong.

But just for adding a number on the operator on the same cluster does not feel like something that should be a pro feature.

Testkube doesn't care about the size of the cluster (how many nodes), it can get really big. 1 Testkube Installation is per cluster and not per node. If it's just one cluster, you can use it for free.

Cliftonz commented 5 months ago

That sounds quite fair, thank you for clearing that up!

@TheBrunoLopes