Run load generation during canary

[ju187]

Summary

Canary deployment needs enough traffic to feed the analysis quires, i.e prometheus. The load generation should be coordinated with the canary so it last the entire progression.

Use Cases

In non production environment, a loader generator is needed to generate traffic for analysis run. There are many ways for this but ideally the load generation should be triggered by the rollout to have better coordination. I notice flagger has a webhook for load testing (https://docs.flagger.app/usage/webhooks), a feature similar to that will be very helpful

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[zachaller]

So I am working on a feature that is not quite load generation but might help anyways. The feature is traffic mirroring with the ability to apply filters so that you only mirror say read only traffic to the canary. This might be something that you could use in the interim to help you out.

I also think that a more generic feature for rollouts could be a step that has the ability to call an argo workflow template. If this feature got implemented it would allow you to easy do load testing and many other things during a rollout.

[ju187]

Thanks for the reply. Not sure the traffic mirroring is helpful as in out test environment, there is lack of any kind of traffic. The second certainly helps, but it requires Argo workflow. I am thinking using a job template to run an image to generate some traffic, the downside of that hard to coordinate with the overall canary steps. Basically I need to have a long running process that stops when canary reach 100%

[zachaller]

As a work around using features today you could do something with the notification engine to a webhook https://argoproj.github.io/argo-rollouts/generated/notification-services/webhook/ and more general https://argoproj.github.io/argo-rollouts/features/notifications/

[MarkSRobinson]

@ju187 You could use the Job analysis type - https://argoproj.github.io/argo-rollouts/analysis/job/. You could write a load generate with k6 - k6.io - and package it up as a image to run against the system.

[anup1384]

Hi @zachaller @ju187 @MarkSRobinson ,

I'm also looking for the same feature which generates load on canary pods before handling traffic to warmup application pods. Any update?

@ju187 Do you have any solutions?

[zachaller]

@anup1384 an Job analysis is a perfect use for this we even have a small example of it being used https://github.com/argoproj/rollouts-demo/blob/master/examples/preview-testing/preview-testing.yaml#L56-L61

[anup1384]

Thanks, @zachaller is it applicable for all new pods? and how to handle during HPA(Horizontal pod autoscaling)

[zachaller]

You would use the service and or ingress to access the pods so all new pods would all be included

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity.

[hpvd]

for load generation (and latency testing...) beside K6 this one looks really interesting: https://github.com/ddosify/ddosify

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity.