Event Hubs: Continued Development of Stress Test Suite

[m-redding]

Summary

After the initial onboarding and development of the stress test suite, this issue is intended to highlight areas for further development based on the work that has already been done. For more context about the initial development of these tests review the discussion in #22436. The considerations below are ideas that were discussed during the initial stress test onboarding or thought of during the implementation but could not be implemented in the first phase. More discussion on these ideas and others not on this list are welcome.

Additional Considerations

Additional scenarios

• Multiple consumer roles: right now, the test only accounts for the processing of all partitions through one consumer, based on the index value assigned to each event by the publisher roles. Expand out the test to allow for multiple consumers with potentially changing ownership while still validating event processing.
• Forced detach test to force consumer and processor ownership
• Burst processing

Deploying more role pods and resources

• Explore using dynamic resource creation in combination with Test Resources
• Right now, the tests sometimes struggle to deploy when there are more than 4 or 5 pods being deployed for the same test. This is because all the pods in a given test are attempting to deploy to the same resource group. Kubernetes doesn't provide a straightforward way to have all pods wait for one to start before they start, so another approach would need to be taken here. This could likely be solved with a controller pod.

Fine-Tune Metrics

• Discuss metrics names and determine if alternative names may be more illustrative.
• Explore events and determine if additional events may be helpful
• Explore Application Insights alerting and determine what thresholds may warrant alerting

Framework Matrix

Right now, the tests only target the .NET 6.0 framework. Update the tests to be able to test all the relevant frameworks. This may involve including multiple Dockerfiles.

Extracting common code for core

Much of the stress test functionality will be common across multiple SDK's. In order to facilitate easier integration of other stress tests for other SDK's, extract relevant code into a common shared folder.

Controller Pods

Leverage the Kubernetes API to develop a controller pod that controls scenarios instead of the fixed set of indexed pods. Right now, the test requires the set of Roles needed for a given test scenario to be defined directly in the source code prior to deployment of a test. These sets also need to align with the number of pods that are deployed through the deploy-job.yaml file. The goal would be for the deploy-job.yaml file to deploy a "controller pod" that would dynamically create role pods depending on the configurations passed into the test scenario.

Reference and Resources

• Current .NET implementation
• Getting Started guide
• Richard's work for JS
• Event Hubs metrics discussion
• Event Hubs quick-n-dirty stress harness
• Stress wiki (Microsoft Internal)
• Teams channel for stress (Microsoft Internal)

[github-actions[bot]]

Hi @m-redding, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.