Benchmark tool

[verdie-g]

Is your feature request related to a problem? Please describe.

In the FAQ, it states

JsonApiDotNetCore scales pretty well under high load and/or large database tables

Unfortunately the linked repository was not updated in the last 3 years and is still targeting .NET Core 3.1 which is 5 years old and reached end of support in 2022.

Additionally, this tool was made to confirm that JADNC has a decent response time when reaching 50 QPS (which is kind of low btw) but it doesn't allow the developer to easily test that a local/recent change improve/worsen the performance.

Finally, the tool runs the JADNC in a docker container which makes profiling challenging and I'm also concerned about the performance impact of running the app in WSL (on Windows).

Describe the solution you'd like

These 3 issues can be easily fix by:

1. moving the repository PerformanceReports in the main repo which would enforce the maintainers to keep the benchmarked project up-to-date and that project could also directly reference (with a ProjectReference) JADNC to test a local/recent change.
2. running the benchmarked project outside of Docker to avoid any overhead from WSL (on Windows) and also easily profile the application using dotnet-trace, DotTrace, perf, ...

Describe alternatives you've considered

The solution proposed will help to easily measure the change of performance but because it doesn't give any internal metrics about the application, it will not help to find bottlenecks easily. One alternative is to use a Grafana stack:

• k6 for load testing
• Mimir as a metrics database
• Grafana for visualization

and the app would push metrics (ASP.NET Core, CLR, etc.) to Mimir so after that a load test is run, one could get some hints about where the bottlenecks are.

[bkoelman]

Thanks for the proposal. It's great to hear you have experience with this. I'm looking forward to what you can come up with.

The PerformanceReports repository was last updated when working on JADNC was our day job. At that time, @wunki was advocating the project, writing blog posts, and measuring performance and scalability. Scalability was a primary goal for the v4 release, because it was terrible at the time. For example, full tables were fetched in memory before filters were applied, and there were memory leaks and race conditions that surfaced only when running with a high load for hours. Many things we did are not publicly visible. I remember we tried to push throughput to the limits on Linux (no docker), watching memory and CPU behind the scenes, but running both the server and the load tests locally (on different computers), we were unable to generate sufficient load to bring down the API. We learned that vegeta was able to produce load magnitudes higher than what modern JavaScript-based frameworks could produce. I don't remember the numbers, but they were surprisingly high. Being open source, we don't have funding to set up cloud agents that can produce loads so high that it takes down the API. The Digital Ocean Droplet support was created by the original author; we weren't able to get it to work anymore, if I remember correctly.

From my experience, the free build agents (AppVeyor, GitHub Actions, Azure DevOps, etc) are very unreliable when it comes to getting stable measurements. So I don't think it's feasible to set something up that verifies performance impact as part of the ci-build, unless we'd use self-hosted runners (which we can't afford).

Since those days, not much has changed in terms of scalability. From time to time, we do take measurements when we anticipate impact. That sometimes involves creating a local project that fires requests concurrently, to BenchmarkDotNet experiments, to using the built-in measurement framework (only stable on Windows) that's part of the example project. Just that it's not publicly visible doesn't mean we never spend time and effort on it. The topic gets visited from time to time, just not continuously. I have a personal list of potential opportunities for improvements, as well as some work-in-progress branches, but it's not in a state to share.

It is my opinion that load tests are best kept in a separate repository, so it doesn't become a daily maintenance burden, but perhaps I'm wrong. It could use a git submodule to reference any version of JADNC. Then, from time to time, we can flexibly run load tests, without having to think about them daily. Most of the time, our changes are not related to throughput and/or scalability.

[verdie-g]

I work in the performance team of a large software company and I thought I could help optimizing, if needed, JADNC. In its current state, the tooling doesn't allow me to do that as long as it's in its own repository.

I feel like the maintenance cost of having one extra JADNC project in this repo is low. Maybe we could use an already existing example project for the benchmark?

[bkoelman]

I would definitely appreciate such help. But I don't see why a git submodule wouldn't work. Alternatively, keeping a local stash around (which changes package references into project references) works well too. I do that for debugging into Swashbuckle sources from time to time. Or a separate local .sln that combines projects from the two repositories. For JADNC, you only need to reference 4 projects (including the sample).

The maintenance cost is not in having one more project in the solution. It's in keeping its dependencies and cibuild up-to-date, fixing bugs, .NET updates, coding style, docker changes, Dependabot, CodeQL, etc. As I said, we only visit the topic once in a while. Tools and frameworks change all the time. So when the need comes up or a contributor comes along, we patch things up or use something else entirely, depending on what makes sense at the time. The same applies to the Ember.js project. I don't want to be held back from developing JADNC because I'm forced to spend time on these things, given how rarely we need them.

Anyway, feel free to just give it a try, and we'll take it from there.

[verdie-g]

I'm concerned that we can't use ProjectReference with a git submodule. I'll git it a try.

Alternatively, keeping a local stash around (which changes package references into project references) works well too

I don't want to be held back from developing JADNC because I'm forced to spend time on these things, given how rarely we need them.

My point is that, if it's not easy to test the performance of a change, and requires manual changes in the code, I fear that it will never be used and the tool would just die.