Automatically test the performance of each CSS commit

[wvdemeer]

Pitch

Changes to CSS can have an impact on the memory, CPU and even bandwidth usage of the CSS. It would be nice to know this impact for each commit to the CSS repo, in order to spot performance impact as soon as possible.

Desired solution

For each commit to the CSS repo, a github push webhook will notify a solidlab server. This server will run one or more "performance tests". These tests should evaluate realistic usage scenarios. When the tests are done, the results are reported back to the CSS repo as commit statuses. These commit status contain link to a full test report. Typically this status will be "success", but if there is a (significant) negative performance impact, the status should be "failed".

Acceptance criteria

For each commit to the CSS repo, a commit status that links to a performance report should pop up.

[jeswr]

Related discussion for upstream packages https://github.com/RubenVerborgh/AsyncIterator/issues/62 - could possibly be implemented simultaneously?

[pheyvaer]

@RubenVerborgh What are the changes that you would like to see here?

[RubenVerborgh]

@pheyvaer I actually revised this again, and we're good to go.

[pheyvaer]

@RubenVerborgh Ok, great! @wvdemeer Seeing that you are assigned to this, are you working on this at the moment?

[wvdemeer]

Currently I'm caught up in some other stuff, but I'm itching to get back into this soon.

A lot of work was done already, a lot of the sub parts required for this actually already work now. The big todo now, is to make it easily configurable and usable by anyone but me. I know how I'm going to do that, it's now "just a matter of writing the code" ;-)

[github-actions[bot]]

Please provide a status update about this challenge. Every ongoing challenge needs at least one status update every 2 weeks. Thanks!

[pheyvaer]

@wvdemeer Can you provide a small update here of the status of the platform you are working on?

[wvdemeer]

Current status: The testing platform can now run a first decent test, process the data and show the results. It can run multiple of these tests safely, and it can handle most of the minor error cases that can occurs along the way.

Next Steps: Improving and expanding the tests that run, to make sure the results are valid and valuable, and thus that we're testing relevant things. After that, we'll focus on making the entire thing usable by others.

[pheyvaer]

Should we create a separate challenge for the "make the entire thing usable by others"? @RubenVerborgh What do you think?

[RubenVerborgh]

Yeah, let's scope challenges as narrowly as possible!

[pheyvaer]

@wvdemeer Can you move the "make the entire thing usable by others" to a new challenge and update the current one to remove that part? Thanks!

[wvdemeer]

Ok, done in #81 (didn't change this issue, this part of it was not mentioned.)

[github-actions[bot]]

Please provide a status update about this challenge. Every ongoing challenge needs at least one status update every 2 weeks. Thanks!

[wvdemeer]

Current status: The tests that runs are being improved and expanded, including tests of CSS with and without authentication. The aggregation of the data from the test has improved.

Next step: Continue improving and adding tests, making sure the results are valid and relevant CSS development. (+ Debugging and improving the platform along the way.)

[wvdemeer]

Current status: There are some interesting (but preliminary) results, related to authentication and multi-threading.

These results will be investigated and debugged further, and will be discussed further once we're sure enough they show CSS performance and not performance test bugs.

In a meeting this morning it was also mentioned it would be nice to finish this challenge by the end of this year. That should be possible, so that's a good goal! (Just to be clear: Finishing this challenge means a first decent version, not "the ultimate version". Improvements will off course continue to be made after this first version.)

[github-actions[bot]]

Please provide a status update about this challenge. Every ongoing challenge needs at least one status update every 2 weeks. Thanks!

[wvdemeer]

I'm ready to add the "SolidLab Performance Tests" GitHub app to the CommunitySolidServer repo.

This app enables testing.solidlab.be to add one check mark to each CSS commit. (Which for now will always be either "pending" or "success". It won't ever fail at the moment. But it will always link to the performance tests when clicked.).

To set it up, I need to transfer ownership of the app to the CommunitySolidServer organisation (It's the only way to keep the app private to the CSS repo.) I'm pretty sure someone with access will need to "accept" that transfer (@pheyvaer @joachimvh ?). And after that, some more steps are needed to setup the App on the CSS repo.

I can probably do all that myself if someone grant me access to the apps of the organisation. (Apparently, you can just give me that specific access and nothing more.) But that access is not essential, someone else can also set this up, it should be fairly simple: The app needs to be "installed" on the CommunitySolidServer repo. To do that, once the app has been transferred, you need to go to the app's installations page and install the app to the CSS org, and specify to install it only to the CSS repo.

Let me know if I should transfer the github app.

[wvdemeer]

Update: The app has been added to the GitHub repo. For each new CSS commit, a "commit status" (also called "check") is added. The check always links to the test report. For now, it's always "success" as the test does not yet detect "performance degradation".

(A lot of performance testing work remains to be done, but... ) This completes the "Acceptance criteria" of this challenge. :partying_face:

[github-actions[bot]]

Please provide a status update about this challenge. Every ongoing challenge needs at least one status update every 2 weeks. Thanks!

[RubenVerborgh]

Excellent, thanks a bunch @wvdemeer and team.

[pheyvaer]

You find the report for this challenge here.