digitalfabrik / integreat-cms

Simplified content management back end for the Integreat App - a multilingual information platform for newcomers
https://digitalfabrik.github.io/integreat-cms/
Apache License 2.0
56 stars 33 forks source link

Add tests to check response time in API #2204

Open seluianova opened 1 year ago

seluianova commented 1 year ago

Motivation

In addition to #2091, we could also measure the API response time.

We probably don't want to generate a load, so we could just capture the current timestamp before sending the request and after getting the response, as an simplest option.

Questions to think about:

timobrembeck commented 1 year ago

To gain at least a little bit of determinism, I think the CircleCI servers should be used as baseline for execution times. Maybe a first step would be to gather some statistics, e.g. for 20 CircleCI runs, what are the min/max/avg execution times for the API endpoints? I still think there are probably edge cases where the timing will be completely off, e.g. because some task is scheduled later etc...

However, I think this really has a low prio because we don't know if this really adds much benefit in addition to the SQL query counter, whereas the other tickets in https://github.com/digitalfabrik/integreat-cms/issues/765 are immediately very useful.

seluianova commented 1 year ago

The response time may depend on the system

But as an option, we could use these tests not to check performance against some golden numbers, but against the master branch. So we could potentially catch performance degradation.

timobrembeck commented 1 year ago

But as an option, we could use these tests not to check performance against some golden numbers, but against the master branch. So we could potentially catch performance degradation.

Sounds like an interesting idea, could be worth investigating. I'm not completely aware of CircleCI's internal load balancing, it could also be the case that they assign resources to containers quite dynamically, which could make the whole thing less predictable and could cause some test cases to have different execution time even if nothing else changed.

I think a good first step would be to write some code to measure the execution times and then let these tests run a few times with the same code base to see whether the results are somewhat consistent or whether there are high variances.