First of all, thanks for putting this test suite together. It's really helpful and I'm glad I didn't have to write this myself.
I've spent the last couple weeks with running performance tests in various environment and I have to say it was a lot of struggle. Setting up a Harbor instance that actually works was not trivial, the tests kept randomly failing and ruining test environments (eg. prepare steps failing and not being able to rerun, tests randomly failing, etc).
I'm not trying to blame anyone or complain about the test suite, just sharing my overall experience about the path to actually using it.
It really made things easier for me and I hope it can be useful to others. It's far from perfect and the results from the test runs on these environments are not conclusive (ie. they still randomly fail or not work at all).
I plan to make some improvements. For example, replacing the builtin postgres and redis with Aurora and ElastiCache seems like a good idea, as postgres gave up quite a few times during the test runs.
As I said, it's far from perfect, but it ended up sparing me hours in the end.
I'd be happy to contribute it to this repo once it's in a better shape if you are interested.
First of all, thanks for putting this test suite together. It's really helpful and I'm glad I didn't have to write this myself.
I've spent the last couple weeks with running performance tests in various environment and I have to say it was a lot of struggle. Setting up a Harbor instance that actually works was not trivial, the tests kept randomly failing and ruining test environments (eg. prepare steps failing and not being able to rerun, tests randomly failing, etc).
I'm not trying to blame anyone or complain about the test suite, just sharing my overall experience about the path to actually using it.
To help the situation, I started to put together a couple terraform modules to make spinning up test environments easier: https://github.com/sagikazarmark/harbor-perf-test
It really made things easier for me and I hope it can be useful to others. It's far from perfect and the results from the test runs on these environments are not conclusive (ie. they still randomly fail or not work at all).
I plan to make some improvements. For example, replacing the builtin postgres and redis with Aurora and ElastiCache seems like a good idea, as postgres gave up quite a few times during the test runs.
As I said, it's far from perfect, but it ended up sparing me hours in the end.
I'd be happy to contribute it to this repo once it's in a better shape if you are interested.