Add automated nightly micro and macro benchmarks

[bwplotka]

In my book, I mentioned amazing work Vitess has done. They have nightly run of unit and macro tests on some cloud provider. It would be amazing if we have the same so we had history of past efficiency across at least set of commits to go in every day.

Prometheus have something like this using https://github.com/prometheus/test-infra

Ideally it is per PR, but that is expensive and it would take too long for PR to get ready to go.

Potentially a GSoC/LFX project?

AC:

• We run nightly the micro and macro benchmark of day before version of Thanos main and today's version.
  • TODO: Macro benchmark is to do as well. For microbenchmarks we could use what we have in our tests already.
• Find sponsor for compute power for this (:
• Optional feature - mark certain commit for ad-hoc test

[squat]

Nice! That's a really good idea. I wonder if maybe we could get additional funding from Civo for this

[matej-g]

Nightly would be awesome, I feel like even doing this between each new tagged release would give us a lot of value.

Tagging @PhilipGough who recently also spent a lot of time doing macrobenchmarks with our receivers.

[epompeii]

Would you also be interested in running it against each PR like Vitess?

If so, I've been working on a continuous benchmarking tool called Bencher: https://github.com/bencherdev/bencher It should check all the same boxes as Vitess's setup on the tracking and performance regression side of things. We could also track nightly and release runs. Bencher doesn't (yet) have a way to handle the compute side of things though, so that would still need to be configured separately.