Create integration tests for SPM

[yurishkuro]

In #5603 the bot is trying to upgrade a dependency of the example. We don't actually know if that upgrade is safe or not, it may also require argument changes.

The main issue is that this docker-compose/monitor/docker-compose.yml file is not being testing in the CI. SPM is a major functionality, it's good to have some basic smoke test that it's working, as well as to test that the corresponding docker compose file is valid. An example of such e2e test is scripts/build-all-in-one-image.sh where we validate that the resulting image is correctly serving the web UI.

Proposal for integration test:

• [x] perform a fresh build of Jaeger code (don't load published images), to test the current branch
• [x] bring up services via docker compose
• [x] wait for all services to respond successfully to health checks
• [x] for each service in microsim
  • [x] loop until at least 3 data non-zero data points are retrieved from the REST API
  • [x] or until the overall timeout (say 10min) is reached, in which case fail the test
  • [ ] we need to test that the response also contains the right labels, specifically to catch this issue: https://github.com/jaegertracing/jaeger/pull/5673

Additional improvements:

• [ ] The prometheus config is set to scrape metrics every 15sec, but in the UI the data points come with granularity of 1min. That means the test will have to run for at least 3min before succeeding, an artificial delay. We should find a way to use shorter interval (it may be already supported by the query API)
• [ ] Check not just calls metric, but also errors, at least for services that may have errors
  • [x] This might require improvement to microsim to generate errors. https://github.com/yurishkuro/microsim/pull/16

[EraKin575]

Hi, @yurishkuro ! I am new contributor to this project! I would love to take this up!

[kunal-511]

Hi, I have made some changes according to this issue's requirements, I am ready to make a PR request. @yurishkuro Please assign me.

[yurishkuro]

@kunal-511 Feel free to open pr. We don't assign issues.

[arujjval]

Is this issue still open?

[yurishkuro]

yes

[akstron]

Hi @yurishkuro , can you help with some more details for this:

The prometheus config is set to scrape metrics every 15sec, but in the UI the data points come with granularity of 1min. That means the test will have to run for at least 3min before succeeding, an artificial delay. We should find a way to use shorter interval (it may be already supported by the query API)

The test is configured to wait for 10 seconds if the desired number of metrics is not found, or probably I am missing something.

[yurishkuro]

Code link?

[akstron]

https://github.com/jaegertracing/jaeger/blob/9c1c3ceac3c3c02af0970cabeac8d91aa5b83529/scripts/spm-integration-test.sh#L141

[yurishkuro]

it doesn't wait 10sec, it waits while [ $SECONDS -lt $end_time ]; do

[akstron]

it doesn't wait 10sec, it waits while [ $SECONDS -lt $end_time ]; do

Yes, I understand that. If I understand it correctly, the script calls the /metric api every 10 seconds, but the backend of /metric itself aggregates the metric from prometheus with a granularity of 1 min?

[yurishkuro]

it makes no difference how often the script calls the endpoint, it does not change the shape of the data. The parameters of the query do change the shape https://github.com/jaegertracing/jaeger/blob/9c1c3ceac3c3c02af0970cabeac8d91aa5b83529/scripts/spm-integration-test.sh#L81