chore: improving e2e testing latency

numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

https://numaflow.numaproj.io

Apache License 2.0

1.01k stars 98 forks source link

chore: improving e2e testing latency #1760

Closed samhith-kakarla closed 1 week ago

samhith-kakarla commented 3 weeks ago

creating the pr for testing of ci do not merge right now

codecov[bot] commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 57.23%. Comparing base (c70540b) to head (d60988c).

:exclamation: Current head d60988c differs from pull request most recent head a8588a8

Please upload reports for the commit a8588a8 to get more accurate results.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1760 +/- ## ========================================== - Coverage 60.94% 57.23% -3.71% ========================================== Files 232 218 -14 Lines 20014 17581 -2433 ========================================== - Hits 12197 10063 -2134 + Misses 6959 6664 -295 + Partials 858 854 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

whynowy commented 3 weeks ago

@samhith-kakarla - you might need to look into the original PR, we want to revert some changes over there:

We do not want to start kafka, nats every time when we run an E2E test, we only want to do that when a Kafka or nats e2e test is running. This means, we still want to use go:generate to do the installation.
With this, it comes a problem for the test api, that it initializes the connection (e.g. Kafka client) when the test api service starts, which requires the kafka/nats services running. To solve the problem, move the client initialization to lazy load mode with a lock.

kohlisid commented 1 week ago

@samhith-kakarla Please add the latency improvement stats that you collected here

kohlisid commented 1 week ago

@samhith-kakarla There are permission issues with the doc you added, can you directly add the raw data here. Might be easier

samhith-kakarla commented 1 week ago

ci runtimes before and after improvemnts:

kohlisid commented 1 week ago

@samhith-kakarla As discussed, Could you add the verification result for the local testing for single test function with the updated go generate to block extra resources for those cases as well.

samhith-kakarla commented 1 week ago

Local Testing results for single test function.

here is the result of the grep command run on the test name:

Here is the results of running make TestKafkaSourceSink:

it finds the right test file, deploys the correct resources and passes.