epic: testing - planning

apollographql / federation-next

Moved to the Router repository

https://github.com/apollographql/router/tree/dev/apollo-federation

Other

19 stars 1 forks source link

epic: testing - planning #24

Closed jeffjakub closed 9 months ago

jeffjakub commented 11 months ago

As an engineer, I would like to:

set up unit, integration tests, performance tests

So that we have quality metrics established at the start of the project to establish a baseline.

Acceptance Criteria

[x] unit tests and integration tests should be written as part of standard development
[ ] migrating existing jest tests, should be done as part of development process
[ ] performance test (query-planner) (get a random sample from test harness, establish a baseline) (M)
[ ] performance tests (router) (@jeff to follow up w/ Polaris team)
[ ] harness tests (we need to revisit the test harness retro tasks) (M)
[ ] additional integration test for wasm support (needed for Studio where fetch plan is visually represented) (@jeff to follow up w/ Pulsar team)
[ ] backwards compatibility tests - runs fed2x and fed-next, compares output to validate equivalence. (tool, take dataset from perf tests, CI) (M)
[ ] setup test fuzzing for unit testing (S)

dariuszkuc commented 11 months ago

[x] unit test setup - project is already setup to run unit tests (and there are some tests already)
[ ] integration tests - we might want to integrate some parts of apollo-smith to generate random data sets to verify we can compose supergraphs and generate query plans
[ ] performance tests - we need to create some well defined representative data set that we could run performance tests against
[ ] harness tests - we should have a mechanism to run federation JS and RUST versions against corpus of customer data

sachindshinde commented 11 months ago

Regarding unit tests, I would ask that we strive to not drop existing unit tests in the federation repo when moving to Rust. (Or at least, I've seen a few rewrites in the past where folks say they'll port the tests over later, and later never comes.)

Regarding performance tests, we should at least take performance samples from customers who have complained about performance issues or are expecting performance improvements (not sure if we have a list of these) along with test harness graphs that have been habitually slow (we currently only collect a little data here, but we could start collecting more).

Regarding the test harness, it shouldn't be too painful to start running both JS and Rust versions of code, assuming we allow compiling to WASM. (We could also do without WASM, but it would be a bit more work.)

jeffjakub commented 11 months ago

Yes, let's leverage @clenfest's recent work for the worker-server as much as possible.

dariuszkuc commented 9 months ago

We have a pretty good testing strategy figured out. Closing this in favor of more specific items.