Make integration tests independent of sample data

WordPress / openverse

Openverse is a search engine for openly-licensed media. This monorepo includes all application code.

https://openverse.org

MIT License

238 stars 188 forks source link

Make integration tests independent of sample data #747

Open dhruvkb opened 3 years ago

dhruvkb commented 3 years ago

Problem

The API integration tests used by the CI workflow use the sample data. This makes them tightly coupled and causes problems when updating the sample data to match changes in models.

Description

The integration tests should use QA indices search-qa-images and search-qa-audio that are created programmatically.

Alternatives

The test cases need to be updated every time there is a change in the sample data.
The tests should do some computation on the sample data first and then adjust the requests accordingly.

Implementation

[ ] 🙋 I would be interested in implementing this feature.

dhruvkb commented 3 years ago

Can be related with WordPress/openverse-api#181 to make the examples work as test cases.

sarayourfriend commented 2 years ago

@dhruvkb Is https://github.com/WordPress/openverse/issues/725 a duplicate of this issue?

@zackkrida is this still high priority?

zackkrida commented 2 years ago

Priority has been downgraded to low.

obulat commented 6 months ago

While working on #3931, I had to update the sample data to include the new audio provider with a different shape of the alt_files (some fields, such as bit_rate, were absent). I replaced the last 10 items, and the integration tests that, as it turned out, rely on them started failing. Updating sample data CSV is not easy, and it would be nice to create the public data programmatically. Provided that we know all of the data inconsistencies or have normalized all of them already.

I wonder if something like Schemathesis exists for synthetic data creation.