Include downloading of raw files in the integration test

BlueBrain / Search

Blue Brain text mining toolbox for semantic search and structured information extraction

https://blue-brain-search.readthedocs.io

GNU Lesser General Public License v3.0

40 stars 10 forks source link

Include downloading of raw files in the integration test #532

Open jankrepl opened 2 years ago

jankrepl commented 2 years ago

🚀 Feature

Recently, we implemented bbs_database download for multiple different sources. It might be a good idea to extend our integration test to actually use this download (rather than using data committed to the repo).

Motivation

We make sure that our ETL pipeline works end to end
Possibility to detect upstream changes on the source servers

Stannislav commented 2 years ago

We might consider only running this sparsely, e.g. only on merges to master or according to some schedule?