As a developer, I want to run the various stages of the Pipeline frequently and without waiting very long, so that I can test the code well. Currently, this is hampered because the URL the refresh stage of the Pipeline uses to get the list of datasets is hard coded, and this means if you want to send just a few datasets through the Pipeline when developing/testing locally, you have to modify src/library/refresher.py so that it doesn't process the full list of datasets.
Acceptance criteria
If this URL was an environment variable, it would be easy to substitute with a local mocked service containing appropriate sets of test data.
As a developer, I want to run the various stages of the Pipeline frequently and without waiting very long, so that I can test the code well. Currently, this is hampered because the URL the refresh stage of the Pipeline uses to get the list of datasets is hard coded, and this means if you want to send just a few datasets through the Pipeline when developing/testing locally, you have to modify
src/library/refresher.pyso that it doesn't process the full list of datasets.Acceptance criteria If this URL was an environment variable, it would be easy to substitute with a local mocked service containing appropriate sets of test data.