search

google / fhir-data-pipes

A collection of tools for extracting FHIR resources and analytics services on top of that data.
https://google.github.io/fhir-data-pipes/
Apache License 2.0
151 stars 84 forks source link

Feature Request: Options to rerun pipelines for selective date ranges #1060

Open Charantl opened 4 months ago

Charantl commented 4 months ago

If the pipeline failed in the past few weeks and went unnoticed, we will need an option to rerun the pipeline to fetch the data for specific date ranges.

chandrashekar-s commented 3 months ago

Hi @Charantl, thanks for filing this issue. Based on the current design of fhir-data-pipes, if a pipeline fails then the next pipeline that runs will start picking up records from the point where the previous pipeline was successfully complete (i.e. from the start time of the previous successful run as that would be the cut off time for that run). Based on this design, I think the scenario of running the pipeline for a specific data range might not be needed as there wont be records left out even if the pipeline fails (as they will be picked up in the next run).

Do you still think that this feature is needed? If yes, can you please specify the use case for the same.

Charantl commented 3 months ago

Yes @chandrashekar-s. If data for a specific date(s) is removed due to human error, we need a feature to reload the data for that particular date (range).