Add number of extracted, loaded, and failed resources in batch responses

arkhn / fhir-river

Live ETL pipeline to standardize Health Data into FHIR.

Apache License 2.0

42 stars 4 forks source link

Add number of extracted, loaded, and failed resources in batch responses #693

Closed Jasopaum closed 2 years ago

Jasopaum commented 2 years ago

TODO: front

Fixes

Fixes #668 by @MiskoG

Description

Back: add a progressions field to Batch
Front: display this information in the "ETL view"

Technical details

If the batch is running, the progression is fetched from redis. Otherwise, it's persisted and fetched from the DB

Definition of Done

[ ] I followed the Arkhn Code Book (I swear!).
[ ] I have written tests for the code I added or updated.
[ ] I have updated the documentation according to my changes.
[ ] I have updated the deployment configuration if needed.

tevariou commented 2 years ago

IMO it's best to use a dedicated table for Progression. having a jsonfield with a custom structure in the Batch table doesn't make sense in a relational db, especially if we want to make queries involving a progression counter by resource_id. @tevariou what's your opinion ?

I agree it may be more practical to have a dedicated table for filtering

Jasopaum commented 2 years ago

Also, @tevariou, @simonvadee, do you know if there is an easy way to know the number of resources that have been updated compared to inserted?

simonvadee commented 2 years ago

do you know if there is an easy way to know the number of resources that have been updated compared to inserted?

good question: there isn't. In loader, we use dao.update() whether we want to insert or update a resource, and we always increment the loaded_counters. We could make a distinction between inserted and updated depending on the return value of dao.update() maybe ? I encourage you to create an issue.