USAID-OHA-SI / Interesting

Custom Indicator Reporting Resources
Other
1 stars 0 forks source link

cir_processing - leverage datim query functions #9

Open baboyma opened 1 year ago

baboyma commented 1 year ago

Given possible memory usage of these reference datasets, it might make sense to extract them 1 at the time and store them under the 0-Reference folder and make sure the local files are re-used for subsequent processes.

2 schenarios:

1) Extract / Read reference files from within cir_processing() 2) Batch validate_initial() and use ou values from meta sheet to extract all reference datasets (orgs & mechs) before proceeding

Note: keep an eye on the memory usage / time to complete task

baboyma commented 1 year ago

I have added a couple of parameters to this function + logic to extract reference datasets.

Todo: extract once and reuse.