Given possible memory usage of these reference datasets,
it might make sense to extract them 1 at the time and store them under the 0-Reference folder
and make sure the local files are re-used for subsequent processes.
2 schenarios:
1) Extract / Read reference files from within cir_processing()
2) Batch validate_initial() and use ou values from meta sheet to extract all reference datasets (orgs & mechs) before proceeding
Note: keep an eye on the memory usage / time to complete task
Given possible memory usage of these reference datasets, it might make sense to extract them 1 at the time and store them under the 0-Reference folder and make sure the local files are re-used for subsequent processes.
2 schenarios:
1) Extract / Read reference files from within cir_processing() 2) Batch validate_initial() and use ou values from meta sheet to extract all reference datasets (orgs & mechs) before proceeding
Note: keep an eye on the memory usage / time to complete task