When exporting data mart data from Pega, we need to use data flows (due to security settings in browsers that block file downloads). This often results in many different zip files. To prep this for PDSTools, we need to merge all the zip (really the json within) to a single file so that ADMDatamart class can operate on it.
In the current file_io there is a readMultiZip function that reads and combines multiple zips. However you need to handle specifying the glob pattern eg:
Once you have these lists of files, you can call the readMultiZip to get the combined file.
Would be nice to have this all available and documented, in both the notebook and quarto examples, where you simply supply the path, and whether it is a mutli-zip file (or multi json).
Is your feature request related to a problem?
No
Describe the solution you'd like
When exporting data mart data from Pega, we need to use data flows (due to security settings in browsers that block file downloads). This often results in many different zip files. To prep this for PDSTools, we need to merge all the zip (really the json within) to a single file so that ADMDatamart class can operate on it.
In the current file_io there is a readMultiZip function that reads and combines multiple zips. However you need to handle specifying the glob pattern eg:
path = "/Users/hamnb/Downloads/ADM2/"
pattern="Data-Decision-ADM-ModelSnapshot_pzModelSnapshots*" model_files = glob.glob(os.path.join(path, pattern))
pattern = "Data-Decision-ADM-PredictorBinningSnapshot*" bin_files = glob.glob(os.path.join(path, pattern))
Once you have these lists of files, you can call the readMultiZip to get the combined file.
Would be nice to have this all available and documented, in both the notebook and quarto examples, where you simply supply the path, and whether it is a mutli-zip file (or multi json).
Describe alternatives you've considered
Wrote the glob pattern in our notebook.
Additional context
No response