combine calculate_blob_metrics export_blob_metrics for newMatrics

precision-sustainable-ag / SemiF-DataReporting

This repository provides detailed reports on Semifield image data, including data contents, species distribution, temporal and spatial distribution, missing data analysis, and status of unprocessed or backlog data.

MIT License

0 stars 0 forks source link

combine calculate_blob_metrics export_blob_metrics for newMatrics #7

Closed mkutu closed 3 months ago

mkutu commented 4 months ago

We want to keep these the creation of the pdf and gathering of the raw blob data into 2 distinct processes so that we can have an easier time writing the formatting code for the PDF.

combine calculate_blob_metrics export_blob_metrics for newMatrics

create a temp text file because of how we use azcopy
output the new version of the csv

mkutu commented 3 months ago

go ahead and uncomment these lines and only save the processed batches. https://github.com/precision-sustainable-ag/SemiF-DataReporting/blob/d0e03b556319f16a16a81666f9031ad75c32288c/src/export_blob_metrics.py#L201

we'll make another csv of "unprocessed" batches that will require access to another container.

mkutu commented 3 months ago

also, save the csv to the blob_container data directory

save_csv_dir = Path(cfg.paths.data_dir, "blob_containers")
save_csv_dir.mkdir(exists_ok=True, parents=True) 
mismatch_statistics.to_csv(Path(save_csv_dir, 'mismatch_statistics_record.csv')), sep=',', encoding='utf-8', index=False, header=True)