precision-sustainable-ag / SemiF-DataReporting

This repository provides detailed reports on Semifield image data, including data contents, species distribution, temporal and spatial distribution, missing data analysis, and status of unprocessed or backlog data.
MIT License
0 stars 0 forks source link

save a separate "unprocessed" batch csv to blob_containers #11

Open mkutu opened 1 month ago

mkutu commented 1 month ago

Let's create a new branch for this

you'll need to read the contents of semifield-uploads and compare the presence of batches in semifield-uploads versus the presences of batches in semifield-developed-images.

this will require a new temp text file to be created, and likely updates to ExportBlobMetrics, or however you see best.

This lets us identify:

  1. unprocessed batches because they're missing data products (metadata, masks)
  2. un-color calibrated batches (they exist in semifield-uploads but not in semifield-developed-images)
  3. unprocessed batches because they're in semifield-uploads and in semifield-developed-images, but don't have the data products (metadata, masks)

This require a new SAS key for that container.

hirunima commented 1 month ago

Analyse semifield-uploads blob container and produce,

  1. un-color calibrated batches (they exist in semifield-uploads but not in semifield-developed-images)

  2. unprocessed batches because they're in semifield-uploads and in semifield-developed-images, but don't have the data products (metadata, masks)

mkutu commented 1 month ago

add column in uncolorized_batches.csv