check flag if duplicate urls are found in a sources s3 bucket.

earthcube / scheduler

Scheduling approaches related to gleaner tooling

Apache License 2.0

0 stars 0 forks source link

check flag if duplicate urls are found in a sources s3 bucket. #118

Open valentinedwv opened 1 week ago

valentinedwv commented 1 week ago

Think that is in a report, so read the report when a (TBD) report is generated, and flag an error if there are duplicates or is this just read the bucketutil_urls.csv asset, and see if there are duplicates... or just add a check to the workflows/ingest bucket_urls method to upload duplicates if they are found and raise an error