search

AquaSat / AquaMatch_harmonize_WQP

https://aquasat.github.io/AquaMatch_harmonize_WQP/
MIT License
1 stars 3 forks source link

add architecture to store drive ids csv of harmonized files #105

Closed steeleb closed 1 month ago

steeleb commented 1 month ago

Just a quick add of architecture to store the drive ids resulting from the harmonization pipeline for siteSR. Closes #93

Note I removed a number of cue = tar_cue("always") in chla and sdd harmonization scripts - I couldn't think of a reason why these would always need to be run and it was causing an error when the pipeline would go to update the p3_PARAM_site_info_drive_file since the pipeline was oddly trying to update the Drive file, even though, by logic, this shouldn't be needed. Maybe this is something strange on my end?

I haven't run this with the admin_update config, as I was concerned I might sully things up that are in the review queue.

mbrousil commented 1 month ago

Thanks! Yeah, I added those recently because when I updated the run date the pipeline wasn't recognizing that files needed to be updated from Drive even though they had changed. I know that this is a quirk of how targets checks for things being outdated. Better to not have the "always" though

steeleb commented 1 month ago

Heard! Yeah, I've noticed a few bugs in {targets} lately where it hasn't recognized outdated targets and I've had to do some deep tar_invalidate() across a bunch of targets to get things to update. I should pay better attention to when that happens, but usually it's something so banal that I just figure it's something dumb I'm doing.

Merging!