search

USGS-R / drb-inland-salinity-ml

Code repo for Delaware River Basin machine learning models that predict inland salinity.
Creative Commons Zero v1.0 Universal
3 stars 4 forks source link

Add timer for when to refresh AWS credentials #216

Closed jds485 closed 1 year ago

jds485 commented 1 year ago

The targets that refresh AWS credentials take a while to run when all dependency targets have been built. ~20 AWS refreshes in a row takes ~30 mins. I think we should add a timer that is used within each of the AWS refresh targets to indicate if the credentials should be refreshed or not.

jds485 commented 1 year ago

Update: The time is largely spent pulling the dummy_var from S3. Generating the credentials file is quick. As far as I can tell, targets will load all of the dependency targets before executing the command. So, I'm not sure how to speed up these AWS refresh targets. I recommend that in the future we regenerate credentials as part of functions instead of as separate targets.