google / weather-tools

Tools to make weather data accessible and useful.
https://weather-tools.readthedocs.io/
Apache License 2.0
203 stars 38 forks source link

Added error handling in license deployment. #417

Closed aniketsinghrawat closed 8 months ago

aniketsinghrawat commented 8 months ago

Currently when there are two cases of failure in dl-v2 license deployment case 1: There is an issue with partition, it is marked as failed and further partitions are continued to be downloaded. If further partitions are also faulty, it can lead to our license being blocked for wrong requests. case 2: The license has expired. In this case the expired license keeps picking up partitions and mark all the partitions as failure (even though they were correct)

This PR aims to handle these cases. For case 1, it successively more than 10 partitions have failed, that config is removed entirely from that license. Also next request is done at an exponential back off rate. For case 2, if we ever get a license expired error, the whole queue of that license is emptied out.