AllenInstitute / datacube

Other
0 stars 1 forks source link

implement exponential backoff in `loop.sh` and `run.sh` scripts #81

Open chrisbarber opened 6 years ago

chrisbarber commented 6 years ago

E.g. https://github.com/AllenInstitute/datacube/blob/167932d6a86eb93ba0a54edf441f52f2b1113f66/services/pandas/run.sh#L5

If a service is failing immediately, this fast restart time will pollute the logs. However, we want a fast restart initially so a production outage is as short as possible. These scripts could save some state regarding when the last crash occurred.

It would make sense to rearrange things so that the services share the same loop script, since at this point they would be non-trivial.