Retry batch tasks on spot preemption

We've had several jobs fail due to spot VM preemption. Batch has built-in tools to retry preemptions: https://cloud.google.com/batch/docs/automate-task-retries#retry-some-failures

eg:

"taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "script": {
              "text": "sleep 30"
            }
          }
        ],
        "maxRetryCount": 3,
        "lifecyclePolicies": [
          {
             "action": "RETRY_TASK",
             "actionCondition": {
               "exitCodes": [50001]
            }
          }
        ]
      }
    }
  ],

dchaley / deepcell-imaging

Retry batch tasks on spot preemption #212