.ci/aws: Jenkins CI Stability Fixes

[a-szegel]

• Always make kill_all_clusters() succeed
  • There is a race that currently exists where cleanup is started by an aborted job, and when the new job comes in, and runs kill_all_clusters(), it fails b/c the previous cleanup finishes half way through the new cleanup. By making this script always succeed, we will continue executing the pipeline, which is the desired behavior in this situation.
• Wait until ODCR has enough capacity to run jobs before we attempt to launch
  • The p3dn ODCR is not immediately refilling the available instance count after all the instances have been terminated. This has put a race condition in our code which sometimes causes us to get ICE'ed. Attempt to fix the ICE by waiting till the ODCR has the required capacity before attempting to launch instances with it.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[a-szegel]

bot:aws:retest

[a-szegel]

docs to why I think the || true will work.

[a-szegel]

bot:aws:retest

[a-szegel]

bot:aws:retest

[a-szegel]

bot:aws:retest

[a-szegel]

I have been purposefully re-triggering CI to test its stability, it looks like it has improved significantly