Add training cluster monitoring, that automatically restart cluster if training doesn't progress.

[abdallag]

Can you please merge from main to allow the updated checks to pass?