So some of the deployers are a little inconsistent. That is, they sometimes will fail the deploys and/or tests for reasons that don't make sense. I'd love to have this resolved eventually. The good news is that this is not a big deal in real life. If it fails 1/10 times, you just run it again and you're set. Some details on specific deployers:
s3: Occasionally fails after creating the bucket, throws a bucket not found error. The bucket creation succeeds, but the next step that tries to modify the bucket says it was never created. When checking through the interface, it wasn't created.
heroku: Occasionally fails only on travis, throws an error that doesn't give any information about why. It failed locally at one point giving an application error when the site was pinged to see if it was up, but when I checked the actual site, it was up. So I added in a check to make sure the deploy was complete and it now is consistently passing locally, although the check is essentially just wasting time. I think there's a small gap between when heroku says it's deployed and when it actually is, and the automated tests slip in there and end up failing because of it.
So some of the deployers are a little inconsistent. That is, they sometimes will fail the deploys and/or tests for reasons that don't make sense. I'd love to have this resolved eventually. The good news is that this is not a big deal in real life. If it fails 1/10 times, you just run it again and you're set. Some details on specific deployers: