search

pantheon-deprecated / terminus-deprecated

Deprecated CLI for Pantheon. Do not install. Current project:
https://github.com/pantheon-systems/cli
82 stars 24 forks source link

Catch/Retry failed API Calls - fail quickly if not possible #124

Closed iamEAP closed 10 years ago

iamEAP commented 10 years ago

Running into this intermittently on Travis builds that invoke terminus commands.

drush psite-ecreate $PUUID $PSITE --source=$PSOURCE
Creating the ci-3rj7f-89 environment..... environment created.
Cloning files from test....... files cloned.
Cloning database from test........[...]...CONNECTION ERROR: Operation timed out after 30001 milliseconds with 0[error]
bytes received
No response for site jobs!                                           [error]
Invalid argument supplied for foreach() terminus.drush.inc:1540      [warning]

Followed by the following in a never-ending loop:

.curl_setopt_array(): 84 is not a valid cURL handle resource          [warning]
terminus.drush.inc:3543
curl_exec(): 84 is not a valid cURL handle resource                  [warning]
terminus.drush.inc:3549
curl_errno(): 84 is not a valid cURL handle resource                 [warning]
terminus.drush.inc:3556
curl_getinfo(): 84 is not a valid cURL handle resource               [warning]
terminus.drush.inc:3562

...until the build is either manually killed or you hit the 70 minute build maximum.

Any thoughts, ways to mitigate? Ways to recover?

joshkoenig commented 10 years ago

Hey Eric,

This looks like something we need to catch and recover from. The API call to clone the database is failing, but Terminus is marching on, which isn't good.

Separately we need to debug why the API call timed out. That could be a problem on our end, or it could be an inevitable networking "oops". Networking failures will always happen, so Terminus should recover gracefully from them if possible, or barring that fail fast and clearly.

iamEAP commented 10 years ago

Just adding an example of a similar failure that's been cropping up recently; 500 would imply something on the Pahtone end though, I'd imagine.

Cloning database from test........................................................................................................................................Request failed: 500 <html><head><title>Processing                    [failed]
Failed</title></head><body><b>Processing Failed</b></body></html>
No response for site jobs!                                           [error]
Invalid argument supplied for foreach() terminus.drush.inc:1540      [warning]

If you can point me in a general direction, I'd be happy to take a stab at this, since we're relying so heavily on this now.

joshkoenig commented 10 years ago

We should catch these better. A 500 does indeed indicate a failed API call, and that "Processing Failed" indicates there should be a backtrace on our end.

Can you give some more context on what caused this?

mikevanwinkle commented 10 years ago

I think this issue will be resolved in our new cli framework where it will be easy to had an Exception handler to provide intelligent feedback on these failures.