builder/amazon: make resilient AWS API calls

[mwhooker]

aws api call may fail, so it's good practice to retry them a couple of times in case of intermittent failure. See amazon's documentation on the subject here http://docs.aws.amazon.com/general/latest/gr/api-retries.html

• [x] merge mitchellh/goamz#18

[mwhooker]

I think a wise way to go about this is by using a retrying http transport. I will implement this using http://github.com/daaku/go.httpcontrol. Transport is a nice interface, so we can swap it out down the road if we need to.

[mwhooker]

I've started implementing this here https://github.com/mwhooker/packer

I get the feeling the retry logic might need to be changed.

go.httpcontrol (rightly) refuses to retry non-GET requests and non-temporary network failures.

Amazon however allows mutating actions via HTTP GET requests, and supports idempotence at the application level

go.httpcontrol also does not provide exponential backoff.

I propose changing the retry logic to something like

retries = 0

DO
    wait for (2^retries * 100) milliseconds

    http_status, err = Make request.

    IF http_status = SUCCESS
        retry = false
    ELSE IF err.Temporary = true
        retry = true
    ELSE IF http_status = 4xxx
        retry = true
    ELSE
        Some other error occurred, so stop calling the API.
        retry = false
    END IF

    retries = retries + 1

WHILE (retry AND (retries < MAX_RETRIES))

perhaps the best method would be plugable retry logic

[mwhooker]

The above psuedocode is implemented in my fork of go.httpcontrol

[mitchellh]

I'll review your amz stuff over there.

[mitchellh]

Done.