r10k should retry on occassional proxy connect failure

electrofelix commented 5 years ago

Getting occasional failures with proxy connects failing which are transient:

2018-11-29 12:04:25.858 GMT |      srv-5: WARN   -> Puppet Forge module 'puppetlabs-docker_platform' has been deprecated, visit https://forge.puppet.com/puppetlabs/docker_platform for more information.
2018-11-29 12:05:01.373 GMT |      srv-5: WARN   -> Puppet Forge module 'basti1302-wait_for' has been deprecated, visit https://forge.puppet.com/basti1302/wait_for for more information.
2018-11-29 12:05:03.140 GMT |      srv-5: WARN   -> Puppet Forge module 'stankevich-python' has been deprecated, visit https://forge.puppet.com/stankevich/python for more information.
2018-11-29 12:05:26.279 GMT |      srv-5: ERROR  -> Unable to connect to https://forgeapi.puppetlabs.com (using proxy http://proxy:8080): Failed to open TCP connection to proxy:8080 (Connection refused - connect(2) for "proxy" port 8080)
2018-11-29 12:05:33.555 GMT |      srv-5: Failed to update puppet modules
2018-11-29 12:05:33.555 GMT |      srv-5: Puppet module update failed

This appears to be related to #42 but the precise errors that should be handled were unknown.

Given that earlier request connections with the same proxy succeeded, this should confirm that there isn't an issue with the proxy settings, they are pointing to a valid proxy, but there has been some transient error.

Looking around the net it appears that 'connection refused' can also occur when a connection buffer is full and subsequent attempts will be rejected until it clears a bit - https://stackoverflow.com/a/2333446/1597808

This suggests it should be possible to configure r10k retry on "connection refused" errors from proxies, provided they have already connected successfully.

The exact error looks to be coming from https://github.com/lostisland/faraday/blob/f08a985bd1dc380ed2d9839f1103318e2fad5f8b/lib/faraday/adapter/httpclient.rb#L54 which is caught and outputted with the error message via: https://github.com/puppetlabs/forge-ruby/blob/a5268161fec1a33c47cd58fcc8e5359b3b8fe31a/lib/puppet_forge/connection/connection_failure.rb#L14

While it's unclear to me where r10k is processing the list of modules to retrieve, but it does seem like that r10k is best placed to be aware of when a proxy has been enabled for a particular module and in turn to track that the connection has previously been successful in order to decide whether it is worth retrying. e.g.

Connection previously successful within the same operation, likely transient, so retry after a back-off
First time with connection, then fail as no indication it should succeed

If someone could provide a bit of guidance on where best to perform this type of logic assuming it doesn't require a substantial refactoring in order to capture connection settings being used in order to establish whether the failed connect attempted was using settings (endpoint + proxy) that previously worked and therefore is worth retrying, I might be able to do something to help.

github-actions[bot] commented 3 years ago

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

electrofelix commented 3 years ago

While I'm no longer using puppet, I'd imagine this issue still impacts other users who have to run their code behind a proxy.

github-actions[bot] commented 3 years ago

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

puppetlabs / r10k

r10k should retry on occassional proxy connect failure #884