Github API rate-limits cause flaky builds

[aej]

Hi!

I've been building and deploying exla on render.com and their IP addresses look to be heavily rate-limited by Github's API. This causes lots of failures while building the xla library into an elixir application.

To get around this I've been passing in my own HTTP auth headers to curl in a fork (see https://github.com/aej/xla/commit/5e88ca775069b24e24e3541d7b083491196e0b09).

It would be great if we could support passing in generic arguments to the two libraries used here (curl & wget). Perhaps we could support an environment variable like NETWORK_TOOL_OPTS which just then gets passed directly into the cli call.

eg:

curl -> curl --fail --silent -L #{System.get_env("NETWORK_TOOL_OPTS")} #{url} wget -> wget -q -O - #{System.get_env("NETWORK_TOOL_OPTS")} #{url}

This would mean we can support any additional arguments library users have and not only http headers.

[jonatanklosko]

Hey! I would do something less specific to the CLI tools. I'm thinking XLA_DOWNLOAD_HEADERS that we can translate to CLI args regardless of what we use underneath. For most practical purposes it should be fine to accept "key: value;key: value", but happy to hear better alternatives. @josevalim wdyt?

[josevalim]

What about XLA_DOWNLOAD_UA?

[jonatanklosko]

@josevalim you mean env var for a specific header? In that case we rather want auth and not user-agent, no? But it may be too limiting to support just one specific header?

[josevalim]

I see. So your header approaches works, :+1:.