Open sarg3nt opened 1 year ago
+1 Not supporting proxies, prevents this from being used in many enterprises.
Hi, this is super critical to us as we currently can't use huber
in our automation platform to auto update docker images that use huber
to install apps.
I added this bug on April 19 and it looks like it's not getting any attention.
Should we give up and abandon huber
?
Not trying to be a jerk but we need this to function to use huber
.
Note: Currently we are getting around this by using WSL on Windows as it uses the Windows proxy and to Linux in WSL it just works. However we can't use this on a real Linux box so we can't use huber
in Jenkins, which is our automation platform and really really need to. Currently I'm manually rebuilding all the Docker images that use huber
but that is not sustainable.
Thank you.
I have some ideas about how to set up a test environment for this. Not everyone has a proxy server or enterprise networking restrictions to test behind; and it's potentially non-trivial for many home-labbers to set up.
No promise on a timeline, I'm just a random SRE that really likes huber
and I want to be helpful.
I think you might have nerd-sniped me by asking an interesting question that should be totally tractable. especially "NOTE: System proxies are enabled by default.
"... I've consumed HTTP Libraries that had proxy support out of the box, what's going on if just setting ENV doesn't work... 🤔
I went digging and I found that most of the HTTP work is actually inside hubcaps-ex
library. It's really building a Github API Client complete with caching, and rarely doing direct HTTP Client style GETs.
I also had a sudden "hunch" to try explicitly specifying HTTPS_PROXY
instead of HTTP_PROXY
....
squid | 1701806670.918 137 172.22.0.1 TCP_TUNNEL/200 7320 CONNECT raw.githubusercontent.com:443 - HIER_DIRECT/185.199.108.133 -
is it perhaps that the library is actually differentiating between HTTP and HTTPS, and because all the Github API Commands operate over HTTPS, the correct environment variable to set is only HTTPS_PROXY
...?
another research note, I think it was the curl
manpage pointed out that historically, both upper-case and lower-case ENV was acceptable and lowercase http_proxy
is handled before HTTP_PROXY
. so you may want to export https_proxy=proto://host:port
instead.
I also observed that it does seem to expect a protocol, it's not going to assume one for you.
@DanceMore Thanks for looking into this. We really appreciate it.
We set all six possible variables, i.e.
HTTP_PROXY
HTTPS_PROXY
http_proxy
https_proxy
NO_PROXY
no_proxy
and what kind of errors / output do you get when it fails ?
uses -l
not ENV LOG_LEVEL
% HTTPS_PROXY="http://localhost:3128" huber search -l trace
...
⢼ Updating managed repos
⣸ Updating managed repos
2023-12-06T01:20:26.527Z TRACE simpledi_rs::di > getting immutable "huber::service::repo::RepoService" from DI system
2023-12-06T01:20:26.527Z TRACE simpledi_rs::di > getting immutable "huber_common::model::config::Config" from DI system
⠁
2023-12-06T01:20:26.527Z TRACE simpledi_rs::di > getting immutable "huber_common::model::config::Config" from DI system
2023-12-06T01:20:26.527Z INFO huber::service::repo > Saving https://raw.githubusercontent.com/DanceMore/huber-dancemore/master/huber.yaml to "/home/neoice/.huber/repos/remote-repo/huber.yaml"
2023-12-06T01:20:26.532Z DEBUG reqwest::connect > starting new connection: https://raw.githubusercontent.com/
2023-12-06T01:20:26.532Z DEBUG reqwest::connect > proxy(http://localhost:3128) intercepts 'https://raw.githubusercontent.com/'
2023-12-06T01:20:26.533Z TRACE mio::poll > registering event source with poller: token=Token(1), interests=READABLE | WRITABLE
2023-12-06T01:20:26.533Z TRACE reqwest::connect > tunneling HTTPS over proxy
2023-12-06T01:20:26.548Z TRACE want > signal: Want
2023-12-06T01:20:26.548Z TRACE want > signal found waiting giver, notifying
2023-12-06T01:20:26.548Z TRACE want > poll_want: taker wants!
2023-12-06T01:20:26.597Z DEBUG reqwest::async_impl::client > response '200 OK' for https://raw.githubusercontent.com/DanceMore/huber-dancemore/master/huber.yaml
...
I will see matching logs on Squid for raw.githubusercontent.com
too.
Here is the trace output when I get when trying to search huber:
> HTTPS_PROXY="http://corporate.proxy.local:8080" huber search -l trace
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > adding Config { log_level: "TRACE", output_format: Console, home_dir: "/home/awddude/.huber", github_token: None, github_key: None } to DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting mutable "huber_common::model::config::Config" from DI system
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > adding PackageService { container: None } to DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting mutable "huber::service::package::PackageService" from DI system
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > adding ReleaseService { container: None } to DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting mutable "huber::service::release::ReleaseService" from DI system
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > adding CacheService { container: None } to DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting mutable "huber::service::cache::CacheService" from DI system
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > adding HuberUpdateService { container: None } to DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting mutable "huber::service::update::HuberUpdateService" from DI system
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > adding RepoService { container: None } to DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting mutable "huber::service::repo::RepoService" from DI system
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > adding ConfigService { container: None } to DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting mutable "huber::service::config::ConfigService" from DI system
2024-02-28T18:34:02.306Z DEBUG simpledi_rs::di > initializing DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::package::PackageService" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::release::ReleaseService" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::cache::CacheService" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::update::HuberUpdateService" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::repo::RepoService" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::config::ConfigService" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber_common::model::config::Config" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::cmd::search::SearchCmd" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::package::PackageService" from DI system
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber::service::cache::CacheService" from DI system
2024-02-28T18:34:02.306Z INFO huber::service::cache > Updating repos
2024-02-28T18:34:02.306Z TRACE simpledi_rs::di > getting immutable "huber_common::model::config::Config" from DI system
2024-02-28T18:34:02.306Z INFO huber::service::cache > Updating "/home/awddude/.huber/repos/huber"
⢹ Updating managed repos 2024-02-28T18:34:02.350Z INFO huber::component::github > Cloning huber github repo
2024-02-28T18:34:02.350Z DEBUG huber::component::github > Merging huber repo update
⡧ Updating managed repos 2024-02-28T18:36:12.931Z DEBUG huber::component::github > Failed to fetch huber github repo, failed to connect to github.com: Connection timed out; class=Os (2)
2024-02-28T18:36:12.940Z INFO huber::component::github > Cloning huber repo via https
error: failed to connect to github.com: Connection timed out; class=Os (2)awddude@awddudelxpc:~/source
As you can see it fails to connect to github.
For comparison here is the output I get when cloning the huber repo using git
on the same workstation:
> HTTPS_PROXY="http://corporate.proxy.local:8080" git clone https://github.com/innobead/huber.git
Cloning into 'huber'...
remote: Enumerating objects: 6489, done.
remote: Counting objects: 100% (992/992), done.
remote: Compressing objects: 100% (467/467), done.
remote: Total 6489 (delta 583), reused 880 (delta 523), pack-reused 5497
Receiving objects: 100% (6489/6489), 1.07 MiB | 7.72 MiB/s, done.
Resolving deltas: 100% (4160/4160), done.
Git
works just fine.
Thanks again for looking into this for us.
When running Huber from behind a proxy it is unable to talk to Github so
huber search
and any other command that need to talk to Github do not work.I think you just need to configure
reqwest
to use theHTTP_PROXY
,HTTPS_PROXY
andNO_PROXY
vars if they exist.See: https://docs.rs/reqwest/latest/reqwest/struct.Proxy.html
NOTE:
reqwest
docs say:So something must be interfering with this or turning it off.
Please let me know if need someone to test changes.