enable THEO for corporate proxies

[eet1mu3]

Some environments still use http_proxy. Theo does currently not readily work in such an environment.

• [ ] document docker proxy config
• [ ] add separate proxy config for maven, which is required for building keycloak (I have it manually, maybe need to generate it automatically from proxy info)
• [ ] no_proxy configuration: list all domains currently: 127.0.0.0/8,192.0.0.0/8,keycloak,netbird,netbird-dashboard,carl,edgar,edgar-leader,netbird-management"
• [ ] make sure every http call that uses http_proxy also supports no_proxy. (or, in the case it does not support no_proxy, also doesn't use the proxy. This seems not to be the case, there is a HTTP GET http://netbird-management/api/users done over the proxy, despite the domain being in no_proxy

[reimarstier]

Proxy configuration is definitely missing. My own setup does not have that requirement, there is a way to entirely work around it. I will add notes for other environments. But for the test environment in the virtual machine, it should only require a proxy configuration for docker to pull the images. For the Development mode I'm not sure, this needs probably some adjustments. It highly depends on your local setup. Can you elaborate on how you run THEO?

[reimarstier]

I have added documentation to follow the steps required to set this up and fixed the name resolution in the traefik docker container that is serving the NetBird management service (traefik was also using the proxy, therefore I changed it to resolve IP addresses to make the list of services not grow even further). A list of known service names is used in the shell script here: https://github.com/eclipse-opendut/opendut/blob/2e50e537a19d3f1303b109503426dcee12d160fe/.ci/docker/vagrant/proxy.sh#L16-L20

In the end there were quite a few more things missing than I thought. So I was wrong to say that it should work in the virtual machine.

Proxy documentation

@eet1mu3 Can you check on your end if it works now and if the documentation is comprehensible?

[eet1mu3]

My starting point

• local forwarding proxy just outside of the VM, for debugging it is currently configured to show all connection requests.
• pre-existing working linux VM with proxy configuration environment variables connecting to that local proxy.
• I have added the config to docker, both daemon and client.
• I have added a settings.xml with my hard-coded proxy config to the keycloak docker (COPY ./.ci/docker/keycloak/settings.xml /root/.m2/settings.xml), as maven inexplicably seems to ignore both environment and command line arguments. (this can be in theory automatically created from the docker config/env, but works for me now)

From there, I used the process described in https://github.com/eclipse-opendut/opendut/blob/2e50e537a19d3f1303b109503426dcee12d160fe/doc/src/development/testenv/theo-setup-docker.md

What is working: Compilation, creation. Then it takes ages to generate the keycloak provisioning, but it finally seems to be successfull. And currently, the process is always stuck at the netbird provisioning:

Waiting for netbird api key to be available...
Waiting for netbird api key to be available...
Error: Timeout: Timeout while waiting for netbird api key to be available.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <T as core::convert::Into<U>>::into
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/convert/mod.rs:757:9
   2: anyhow::kind::Trait::new
             at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/kind.rs:93:9
   3: opendut_theo::core::docker::netbird::wait_for_netbird_api_key
             at ./.ci/docker/theo/src/core/docker/netbird.rs:13:24
   4: opendut_theo::commands::testenv::TestenvCli::default_handling
             at ./.ci/docker/theo/src/commands/testenv.rs:66:17
   5: opendut_theo::main
             at ./.ci/docker/theo/src/main.rs:39:47
   6: core::ops::function::FnOnce::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
   7: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
   8: std::rt::lang_start::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:166:18
   9: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284:13
  10: std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  11: std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  12: std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  13: std::rt::lang_start_internal::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:48
  14: std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  15: std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  16: std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  17: std::rt::lang_start_internal
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:20
  18: std::rt::lang_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:165:17
  19: main
  20: __libc_start_call_main
             at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  21: __libc_start_main_impl
             at ./csu/../csu/libc-start.c:360:3
  22: _start
2024-02-29T15:50:18.905838Z  INFO theo:run: opendut_ci::tasks::run: close time.busy=1425s time.idle=25.1µs package=Theo
2024-02-29T15:50:18.905991Z  INFO theo: opendut_ci::packages::theo: close time.busy=1425s time.idle=64.2µs

About your config: During debugging, my impression was that wildcards for partial hostnames in no_proxy are often not recognized, but I was able circumvent that by just listing all hosts.

I am not quite sure why my setup stops here, the only problem I am currently experiencing is that something inside gets to make a request "GET http://netbird-management/api/groups" via the proxy, despite I explicitly excluded netbird-management in the no_proxy config. This could be some implementation other than curl that recognizes the http_proxy, but not the no_proxy. Or some other misconfiguration.

[reimarstier]

Hey, I have opened a pull request #114 and linked this issue. Could you check on the branch 'proxy' used in the pull request? The netbird-management url does not work on development branch because the request is going to the Netbird traefik proxy service and traefik in turn tries to redirect the request to a container name called 'netbird-management-container'. This eventually fails because it is missing in your NO_PROXY exception list.

[eet1mu3]

The proxy branch works until now, thanks! For reference, without vagrant:

~/.docker/config.json:

{
 "proxies": {
   "default": {
     "httpProxy": "http://x.x.x.x:3128",
     "httpsProxy": "http://x.x.x.x:3128",
     "noProxy": "localhost,127.0.0.1,netbird-management,netbird-dashboard,netbird-signal,netbird-coturn,keycloak,edgar-leader,edgar-*,carl,192.168.0.0/16"
   }
 }

/etc/docker/daemon.json:

{
  "proxies": {
    "http-proxy": "http://x.x.x.x:3128",
    "https-proxy": "http://x.x.x.x:3128",
    "no-proxy": "localhost,127.0.0.1,netbird-management,netbird-dashboard,netbird-signal,netbird-coturn,keycloak,edgar-leader,edgar-*,carl,192.168.0.0/16"
  }
}