kiwi generates apt config file without proxy

kagehisa commented 1 year ago

Problem description

I'm building behind a corporate proxy. Suse based Images work fine but while trying to build Ubuntu the build failed as soon as the chroot environment was entered while complaining about apt not being able to locate the desired packages. After inspecting the kiwi generated apt configuration file that is used in the chroot command I noticed the configured system proxy was not present in the apt configuration file. During another build run I added the lines

Acquire::http::Proxy "http://web-proxy.company.de";
Acquire::https::Proxy "http://web-proxy.company.de";

to the generated apt config file and the prepare phase of the build was run successfully.

Expected behaviour

Kiwi generated apt configuration file respects the system wide enabled proxy and creates an entry for it or at least provides a way to configure it without the need to manually sneak in the lines during the build process.

Steps to reproduce the behaviour

Use build host with proxy configuration
try building an apt based appliance with a package source outside your company network
inspect the Kiwi generated apt configuration file in the build folder
OS and Software information
KIWI version: 9.24.55
Operating system host version: SLES 15 SP4
Operating system target version: Ubuntu Focal
Open Build Service version (N/A if not using OBS): N/A
Koji version (N/A if not using Koji): N/A

schaefi commented 1 year ago

Thanks for digging into this. I agree the code to add the proxy setup to the apt config is missing in kiwi. I have one question. kiwi cares for the proxy setting in a way that it copies /etc/sysconfig/proxy temporarily into the image root tree during build time. Do you know if Debian/Ubuntu systems also contains this proxy configuration file ?

If yes my solution here would be to take the contents of /etc/sysconfig/proxy and add the missing fields to the kiwi created apt config.

Thanks

iammattcoleman commented 1 year ago

There is no central proxy configuration in Debian/Ubuntu. You could check for the http_proxy, https_proxy, and ftp_proxy variables and use their values, if present. However, APT's proxy settings are set either in /etc/apt/apt.conf or a file within /etc/apt/apt.conf.d/. So, those variables aren't necessarily the same as what APT is using.

kagehisa commented 1 year ago

The /etc/sysconfig/proxy gets copied to the buildroot and after the preparation phase it is removed. But I agree with @iammattcoleman, this file only exists on a Suse build host. Debian/Ubuntu or Arch don't have that file. The safest way would be to use http_proxy, https_proxy and no_proxy environment variables or add the proxy value to the repository tag in the kiwi description. If those are set it is a pretty safe bet that these are also used for apt but not the other way around. On many of our Ubuntu systems only apt has the proxy configured.

iammattcoleman commented 1 year ago

I would prefer adding explicit proxy configuration to the image description XML rather than trying to deduce it from the system configuration. Then, if you need to change it dynamically (during CI builds, maybe?), you could alter the XML to include the build environment's proxy settings using a tool like xmlstarlet prior to running KIWI.

kagehisa commented 1 year ago

I would prefer adding explicit proxy configuration to the image description XML rather than trying to deduce it from the system configuration. Then, if you need to change it dynamically (during CI builds, maybe?), you could alter the XML to include the build environment's proxy settings using a tool like xmlstarlet prior to running KIWI.

I guess this would also be the cleanest way to get a boxbuild work behind a proxy. I'm just getting started with the boxbuild addon, but I see no real way to sneak in my apt proxy configuration during a build like I could with a normal system build.

Conan-Kudo commented 1 year ago

Note that what we do for SUSE doesn't work for Red Hat/Fedora either.

Conan-Kudo commented 5 months ago

I guess this would also be the cleanest way to get a boxbuild work behind a proxy. I'm just getting started with the boxbuild addon, but I see no real way to sneak in my apt proxy configuration during a build like I could with a normal system build.

Since boxbuild uses a VM, that VM process would go through host networking. If you've got global proxy settings in place in host networking, then all traffic should go through it just fine. No need to do anything weird for proxy settings in the build itself.

schaefi commented 5 months ago

Since boxbuild uses a VM, that VM process would go through host networking. If you've got global proxy settings in place in host networking, then all traffic should go through it just fine. No need to do anything weird for proxy settings in the build itself.

yes this is correct, boxbuild should be completely transparent regarding your host network setup

You can give it a try on actually any linux like this

# get some image descriptions
git clone https://github.com/OSInside/kiwi-descriptions.git

# fetch the plugin (this also fetches kiwi)
pip install --upgrade kiwi-boxed-plugin

# build Ubuntu noble live ISO as an example, using the universal box
kiwi-ng --debug system boxbuild --box universal -- --description kiwi-descriptions/ubuntu/x86_64/ubuntu-noble/ --target-dir /tmp/myubuntu

You can do this as normal user. That user must have permissions to run qemu kvm instances. This usually means the user needs to be in the kvm group. qemu-kvm needs to be installed on the system.

schaefi commented 5 months ago

If you run the boxbuild with --box-debug you will stay inside the box and can debug and test things.

kagehisa commented 5 months ago

Thank you, I'm currently playing around with it. I have not managed to get a working build yet but that is a problem with my network setup and the fact that I have to use nested virtualization to run the boxbuild vm. If I get it working this will be perfect for pipeline based image creation. Thanks for all the Information!

OSInside / kiwi