upgrade to 2.10 broke http_proxy support

[luizluca]

Issue Description

I have a private OBS instance. After update to 2.10, all build fails because it cannot reach api.opensuse.org.

It might be related to the systemd migration.

Expected Result

It should work behind http_proxy

How to Reproduce

Setup a server that can only reach internet through a http_proxy. Setup http proxy at:

/etc/sysconfig/proxy:
HTTP_PROXY="http://proxy.mydomain:3128"
HTTPS_PROXY="http://proxy.mydomain:3128"
NO_PROXY="localhost, 127.0.0.1, .mydomain"

/usr/lib/obs/server/BSConfig.pm:
our $proxy = "http://proxy.mydomain:3128/";
our $noproxy = "localhost, 127.0.0.1, .mydomain";

/srv/obs/configuration.xml:
  <http_proxy>http://proxy.mydomain:3128</http_proxy>
  <no_proxy>.mydomain</no_proxy>

/srv/www/obs/api/config/options.yml:
  http_proxy: "http://proxy.mydomain:3128"
  no_proxy: ".mydomain"

Let packages break. I setup a auditd rule to match it and I can see bs_srcserver trying to acess 195.135.221.133 (api.opensuse.org?).

I added a print at top of /usr/lib/obs/server/bs_srcserver and it does have $proxy defined. It might be something that is not using it.

I get "no route to host" errors in:

/srv/obs/log/scheduler_x86_64.log:myproject/openSUSE_Leap_42.3: openSUSE.org:openSUSE:Leap:42.3: interconnect error: connect to api.opensuse.org:443: No route to host

/srv/www/obs/api/log/production.log:E, [2019-08-01T18:31:13.712504 #1896] ERROR -- : [1896:2967.78] 2019-08-01T18:31:13-0300: [Worker(delayed_job.1010 host:vms095l pid:1896)] Job IssueTrackerUpdateIssuesJob [e30ddde5-a0dc-4aa9-8254-748335015333] from DelayedJob(issuetracking) with arguments: [15] (id=5043469) (queue=issuetracking) FAILED (19 prior attempts) with Errno::EHOSTUNREACH: Failed to open TCP connection to bugzilla.opensuse.org:443 (No route to host - connect(2) for "bugzilla.opensuse.org" port 443)

Further Information

• obs-*-2.10-10.1.noarch (private instance)

HTTP Proxy usage in OBS is still a mess, with multiple places to be defined. It could allow individual proxy settings for each service but it should, by default, simply accept system settings.

Is it too hard to add an autotest for http_proxy support? It just need an squid and a firewall rule to reject direct internet access.

[luizluca]

I debugged it further. It seems that "fetching remote project data for" is using proxy:

13532 write(1, "fetching remote project data for"..., 60) = 60
(...)
13532 connect(4, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("10.9.1.137")}, 16) = 0
13532 write(4, "CONNECT api.opensuse.org:443 HTT"..., 69) = 69
13532 read(4, "HTTP/1.0 200 Connection establis"..., 1024) = 39

While "fetching remote project config" is not:

13532 write(1, "fetching remote project config f"..., 62) = 62
13532 connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("195.135.221.133")}, 16) = -1 EHOSTUNREACH (No route to host)
13532 close(4)                          = 0

I added some prints and sometimes it has remote_proxy defined for a project while others it does not have.

I forced set it with:

$proj->{'remoteproxy'} = "http://proxy.mydomain:3128";

And packages started to get built. It seems that $proj->{'remoteproxy'} is only sometimes set in fetchremoteconfig, even with the same caller

/usr/lib/obs/server/bs_srcserver:1092 in function (eval)
/usr/lib/obs/server/bs_srcserver:1081 in function (eval)

[luizluca]

I'm trying to identify what makes a project include or not remoteproxy property.

I added

print "fetchremoteconfig(): $proj->{'remoteurl'}/source/$proj->{'remoteproject'}/_config via $proj->{'remoteproxy'}\n";

And it is not related to a specific project. Factory both included (1st) and missed (4th) remoteproxy in a short period of time and still in the same perl process:

fetchremoteconfig(): https://api.opensuse.org/public/source/openSUSE:Factory/_config via http://proxy.mydomain:3128/
fetchremoteconfig(): https://api.opensuse.org/public/source/SUSE:SLE-12:SLE-Module-Toolchain/_config via http://proxy.mydomain:3128/
fetchremoteconfig(): https://api.opensuse.org/public/source/SUSE:SLE-12:SLE-Module-Web-Scripting/_config via http://proxy.mydomain:3128/
fetchremoteconfig(): https://api.opensuse.org/public/source/openSUSE:Factory/_config via 
fetchremoteconfig(): https://api.opensuse.org/public/source/SUSE:SLE-12-SP4:GA/_config via http://proxy.mydomain:3128/
fetchremoteconfig(): https://api.opensuse.org/public/source/SUSE:SLE-12-SP4:GA/_config via http://proxy.mydomain:3128/

However, it only happens in less than 1% of calls and, currently, only with factory. I hit a dead end. It looks like remoteproxy is defined only in two places using global proxy value. So, it might be someplace where that property is removed or project is cloned but without that field.

[achaios]

For me this bug causes all packages to always stay in "broken" state. I am only building against Factory. I spent two days trying to debug this to no avail. The only solution was the above mentioned fix in fetchremoteconfig()

-    'proxy' => $proj->{'remoteproxy'}, 
+    'proxy' => $proxy,

[achaios]

I just updated to 2.10.7 and the problem seems to be fixed.

[hennevogel]

Thank you for the feedback @achaios :bouquet:

[reusch]

Sorry if I wake sleeping dogs: This problem is still not fixed on our freshly installed machine. Same fix works here. The OBS packages we use are from http://download.opensuse.org/repositories/OBS:/Server:/2.10/15.3/ with versions:

 i+ | obs-api                   | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64
 i+ | obs-api-deps              | package | 2.10.15-lp153.92.1 | x86_64 | integration-open-build-service-sles15-sp3-x86_64
 i  | obs-backend-testsuite     | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64
 i  | obs-bundled-gems          | package | 2.10.15-lp153.92.1 | x86_64 | integration-open-build-service-sles15-sp3-x86_64
 i  | obs-common                | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64
 i+ | obs-server                | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64
 i  | obs-service-download_url  | package | 0.1.1-bp153.1.11   | noarch | integration-suse-packagehub-15-sp3-backports-pool-x86_64
 i  | obs-service-verify_file   | package | 0.1.1-bp153.1.14   | noarch | integration-suse-packagehub-15-sp3-backports-pool-x86_64
 i  | obs-signd                 | package | 2.5.10-lp153.1.1   | x86_64 | integration-open-build-service-sles15-sp3-x86_64
 i+ | obs-worker                | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64
 i  | system-user-obsapidelayed | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64
 i  | system-user-obsrun        | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64
 i  | system-user-obsservicerun | package | 2.10.15-lp153.92.1 | noarch | integration-open-build-service-sles15-sp3-x86_64

Proxy is configured in all the files as stated above (and even some more for good measure). Still, the $proj->{'remoteproxy'} value is empty within the fetchremoteconfig() function in some cases. After tracing down the error, I get the following results:

======== Project openSUSE.org:SUSE:SLE-15:GA has proxy (within fetchremoteconfig): empty
Trace begun at /usr/lib/obs/server/BSSrcServer/Remote.pm line 192
BSSrcServer::Remote::fetchremoteconfig('HASH(0x55775f7ed618)', 'openSUSE.org:SUSE:SLE-15:GA', 'HASH(0x55775f798a58)') called at /usr/lib/obs/server/bs_srcserver line 1110

In this case, my $aproj = $remotemap->{$aprojid}; is used as $proj parameter.

======== Project openSUSE.org:SUSE:SLE-15:GA has proxy (within fetchremoteconfig): http://user:pass@myproxy:4711
Trace begun at /usr/lib/obs/server/BSSrcServer/Remote.pm line 192
BSSrcServer::Remote::fetchremoteconfig('HASH(0x55775f7d8e20)', 'openSUSE.org:SUSE:SLE-15:GA', 'HASH(0x55775f798a58)') called at /usr/lib/obs/server/bs_srcserver line 2234

In this case, my $proj = BSSrcServer::Remote::remoteprojid($p);is used as $proj parameter.

I hope this helps. We were fighting with this error for a while now. Proxy configuration in OBS is quite painful. Please let me know if you need more information.

[reusch]

Hello again, just wanted to say that we fixed it on our site by patching the file /usr/lib/obs/server/bs_srcserver:

--- /usr/lib/obs/server/bs_srcserver.old        2022-06-21 08:03:38.846853838 +0200
+++ /usr/lib/obs/server/bs_srcserver    2022-06-21 08:03:46.535614662 +0200
@@ -1107,6 +1107,7 @@
                $aproj->{'config'} = readstr("$projectsdir/$aprojid.conf", 1);
                $aproj->{'config'} = '' unless defined $aproj->{'config'};
              } else {
+                $aproj->{'remoteproxy'} = $proxy;
                eval { BSSrcServer::Remote::fetchremoteconfig($aproj, $aprojid, $remotemap) };
              }
            }

(This corresponds to upstream https://github.com/openSUSE/open-build-service/blob/master/src/backend/bs_srcserver#L1175 )

This works and the root cause more likely seems to be in this file, but I feel like it's not the right place for the fix to be made. Sadly, I cannot really see, where the $remotemap variable is filled. I may be wrong, but I think the problem starts here: https://github.com/openSUSE/open-build-service/blob/master/src/backend/bs_srcserver#L1107

I am not a ruby guy and therefor this huge project is beyond my understanding. I just fiddle 'til it works 😀

[luizluca]

I am not a ruby guy and therefor this huge project is beyond my understanding. I just fiddle 'til it works

Yeah, you are really not a ruby guy. This code is perl ;-)

Yes, I've been breaking and patching OBS 2.10 for many months on every update. That's not funny to maintain.

I'm using this two patches for fixing proxy usage:

--- /srv/www/obs/api/app/models/project/remote_url.rb.old       2020-01-16 15:50:02.388452444 -0300
+++ /srv/www/obs/api/app/models/project/remote_url.rb   2021-08-27 20:14:00.819593773 -0300
@@ -7,6 +7,9 @@
     if ENV['http_proxy'].blank?
       ENV['http_proxy'] = Configuration.first.http_proxy
     end
+    if ENV['https_proxy'].blank?
+      ENV['https_proxy'] = Configuration.first.http_proxy
+    end    
     if ENV['no_proxy'].blank?
       ENV['no_proxy'] = Configuration.first.no_proxy
     end

--- usr/lib/obs/server/BSSrcServer/Remote.pm.old   2020-04-03 06:25:16.000000000 -0300
+++ usr/lib/obs/server/BSSrcServer/Remote.pm       2020-03-27 14:21:02.239372724 -0300
@@ -199,6 +199,9 @@
       $remotemap->{$projid} = $rproj;
     }
   }
+
+  $proj->{'remoteproxy'} = $proxy;
+
   print "fetching remote project config for $projid\n";
   my $param = {
     'uri' => "$proj->{'remoteurl'}/source/$proj->{'remoteproject'}/_config",

The first patch fixes the issue when you try to add a new repo to a project (when the list of distributions is shown). I know it is dirty but those two patches get the job done.

Please, backport the fixes for 2.10, including https://github.com/openSUSE/open-build-service/commit/ce9fe2f0c0c81a01df0d553244930f1d7f130a6c

I hope 2.11 with all those fixes will get out soon.

[reusch]

Uhm... Yeah. Meant that. Also had to fix the ruby file. Mixed that up. :-D