search

uyuni-project / uyuni

Source code for Uyuni
https://www.uyuni-project.org/
GNU General Public License v2.0
430 stars 180 forks source link

uyuni proxy doesn't work as expected: all proxed systems cannot access repos #2963

Closed milari closed 3 years ago

milari commented 3 years ago

Hi. I've a setup of a uyuni server with a proxy.

What's report /var/log/messages, on client system, when I try to install a pkg from uyuni:

Dec 3 09:53:03 labgraphite systemd: Started /usr/bin/yum --quiet --assumeyes clean expire-cache. Dec 3 09:53:03 labgraphite systemd: Started /usr/bin/yum --quiet --assumeyes check-update --setopt=autocheck_running_kernel=false. Dec 3 09:53:04 labgraphite systemd: Started /usr/bin/yum -y install telegraf-1.16.3-1. Dec 3 09:53:04 labgraphite salt-minion: [ERROR ] Command '[u'systemd-run', u'--scope', u'yum', u'-y', u'install', u'telegraf-1.16.3-1']' failed with return code: 1 Dec 3 09:53:04 labgraphite salt-minion: [ERROR ] stdout: Running scope as unit run-24285.scope. Dec 3 09:53:04 labgraphite salt-minion: Loaded plugins: fastestmirror, susemanagerplugin, yumnotify Dec 3 09:53:04 labgraphite salt-minion: Loading mirror speeds from cached hostfile Dec 3 09:53:04 labgraphite salt-minion: https://uyuniproxy.example.it:443/rhn/manager/download/centos7-uyuni-client-x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 500 - Internal Server Error Dec 3 09:53:04 labgraphite salt-minion: Trying other mirror. Dec 3 09:53:04 labgraphite salt-minion: One of the configured repositories failed (Uyuni Client Tools for CentOS 7 (x86_64)), Dec 3 09:53:04 labgraphite salt-minion: and yum doesn't have enough cached data to continue. At this point the only Dec 3 09:53:04 labgraphite salt-minion: safe thing yum can do is fail. There are a few ways to work "fix" this: Dec 3 09:53:04 labgraphite salt-minion: 1. Contact the upstream for the repository and get them to fix the problem. Dec 3 09:53:04 labgraphite salt-minion: 2. Reconfigure the baseurl/etc. for the repository, to point to a working Dec 3 09:53:04 labgraphite salt-minion: upstream. This is most often useful if you are using a newer Dec 3 09:53:04 labgraphite salt-minion: distribution release than is supported by the repository (and the Dec 3 09:53:04 labgraphite salt-minion: packages for the previous distribution release still work). Dec 3 09:53:04 labgraphite salt-minion: 3. Run the command with the repository temporarily disabled Dec 3 09:53:04 labgraphite salt-minion: yum --disablerepo=susemanager:centos7-uyuni-client-x86_64 ... Dec 3 09:53:04 labgraphite salt-minion: 4. Disable the repository permanently, so yum won't use it by default. Yum Dec 3 09:53:04 labgraphite salt-minion: will then just ignore the repository until you permanently enable it Dec 3 09:53:04 labgraphite salt-minion: again or use --enablerepo for temporary usage: Dec 3 09:53:04 labgraphite salt-minion: yum-config-manager --disable susemanager:centos7-uyuni-client-x86_64 Dec 3 09:53:04 labgraphite salt-minion: or Dec 3 09:53:04 labgraphite salt-minion: subscription-manager repos --disable=susemanager:centos7-uyuni-client-x86_64 Dec 3 09:53:04 labgraphite salt-minion: 5. Configure the failing repository to be skipped, if it is unavailable. Dec 3 09:53:04 labgraphite salt-minion: Note that yum will try to contact the repo. when it runs most commands, Dec 3 09:53:04 labgraphite salt-minion: so will have to try and fail each time (and thus. yum will be be much Dec 3 09:53:04 labgraphite salt-minion: slower). If it is a very temporary problem though, this is often a nice Dec 3 09:53:04 labgraphite salt-minion: compromise: Dec 3 09:53:04 labgraphite salt-minion: yum-config-manager --save --setopt=susemanager:centos7-uyuni-client-x86_64.skip_if_unavailable=true Dec 3 09:53:04 labgraphite salt-minion: failure: repodata/repomd.xml from susemanager:centos7-uyuni-client-x86_64: [Errno 256] No more mirrors to try. Dec 3 09:53:04 labgraphite salt-minion: https://uyuniproxy.example.it:443/rhn/manager/download/centos7-uyuni-client-x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 500 - Internal Server Error Dec 3 09:53:04 labgraphite salt-minion: [ERROR ] retcode: 1 Dec 3 09:53:06 labgraphite salt-minion: [ERROR ] Error occurred installing package(s). Additional info follows: Dec 3 09:53:06 labgraphite salt-minion: errors: Dec 3 09:53:06 labgraphite salt-minion: - Running scope as unit run-24285.scope. Dec 3 09:53:06 labgraphite salt-minion: Loaded plugins: fastestmirror, susemanagerplugin, yumnotify Dec 3 09:53:06 labgraphite salt-minion: Loading mirror speeds from cached hostfile Dec 3 09:53:06 labgraphite salt-minion: https://uyuniproxy.example.it:443/rhn/manager/download/centos7-uyuni-client-x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 500 - Internal Server Error Dec 3 09:53:06 labgraphite salt-minion: Trying other mirror. Dec 3 09:53:06 labgraphite salt-minion: One of the configured repositories failed (Uyuni Client Tools for CentOS 7 (x86_64)), Dec 3 09:53:06 labgraphite salt-minion: and yum doesn't have enough cached data to continue. At this point the only Dec 3 09:53:06 labgraphite salt-minion: safe thing yum can do is fail. There are a few ways to work "fix" this: Dec 3 09:53:06 labgraphite salt-minion: 1. Contact the upstream for the repository and get them to fix the problem. Dec 3 09:53:06 labgraphite salt-minion: 2. Reconfigure the baseurl/etc. for the repository, to point to a working Dec 3 09:53:06 labgraphite salt-minion: upstream. This is most often useful if you are using a newer Dec 3 09:53:06 labgraphite salt-minion: distribution release than is supported by the repository (and the Dec 3 09:53:06 labgraphite salt-minion: packages for the previous distribution release still work). Dec 3 09:53:06 labgraphite salt-minion: 3. Run the command with the repository temporarily disabled Dec 3 09:53:06 labgraphite salt-minion: yum --disablerepo=susemanager:centos7-uyuni-client-x86_64 ... Dec 3 09:53:06 labgraphite salt-minion: 4. Disable the repository permanently, so yum won't use it by default. Yum Dec 3 09:53:06 labgraphite salt-minion: will then just ignore the repository until you permanently enable it Dec 3 09:53:06 labgraphite salt-minion: again or use --enablerepo for temporary usage: Dec 3 09:53:06 labgraphite salt-minion: yum-config-manager --disable susemanager:centos7-uyuni-client-x86_64 Dec 3 09:53:06 labgraphite salt-minion: or Dec 3 09:53:06 labgraphite salt-minion: subscription-manager repos --disable=susemanager:centos7-uyuni-client-x86_64 Dec 3 09:53:06 labgraphite salt-minion: 5. Configure the failing repository to be skipped, if it is unavailable. Dec 3 09:53:06 labgraphite salt-minion: Note that yum will try to contact the repo. when it runs most commands, Dec 3 09:53:06 labgraphite salt-minion: so will have to try and fail each time (and thus. yum will be be much Dec 3 09:53:06 labgraphite salt-minion: slower). If it is a very temporary problem though, this is often a nice Dec 3 09:53:06 labgraphite salt-minion: compromise: Dec 3 09:53:06 labgraphite salt-minion: yum-config-manager --save --setopt=susemanager:centos7-uyuni-client-x86_64.skip_if_unavailable=true Dec 3 09:53:06 labgraphite salt-minion: failure: repodata/repomd.xml from susemanager:centos7-uyuni-client-x86_64: [Errno 256] No more mirrors to try. Dec 3 09:53:06 labgraphite salt-minion: https://uyuniproxy.example.it:443/rhn/manager/download/centos7-uyuni-client-x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 500 - Internal Server Error

Additional Information

Version of Uyuni Server and Proxy (if used)

zypper info Uyuni-Server-release and zypper info Uyuni-Proxy-release (if used) Information for package Uyuni-Server-release:

Repository : uyuni-server-stable Name : Uyuni-Server-release Version : 2020.09-98.2.uyuni Arch : x86_64 Vendor : obs://build.opensuse.org/systemsmanagement:Uyuni Support Level : Level 3 Installed Size : 1.4 KiB Installed : Yes (automatically) Status : up-to-date Source package : Uyuni-Server-release-2020.09-98.2.uyuni.src Summary : Uyuni Server Description : Uyuni lets you efficiently manage physical, virtual, and cloud-based Linux systems. It provides automated and cost-effective configuration and software management, asset management, and system provisioning.

Information for package Uyuni-Proxy-release:

Repository : @System Name : Uyuni-Proxy-release Version : 2020.09-98.2.uyuni Arch : x86_64 Vendor : obs://build.opensuse.org/systemsmanagement:Uyuni Installed Size : 1.4 KiB Installed : Yes (automatically) Status : up-to-date Source package : Uyuni-Proxy-release-2020.09-98.2.uyuni.src Summary : Uyuni Proxy Description : Uyuni Proxies extend large and/or geographically dispersed Uyuni environments to reduce load on the Uyuni Server, lower bandwidth needs, and provide faster local updates.

Details about issue

After setup, checking proxy content of dirs, no data contained: uyuniproxy:~ # ll /var/spool/rhn total 0 uyuniproxy:~ # ll /var/spool/rhn-proxy/ total 0 drwxr-x--- 1 wwwrun www 0 Sep 18 12:57 list uyuniproxy:~ # ll /var/spool/rhn-proxy/list/ total 0

I see something like that onto: [uyuni Proxy] package_mapping in /var/spool/rhn-proxy/list not created #2473

Have You any updates on it? May be issue related? Thanks.

mcalmer commented 3 years ago

You got an 500 ISE. So the error is either on the proxy or on the server. Check the logs. Maybe you find a hint.

milari commented 3 years ago

Hi, thanks for your answer. If this may be a trace, I see on proxy logs, that broker trace request from client, but no redirect trace requests:

uyuniproxy:/var/log/rhn # ll total 40 -rw-rw---- 1 wwwrun www 29909 Dec 14 08:50 rhn_proxy_broker.log -rw-rw---- 1 wwwrun www 4505 Nov 27 17:35 rhn_proxy_redirect.log

Please let me know if there're other files where I can search for hints. Thanks a lot.

mcalmer commented 3 years ago

Check also the apache error_log

milari commented 3 years ago

[Mon Dec 14 09:05:09.133539 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.enable_sendfile: 0 [Mon Dec 14 09:05:09.133550 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.handler_script: [Mon Dec 14 09:05:09.133560 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.ignore_activity: 0 [Mon Dec 14 09:05:09.133569 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.listener_host: [Mon Dec 14 09:05:09.133579 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.listener_port: 443 [Mon Dec 14 09:05:09.133589 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.path_info: /centos7-uyuni-client-x86_64/repodata/repomd.xml [Mon Dec 14 09:05:09.133599 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.process_group: [Mon Dec 14 09:05:09.133609 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.request_handler: wsgi-script [Mon Dec 14 09:05:09.133619 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.request_id: 7g/AGehxMm4 [Mon Dec 14 09:05:09.133629 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.request_start: 1607933108424686 [Mon Dec 14 09:05:09.133639 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.script_name: /rhn/manager/download [Mon Dec 14 09:05:09.133649 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.script_reloading: 1 [Mon Dec 14 09:05:09.133659 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.script_start: 1607933108537548 [Mon Dec 14 09:05:09.133669 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.thread_id: 0 [Mon Dec 14 09:05:09.133679 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.thread_requests: 0 [Mon Dec 14 09:05:09.133689 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.total_requests: 0 [Mon Dec 14 09:05:09.133698 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \tmod_wsgi.version: (4, 5, 18) [Mon Dec 14 09:05:09.133708 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.errors: <_io.TextIOWrapper name='' encoding='utf-8'> [Mon Dec 14 09:05:09.133718 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.file_wrapper: <class 'mod_wsgi.FileWrapper'> [Mon Dec 14 09:05:09.133728 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.input: <rhn.SmartIO.SmartIO object at 0x7feb3d73a358> [Mon Dec 14 09:05:09.133738 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.multiprocess: True [Mon Dec 14 09:05:09.133748 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.multithread: False [Mon Dec 14 09:05:09.133758 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.run_once: False [Mon Dec 14 09:05:09.133768 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.url_scheme: https [Mon Dec 14 09:05:09.133778 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] \twsgi.version: (1, 0) [Mon Dec 14 09:05:09.133788 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Extra information about this error: [Mon Dec 14 09:05:09.133797 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Unhandled exception type [Mon Dec 14 09:05:09.133807 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.133817 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Exception Handler Information [Mon Dec 14 09:05:09.133827 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Traceback (most recent call last): [Mon Dec 14 09:05:09.133837 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/lib/python3.6/site-packages/spacewalk/common/suseLib.py", line 269, in accessible [Mon Dec 14 09:05:09.133847 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] curl.perform() [Mon Dec 14 09:05:09.133857 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] pycurl.error: (60, "SSL: no alternative certificate subject name matches target host name 'uyuniserver'") [Mon Dec 14 09:05:09.133965 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.133979 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] During handling of the above exception, another exception occurred: [Mon Dec 14 09:05:09.133990 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.134005 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Traceback (most recent call last): [Mon Dec 14 09:05:09.134034 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/proxy/apacheServer.py", line 62, in call [Mon Dec 14 09:05:09.134050 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] ret = f(req) [Mon Dec 14 09:05:09.134061 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/proxy/apacheHandler.py", line 368, in handler [Mon Dec 14 09:05:09.134071 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] ret = handlerObj.handler() [Mon Dec 14 09:05:09.134081 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 286, in handler [Mon Dec 14 09:05:09.134091 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] if not suseLib.accessible(checkURL): [Mon Dec 14 09:05:09.134102 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/lib/python3.6/site-packages/spacewalk/common/suseLib.py", line 271, in accessible [Mon Dec 14 09:05:09.134112 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] if e[0] == 56: # Proxy requires authentication [Mon Dec 14 09:05:09.134122 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] TypeError: 'error' object does not support indexing [Mon Dec 14 09:05:09.134141 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.134774 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] mod_wsgi (pid=2089): Exception occurred processing WSGI script '/usr/share/rhn/wsgi/xmlrpc.py'. [Mon Dec 14 09:05:09.136850 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Traceback (most recent call last): [Mon Dec 14 09:05:09.136926 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/lib/python3.6/site-packages/spacewalk/common/suseLib.py", line 269, in accessible [Mon Dec 14 09:05:09.136947 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] curl.perform() [Mon Dec 14 09:05:09.136982 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] pycurl.error: (60, "SSL: no alternative certificate subject name matches target host name 'uyuniserver'") [Mon Dec 14 09:05:09.137010 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.137024 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] During handling of the above exception, another exception occurred: [Mon Dec 14 09:05:09.137035 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.137054 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Traceback (most recent call last): [Mon Dec 14 09:05:09.137096 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/proxy/apacheServer.py", line 62, in call [Mon Dec 14 09:05:09.137113 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] ret = f(req) [Mon Dec 14 09:05:09.137133 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/proxy/apacheHandler.py", line 368, in handler [Mon Dec 14 09:05:09.137147 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] ret = handlerObj.handler() [Mon Dec 14 09:05:09.137166 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 286, in handler [Mon Dec 14 09:05:09.137180 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] if not suseLib.accessible(checkURL): [Mon Dec 14 09:05:09.137199 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/lib/python3.6/site-packages/spacewalk/common/suseLib.py", line 271, in accessible [Mon Dec 14 09:05:09.137226 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] if e[0] == 56: # Proxy requires authentication [Mon Dec 14 09:05:09.137258 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] TypeError: 'error' object does not support indexing [Mon Dec 14 09:05:09.137286 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.137300 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] During handling of the above exception, another exception occurred: [Mon Dec 14 09:05:09.137310 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] [Mon Dec 14 09:05:09.137329 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] Traceback (most recent call last): [Mon Dec 14 09:05:09.137378 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/wsgi/xmlrpc.py", line 23, in application [Mon Dec 14 09:05:09.137395 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] "proxy.apacheServer") [Mon Dec 14 09:05:09.137414 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/wsgi/wsgiHandler.py", line 44, in handle [Mon Dec 14 09:05:09.137428 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] ret = appServ(req) [Mon Dec 14 09:05:09.137447 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/share/rhn/proxy/apacheServer.py", line 68, in call [Mon Dec 14 09:05:09.137460 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] severity="unhandled") [Mon Dec 14 09:05:09.137479 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/lib/python3.6/site-packages/spacewalk/common/rhnTB.py", line 188, in Traceback [Mon Dec 14 09:05:09.137493 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] rhnMail.send(headers, outstring) [Mon Dec 14 09:05:09.137512 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/lib/python3.6/site-packages/spacewalk/common/rhnMail.py", line 54, in send [Mon Dec 14 09:05:09.137525 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] (headers, toaddrs) = __check_headers(headers) [Mon Dec 14 09:05:09.137544 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] File "/usr/lib/python3.6/site-packages/spacewalk/common/rhnMail.py", line 46, in __check_headers [Mon Dec 14 09:05:09.137557 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] toaddrs = to.split(',') [Mon Dec 14 09:05:09.137584 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] AttributeError: 'NoneType' object has no attribute 'split'

mcalmer commented 3 years ago

[Mon Dec 14 09:05:09.133857 2020] [wsgi:error] [pid 2089] [client 172.16.37.150:41640] pycurl.error: (60, "SSL: no alternative certificate subject name matches target host name 'uyuniserver'")

This sounds like a problem with your SSL certificates. Please always use FQDNs (name.domain.top) - everywhere! Otherwise you will hit tons of these problems all around your systems. A DNS server is a requirement anyway, so specify your own domain if you do not have one.

The other error seems to be a followup if the first error. But I will ask somebody to check it.

mcalmer commented 3 years ago

What does rpm -q python3-pycurl return executed on the proxy ?

milari commented 3 years ago

uyuniproxy:/var/log/apache2 # rpm -q python3-pycurl python3-pycurl-7.43.0.2-lp152.2.9.x86_64

mcalmer commented 3 years ago

And have you traceback_mail configured in /etc/rhn/rhn.conf?

milari commented 3 years ago

not yet

mcalmer commented 3 years ago

not yet

ok, that explains the last traceback. It try to send the error as email, but as no email address is defined it could not. But it should not crash :-)

milari commented 3 years ago

ok, so in the meanwhile, starting from absence of dns (my actual domain is example.it):

vzhestkov commented 3 years ago

@milari , could you please check with changing lines 271-272 at /usr/lib/python3.6/site-packages/spacewalk/common/suseLib.py to:

   271              if e.args[0] == 56:  # Proxy requires authentication
   272                  log_debug(2, e.args[1])
milari commented 3 years ago

Hi, after change: https://uyuniproxy.example.it:443/rhn/manager/download/centos7-uyuni-client-x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden

and from proxy apache2 logs: [Mon Dec 14 13:25:50.227728 2020] [wsgi:error] [pid 2091] [client 172.16.37.150:41756] CONFIG PARSE WARNING: File not found or can't be read /usr/share/rhn/config-defaults/rhn_server.conf [Mon Dec 14 13:25:50.227788 2020] [wsgi:error] [pid 2091] [client 172.16.37.150:41756] CONFIG PARSE WARNING: File not found or can't be read /usr/share/rhn/config-defaults/rhn_server_satellite.conf [Mon Dec 14 13:25:50.228147 2020] [wsgi:error] [pid 2091] [client 172.16.37.150:41756] CONFIG PARSE WARNING: key not found in config default dict ('server',) [Mon Dec 14 13:25:50.228187 2020] [wsgi:error] [pid 2091] [client 172.16.37.150:41756] CONFIG PARSE WARNING: key not found in config default dict ('server', 'satellite')

vzhestkov commented 3 years ago

@milari, are these messages from the uyuni proxy, right?

milari commented 3 years ago
vzhestkov commented 3 years ago

Is there proxy configured in the environment variables on the uyuni proxy system? If so add uyuni server to exclude list. The other possible items to check: Is mgr_server pillar pointing to the proxy server?: salt MINION_ID pillar.items mgr_server Also try to perform salt MINION_ID state.apply channels.

milari commented 3 years ago

Hi. first part of suggestion isn't really clear for me. I've followed setup guide during proxy install.

Second part, generally salt works on proxy managed minions; from master: labgraphite.dtc.local:

mgr_server:
    uyuniproxy.example.it

uyuniserver:/srv/www/htdocs/pub/bootstrap # salt labgraphite.dtc.local state.apply labgraphite.dtc.local: Data failed to compile:

State 'base' in SLS 'manager_org_1.telegraf_for_centos' is not formed as a list

ERROR: Minions returned with non-zero exit code

vzhestkov commented 3 years ago

@milari you tried to apply the highstate, not channels state. The last message is that manager_org_1.telegraf_for_centos state is malformed. execute the following: salt labgraphite.dtc.local state.apply channels provide the output if possible and if it changes anything.

milari commented 3 years ago

labgraphite.dtc.local:

      ID: mgrchannels_susemanagerplugin_yum
Function: file.managed
    Name: /usr/share/yum-plugins/susemanagerplugin.py
  Result: True
 Comment: File /usr/share/yum-plugins/susemanagerplugin.py is in the correct state
 Started: 17:57:58.185806
Duration: 151.482 ms
 Changes:
      ID: mgrchannels_susemanagerplugin_conf_yum
Function: file.managed
    Name: /etc/yum/pluginconf.d/susemanagerplugin.conf
  Result: True
 Comment: File /etc/yum/pluginconf.d/susemanagerplugin.conf is in the correct state
 Started: 17:57:58.337674
Duration: 48.65 ms
 Changes:
      ID: mgrchannels_repo
Function: file.managed
    Name: /etc/yum.repos.d/susemanager:channels.repo
  Result: True
 Comment: File /etc/yum.repos.d/susemanager:channels.repo updated
 Started: 17:57:58.387965
Duration: 157.959 ms
 Changes:
          ----------
          diff:
              ---
              +++
              @@ -1,6 +1,16 @@
               # Channels managed by SUSE Manager
               # Do not edit this file, changes will be overwritten
               #
              +
              +[susemanager:centos7-x86_64-updates]
              +name=CentOS 7 Updates (x86_64)
              +enabled=1
              +baseurl=https://uyuniproxy.example.it:443/rhn/manager/download/centos7-x86_64-updates
              +susemanager_token=eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2Mzg1MjA0OTksImlhdCI6MTYwNjk4NDQ5OSwibmJmIjoxNjA2OTg0Mzc5LCJqdGkiOiJCWnBYNlRpSXZJN2ZnRHRpNHZHbC1RIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiY2VudG9zNy14ODZfNjQtdXBkYXRlcyJdfQ.uhge2vyt6P0zVxFuDpQZ72zCKkws6ctP3n7eA0Z_mkU
              +gpgcheck=1
              +repo_gpgcheck=0
              +type=rpm-md
              +

               [susemanager:centos7-x86_64]
               name=CentOS 7 (x86_64)
              @@ -12,11 +22,11 @@
               type=rpm-md

              -[susemanager:centos7-x86_64-updates]
              -name=CentOS 7 Updates (x86_64)
              +[susemanager:influxdb-repo-centos7based-srvs]
              +name=influxdb repository centos7-based servers
               enabled=1
              -baseurl=https://uyuniproxy.example.it:443/rhn/manager/download/centos7-x86_64-updates
              -susemanager_token=eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2Mzg1MjA0OTksImlhdCI6MTYwNjk4NDQ5OSwibmJmIjoxNjA2OTg0Mzc5LCJqdGkiOiJCWnBYNlRpSXZJN2ZnRHRpNHZHbC1RIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiY2VudG9zNy14ODZfNjQtdXBkYXRlcyJdfQ.uhge2vyt6P0zVxFuDpQZ72zCKkws6ctP3n7eA0Z_mkU
              +baseurl=https://uyuniproxy.example.it:443/rhn/manager/download/influxdb-repo-centos7based-srvs
              +susemanager_token=eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2Mzg1MjA0OTksImlhdCI6MTYwNjk4NDQ5OSwibmJmIjoxNjA2OTg0Mzc5LCJqdGkiOiJkbDhYam9JX05GXzBvbWt2NGxaeEJnIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiaW5mbHV4ZGItcmVwby1jZW50b3M3YmFzZWQtc3J2cyJdfQ.Ghv7TyyXltjhBLx_C5lCUjY1v4IDEVbd3RJ6mUr6KxE
               gpgcheck=1
               repo_gpgcheck=0
               type=rpm-md
              @@ -32,13 +42,3 @@
               type=rpm-md

              -[susemanager:influxdb-repo-centos7based-srvs]
              -name=influxdb repository centos7-based servers
              -enabled=1
              -baseurl=https://uyuniproxy.example.it:443/rhn/manager/download/influxdb-repo-centos7based-srvs
              -susemanager_token=eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2Mzg1MjA0OTksImlhdCI6MTYwNjk4NDQ5OSwibmJmIjoxNjA2OTg0Mzc5LCJqdGkiOiJkbDhYam9JX05GXzBvbWt2NGxaeEJnIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiaW5mbHV4ZGItcmVwby1jZW50b3M3YmFzZWQtc3J2cyJdfQ.Ghv7TyyXltjhBLx_C5lCUjY1v4IDEVbd3RJ6mUr6KxE
              -gpgcheck=1
              -repo_gpgcheck=0
              -type=rpm-md
              -
              -
milari commented 3 years ago 
      ID: mgrchannels_yum_clean_all
Function: cmd.run
    Name: /usr/bin/yum clean all
  Result: True
 Comment: unless condition is true
 Started: 17:57:58.556144
Duration: 1960.974 ms
 Changes:

Summary for labgraphite.dtc.local

Succeeded: 4 (changed=1) Failed: 0

Total states run: 4 Total run time: 2.319 s

milari commented 3 years ago

Indeed, from client, after a yum update: https://uyunip.example.it:443/rhn/manager/download/centos7-uyuni-client-x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden [root@labgraphite ~]#

The same from proxy apache logs: [Mon Dec 14 18:02:39.283864 2020] [wsgi:error] [pid 2090] [client 172.16.37.150:41944] CONFIG PARSE WARNING: File not found or can't be read /usr/share/rhn/config-defaults/rhn_server.conf [Mon Dec 14 18:02:39.284143 2020] [wsgi:error] [pid 2090] [client 172.16.37.150:41944] CONFIG PARSE WARNING: File not found or can't be read /usr/share/rhn/config-defaults/rhn_server_satellite.conf [Mon Dec 14 18:02:39.284563 2020] [wsgi:error] [pid 2090] [client 172.16.37.150:41944] CONFIG PARSE WARNING: key not found in config default dict ('server',) [Mon Dec 14 18:02:39.284603 2020] [wsgi:error] [pid 2090] [client 172.16.37.150:41944] CONFIG PARSE WARNING: key not found in config default dict ('server', 'satellite')

Thanks a lot.

vzhestkov commented 3 years ago

Try to get the repomd.xml with the following command: curl -i https://uyuniproxy.example.it:443/rhn/manager/download/centos7-x86_64-updates/repodata/repomd.xml?eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2Mzg1MjA0OTksImlhdCI6MTYwNjk4NDQ5OSwibmJmIjoxNjA2OTg0Mzc5LCJqdGkiOiJCWnBYNlRpSXZJN2ZnRHRpNHZHbC1RIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiY2VudG9zNy14ODZfNjQtdXBkYXRlcyJdfQ.uhge2vyt6P0zVxFuDpQZ72zCKkws6ctP3n7eA0Z_mkU from the minion itself and from the proxy but replacing the hostname of the proxy with uyuni server's hostname

milari commented 3 years ago

from proxy -> server: uyunip:/tmp # curl -i https://uyuniserver.example.it:443/rhn/manager/download/centos7-x86_64-updates/repodata/repomd.xml?eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2Mzg1MjA0OTksImlhdCI6MTYwNjk4NDQ5OSwibmJmIjoxNjA2OTg0Mzc5LCJqdGkiOiJCWnBYNlRpSXZJN2ZnRHRpNHZHbC1RIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiY2VudG9zNy14ODZfNjQtdXBkYXRlcyJdfQ.uhge2vyt6P0zVxFuDpQZ72zCKkws6ctP3n7eA0Z_mkU HTTP/1.1 200 200 Date: Mon, 14 Dec 2020 17:13:21 GMT Server: Apache X-Frame-Options: SAMEORIGIN Content-Disposition: attachment; filename=repomd.xml Content-Type: application/octet-stream;charset=UTF-8 Set-Cookie: JSESSIONID=7BC98EE244DE72A0A4FF7CEB70D67A85; Path=/; Secure; HttpOnly; HttpOnly;HttpOnly;Secure Set-Cookie: pxt-session-cookie=758xa5251fed24967e78257b5bf4a7e44697a1b601b677e3df4f5ae7a779ff4996e4; Max-Age=0; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/; Secure; HttpOnly;HttpOnly;Secure Content-Security-Policy: default-src 'self' https: wss: ; script-src 'self' https: 'unsafe-inline' 'unsafe-eval'; img-src 'self' https: data: ;style-src 'self' https: 'unsafe-inline' X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff X-Permitted-Cross-Domain-Policies: master-only Last-Modified: Sun, 13 Dec 2020 20:30:22 GMT ETag: "5b65e651a3658" Content-Length: 1130

<?xml version="1.0" encoding="UTF-8"?>

9768923e32a209fc9b461f01e2751e5d9263e30864b229d6cd3cc424089657aa51898e19576ad9611607891422af89d39dd43784d324b44e57df2442e2e13de453e8670de46103addde286df8803781df868e5690516078914223fac583fdac061b6c9fbb38487a4accb356a86ed577dbdccf0586c2eed955b83fa645a41bdcbf6a216078914220b89c1ec5a8edec60f4b34d2548c02077fc76b7ce45a115815c86f8516afefbf5e382d4aab94d4181607891422

uyuniproxy:/tmp #

milari commented 3 years ago

from minion -> proxy: curl -i https://uyuniproxy.example.it:443/rhn/manager/download/centos7-x86_64-updates/repodata/repomd.xml?eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2Mzg1MjA0OTksImlhdCI6MTYwNjk4NDQ5OSwibmJmIjoxNjA2OTg0Mzc5LCJqdGkiOiJCWnBYNlRpSXZJN2ZnRHRpNHZHbC1RIiwib3JnIjoxLCJvbmx5Q2hhbm5lbHMiOlsiY2VudG9zNy14ODZfNjQtdXBkYXRlcyJdfQ.uhge2vyt6P0zVxFuDpQZ72zCKkws6ctP3n7eA0Z_mkU HTTP/1.1 403 Forbidden Date: Mon, 14 Dec 2020 17:15:35 GMT Server: Apache Content-Length: 0 Content-Type: text/xml

vzhestkov commented 3 years ago

It seems that the traceback was fixed with the modification I provided, but the rest issues are related to some issues with the setup. Most probably some of the packages missing on the proxy server.

vzhestkov commented 3 years ago

Try to apply the highstate for the proxy server: salt MINION_IF_OF_THE_PROXY state.apply

milari commented 3 years ago

It seems that the traceback was fixed with the modification I provided, but the rest issues are related to some issues with the setup. Most probably some of the packages missing on the proxy server.

But I currently use same packages with servers "without proxy" and they are fine!

vzhestkov commented 3 years ago

I mean the uyuni proxy is missing some of the configuration files most probably because some of required packages are not installed.

milari commented 3 years ago

Try to apply the highstate for the proxy server: salt MINION_IF_OF_THE_PROXY state.apply

I hope to have understand correctly your request... :) Thanks. M. proxy.txt

vzhestkov commented 3 years ago

@milari are there spacewalk-proxy-installer and spacewalk-backend packages installed on the proxy? Please attach the output of rpm -qa | grep -i spacewalk

milari commented 3 years ago

uyuniproxy:~ # rpm -qa | grep -i spacewalk spacewalk-proxy-docs-4.2.1-1.1.uyuni.noarch spacewalk-base-minimal-4.2.2-1.1.uyuni.noarch spacewalk-proxy-management-4.2.1-1.1.uyuni.noarch spacewalk-proxy-package-manager-4.2.1-1.1.uyuni.noarch spacewalk-ssl-cert-check-4.2.1-1.1.uyuni.noarch spacewalk-proxy-html-4.2.1-1.1.uyuni.noarch spacewalk-base-minimal-config-4.2.2-1.1.uyuni.noarch python3-spacewalk-certs-tools-4.2.1-1.1.uyuni.noarch spacewalk-proxy-broker-4.2.1-1.1.uyuni.noarch spacewalk-proxy-redirect-4.2.1-1.1.uyuni.noarch spacewalk-client-tools-4.2.3-2.1.uyuni.noarch spacewalk-backend-4.2.2-1.2.uyuni.noarch spacewalk-proxy-common-4.2.1-1.1.uyuni.noarch spacewalk-proxy-salt-4.2.1-1.1.uyuni.noarch python3-spacewalk-client-tools-4.2.3-2.1.uyuni.noarch spacewalk-setup-jabberd-4.2.1-1.1.uyuni.noarch spacewalk-certs-tools-4.2.1-1.1.uyuni.noarch spacewalk-proxy-installer-4.2.1-1.1.uyuni.noarch

mcalmer commented 3 years ago

When a client try to access a package in the proxy, the proxy issue a HEAD request to the server to test if the "token" has access to the package. If this failed for whatever reason the proxy return 403 as the authentication test failed.

You had also problems with the SSL certificate. Please check if there are still errors reported in apache error_log. If the Head request failed because of SSL errors, you will also get the 403. Check also what "rhn_parent" is configured on the proxy in /etc/rhn/rhn.conf . This is the hostname which is used to create the HEAD request. If you have configured here only the short hostname which is not part of the server certificate, you have the reason for possible SSL problems.

milari commented 3 years ago

Hi. Thanks for the hint! After changing in the proxy config the short with the fully qualified server name, something has changed (passing from 403 to 404). In every case, this may be an expected result, so please let me check some more things before. I'll let You know.

milari commented 3 years ago

Hi, as said, it seems working now: only a key hasn't been passed from master to proxy and I'm finding to understand why. In any way following new error_log of apache:

[Tue Dec 15 10:19:09.255135 2020] [wsgi:error] [pid 19271] [client 172.16.37.150:42436] CONFIG PARSE WARNING: File not found or can't be read /usr/share/rhn/config-defaults/rhn_server.conf [Tue Dec 15 10:19:09.255194 2020] [wsgi:error] [pid 19271] [client 172.16.37.150:42436] CONFIG PARSE WARNING: File not found or can't be read /usr/share/rhn/config-defaults/rhn_server_satellite.conf [Tue Dec 15 10:19:09.255590 2020] [wsgi:error] [pid 19271] [client 172.16.37.150:42436] CONFIG PARSE WARNING: key not found in config default dict ('server',) [Tue Dec 15 10:19:09.255634 2020] [wsgi:error] [pid 19271] [client 172.16.37.150:42436] CONFIG PARSE WARNING: key not found in config default dict ('server', 'satellite')

Really different from previous one! :) I'll let You know. Thanks. M.