search

uyuni-project / uyuni

Source code for Uyuni
https://www.uyuni-project.org/
GNU General Public License v2.0
432 stars 181 forks source link

Forbidden: Token does not provide access to channel #7671

Closed hbrown-uiowa closed 8 months ago

hbrown-uiowa commented 1 year ago

Problem description

We updated our test instance to 2023.09 and did fresh registrations of Ubuntu 22.04, Ubuntu 20.04, and Rocky 8 to ORG 2

All three systems register via bootstrap.sh just fine to org 2, but all attempts to pull packages are met with 403 Forbidden

The rhn_web_ui.log has lots of [ajp-nio-127.0.0.1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: Token does not provide access to channel followed by the channel name.

We can't patch any of the systems registered against it.

If I re-register one of the systems to org 1, then it can patch just fine.

Steps to reproduce

1. 2. 3. ...

Uyuni version

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : Uyuni Server Stable
Name           : Uyuni-Server-release
Version        : 2023.09-230900.207.8.uyuni3
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : Uyuni-Server-release-2023.09-230900.207.8.uyuni3.src
Summary        : Uyuni Server
Description    : 
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Uyuni proxy version (if used)

No response

Useful logs

No response

Additional information

No response

ppanon2022 commented 1 year ago

We're having the same problem. CentOS systems seem to still update fine after our Uyuni server update but Ubuntu systems are failing to download packages with the same token error on the server log. I tried to manually update (i.e. use salt cmd.run to download and apt install) the venv-salt-minion package on the client to the latest release because it was still on 3004-14.4.uyuni. However using 3006.0-15.1.uyuni didn't make a difference.

Is there any way this could have been broken by the changes for the Automated RHUI credential update? Seems like a long shot, but if it involves token maintenance, maybe it changed the tokens for Ubuntu systems somehow? Is there a shortcut for re-sync/regenerating tokens for Ubuntu clients that doesn't involve a re-registration?

hbrown-uiowa commented 1 year ago

@ppanon2022 we have run into this issue: https://github.com/uyuni-project/uyuni/issues/6842 and the solution was just to apply a highstate.

Unfortunately, applying the highstate doesn't fix it any more. I can only seem to get patches if I register against ORG 1.

I'm trying to replicate on a fresh install.

avshiliaev commented 1 year ago

Hey @hbrown-uiowa Thanks for your report! @lucidd does it seem like a known issue? I think there might've been something similar recently with the tokens.

hbrown-uiowa commented 1 year ago

I am able to replicate this on a fresh install.

Steps taken:

  1. Install opensuse Leap 15.5
  2. Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/install-uyuni.html
  3. Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/uyuni-server-setup.html
  4. create UYUNI as org 1
  5. create testorg as org 2
  6. set UYUNI to trust testorg
  7. /usr/bin/spacewalk-common-channels -u uyuni-admin -a amd64-deb ubuntu-2204-pool-amd64-uyuni ubuntu-2204-amd64-main-uyuni ubuntu-2204-amd64-main-updates-uyuni ubuntu-2204-amd64-main-security-uyuni ubuntu-2204-amd64-universe-uyuni ubuntu-2204-amd64-universe-updates-uyuni ubuntu-2204-amd64-uyuni-client
  8. Set all channels to public in UYUNI org
  9. Create an activation key in testorg with key 2-ubuntu-2204 
    
spacecmd {SSM:0}> activationkey_details 2-ubuntu-2204
Key:                    2-ubuntu-2204
Description:            ubuntu-22.04
Universal Default:      False
Usage Limit:            0
Deploy Config Channels: False
Contact Method:         default

Software Channels

ubuntu-22.04-pool-amd64-uyuni |-- ubuntu-2204-amd64-main-security-uyuni |-- ubuntu-2204-amd64-main-updates-uyuni |-- ubuntu-2204-amd64-main-uyuni |-- ubuntu-2204-amd64-universe-updates-uyuni |-- ubuntu-2204-amd64-universe-uyuni |-- ubuntu-2204-amd64-uyuni-client

Configuration Channels

Entitlements

System Groups

Packages



10. Generate the bootstrap script
11. Load an ubuntu 22.04 system, wget http://<server>/pub/bootstrap/bootstrap.sh; edit script to include activation key above and run it, sign salt key, run salt-call state.highstate (which has lots of errors too). Run apt update and observe lots of 403 forbidden errors.

If I then repoint it to the the activation key for the UYUNI org (org 1), then it works fine.
ppanon2022 commented 1 year ago

@ppanon2022 we have run into this issue: #6842 and the solution was just to apply a highstate.

Unfortunately, applying the highstate doesn't fix it any more. I can only seem to get patches if I register against ORG 1.

I'm trying to replicate on a fresh install.

I think I may have seen that and did try to apply a high state already. Applying the high state works fine with no errors, but I still get the 403 failure.

The interesting thing is that we only have the one Org. Nothing fancy with trusts, but we still have that with systems joined to Org1. So there's no way for us to switch between orgs to a different org with a different activation key. I'm wondering if the fact that it's the last org in both our cases points to an off-by-one error on a loop or test somewhere. Because there seems to be a null org 

# ls /var/spacewalk/packages/
1  NULL

maybe that's caused an issue for an undercount?

lucidd commented 1 year ago

@avshiliaev

Hey @hbrown-uiowa Thanks for your report! @lucidd does it seem like a known issue? I think there might've been something similar recently with the tokens.

what i can already say its a different issue from what i was working on given the error message and that it also happens on non ubuntu systems.

@hbrown-uiowa

I am able to replicate this on a fresh install.

Steps taken:

  1. Install opensuse Leap 15.5
  2. Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/install-uyuni.html
  3. Follow process at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/uyuni-server-setup.html
  4. create UYUNI as org 1
  5. create testorg as org 2
  6. set UYUNI to trust testorg
  7. /usr/bin/spacewalk-common-channels -u uyuni-admin -a amd64-deb ubuntu-2204-pool-amd64-uyuni ubuntu-2204-amd64-main-uyuni ubuntu-2204-amd64-main-updates-uyuni ubuntu-2204-amd64-main-security-uyuni ubuntu-2204-amd64-universe-uyuni ubuntu-2204-amd64-universe-updates-uyuni ubuntu-2204-amd64-uyuni-client
  8. Set all channels to public in UYUNI org
  9. Create an activation key in testorg with key 2-ubuntu-2204
spacecmd {SSM:0}> activationkey_details 2-ubuntu-2204
Key:                    2-ubuntu-2204
Description:            ubuntu-22.04
Universal Default:      False
Usage Limit:            0
Deploy Config Channels: False
Contact Method:         default

Software Channels
-----------------
ubuntu-22.04-pool-amd64-uyuni
 |-- ubuntu-2204-amd64-main-security-uyuni
 |-- ubuntu-2204-amd64-main-updates-uyuni
 |-- ubuntu-2204-amd64-main-uyuni
 |-- ubuntu-2204-amd64-universe-updates-uyuni
 |-- ubuntu-2204-amd64-universe-uyuni
 |-- ubuntu-2204-amd64-uyuni-client

Configuration Channels
----------------------

Entitlements
------------

System Groups
-------------

Packages
--------
  1. Generate the bootstrap script
  2. Load an ubuntu 22.04 system, wget http:///pub/bootstrap/bootstrap.sh; edit script to include activation key above and run it, sign salt key, run salt-call state.highstate (which has lots of errors too). Run apt update and observe lots of 403 forbidden errors.

If I then repoint it to the the activation key for the UYUNI org (org 1), then it works fine.

does it only happen with the trusted org setup or is it also reproducible with just a simple single org setup? if not i assume there might be a problem related to trusted orgs.

hbrown-uiowa commented 1 year ago

I have assumed it is fine with just a single org (since registering to ORG 1 is fine both on our longstanding test instance and the one I spun up to replicate the issue).

Do you want me to try and test with just setting up ORG1 and nothing else?

For our production environment, collapsing the orgs is a non-starter. This is a blocker for us updating the production environment (which is limping along due to other bugs).

ppanon2022 commented 1 year ago

We are experiencing it with a single organization, although I don't have a way of reproducing it. When I was doing a spacewalk-data-fsck, I was getting a lot of errors about CentOS 7 aarch64 packages (from a product channel I had tried to delete) being in NULL instead of 1 (guessing based on the file path), so maybe that also triggered the bug or is related somehow. For example, a couple of lines from the spacewalk-data-fsck output:

File path mismatch: /var/spacewalk/packages/NULL/5e6/hsqldb-demo/1:1.8.1.3-14.el7/noarch/5e6215650dac3faf700f8f408a6384eaff94a043040afcdf361721af6260a 220/hsqldb-demo-1.8.1.3-14.el7.noarch.rpm (org_id: 1 vs. NULL) File path mismatch: /var/spacewalk/packages/NULL/437/maven-jar-plugin-javadoc/2.4-8.el7/noarch/4378c2a6d7352cd15487ad009992da250c7615e9bfc1653204aa999 c87a40b15/maven-jar-plugin-javadoc-2.4-8.el7.noarch.rpm (org_id: 1 vs. NULL)

If that is relevant then perhaps the issue is in the query that is pulling whatever token applies for that org somehow, although why it would apply to Ubuntu systems/channels and not CentOS is a mystery to me. But maybe the DB metadata for those packages or their rump product channel (that I can't seem to delete) is affecting a SQL query result in the same way that a second trusted channel does.

ppanon2022 commented 1 year ago

@avshiliaev You've probably already thought of this, but since I would expect this part of the code base to not need a lot of updating, there may be a fairly small number of code or schema change checkins since 2023.06 which could affect this scenario. Perhaps it might be useful to identify and review checkins to that functionality (token authentication and related references/dependencies) in the last 3 months? It definitely was working with 2023.06. I don't know what parts of the code base that might involve so I don't really know where to start on such a search for checkins.

If you find a query that you suspect it might be related to, I can certainly run that query on our database to return (sanitized) results.

hbrown-uiowa commented 1 year ago

It is working on 2023.04 (our production environment)

I re-ran my reproducer process but this time, I did not set any org trusts and I pulled in the ubuntu channels into org2. The bootstrap.sh didn't work because of apt key problems for an ubuntu 22.04 client, but after switching from venv-salt to regular salt, I was able to get bootstrapped and to patch.

It does appear to be something in the trusts (at least for me).

ppanon2022 commented 1 year ago

Because we have some packages appearing to be associated with a NULL org maybe there is some kind of equivalent implicit trust in our configuration/database as well. Perhaps something for that was changed in the rpm repo/minion handling code but without parallel changes in the deb repo/minion handling code. Or a mistake was made in a change to the deb/apt-related code that wasn't made in the rpm/yum handling code.

ppanon2022 commented 1 year ago

Should Uyuni even be expecting tokens for Ubuntu clients? It's been a few days since I was investigating, but I thought I remembered seeing that the token was kept in a client config file setting, and I couldn't find that setting on an Ubuntu minion. Which modules/methods on the server side handle the https calls and check for tokens?

ppanon2022 commented 1 year ago

Alright, the webui log error appears to be generated in DownloadController.java

           Opt.consume(orgClaim, () -> {
                LOG.info("Forbidden: Token does not specify the organization");
                halt(HttpStatus.SC_BAD_REQUEST, "Token does not specify the organization");
            }, orgId -> {
==>             if (!ChannelFactory.isAccessibleBy(channel, orgId)) {
                    LOG.info("Forbidden: Token does not provide access to channel {}", channel);
                    halt(HttpStatus.SC_FORBIDDEN, "Token does not provide access to channel " + channel);
                }

and that call in ChannelFactory runs a SQL Query 

    /**
     * Checks if a channel is accessible by an Org.
     * @param channelLabel the channel label
     * @param orgId the Org ID
     * @return true if it is accessible
     */
    public static boolean isAccessibleBy(String channelLabel, Long orgId) {
        return (int)singleton.lookupObjectByNamedQuery("Channel.isAccessibleBy",
                Map.of("channel_label", channelLabel, ORG_ID, orgId)) > 0;
    }

and that query is

   <sql-query name="Channel.isAccessibleBy">
        <return-scalar type="int" column="result"/>
         <![CDATA[SELECT case when (EXISTS (
              SELECT 1
              FROM rhnChannel c
              JOIN rhnChannelFamilyMembers cfm ON cfm.channel_id = c.id
              JOIN rhnPrivateChannelFamily pcf ON pcf.channel_family_id = cfm.channel_family_id
              WHERE c.label = :channel_label
              AND pcf.org_id = :org_id
              LIMIT 1
        ) OR EXISTS (
              SELECT 1
              FROM rhnChannel c
              JOIN rhnChannelFamilyMembers cfm ON cfm.channel_id = c.id
              JOIN rhnPublicChannelFamily pcf ON pcf.channel_family_id = cfm.channel_family_id
              WHERE c.label = :channel_label
              LIMIT 1
        ) OR EXISTS (
              SELECT 1
              FROM rhnChannel c
              JOIN rhnTrustedOrgs tr ON c.org_id = tr.org_id
              WHERE c.parent_channel IS NULL AND c.channel_access = 'public'
              AND c.label = :channel_label
              AND tr.org_trust_id = :org_id
              LIMIT 1
        ) OR EXISTS (
              SELECT 1
              FROM rhnChannel c
              JOIN rhnChannelTrust tr ON c.id = tr.channel_id
              WHERE c.channel_access = 'protected'
              AND c.label = :channel_label
              AND tr.org_trust_id = :org_id
              LIMIT 1
        )) then 1 else 0 end AS result
        ]]>
    </sql-query>

I'll bet dollars to donuts that this commit is responsible. https://github.com/uyuni-project/uyuni/commit/cb6ed8b9a5f410f03225528854b3b32d7a74fdf5

So the next step is to run that query in our environment, substituting an Ubuntu channel label and the org_id and see what is being returned and why. Hbrown, can you get a chance to do that in your environment? The result is presumably 0, whereas one of those subselects should probably be returning a value. So our next step is to figure out what that data looks like, why it's not being matched for those Ubuntu channels but is for the ones that work, and how to fix it.

ppanon2022 commented 1 year ago

Oh, well that's how to solve hbrown's issue. You can compare the results of running the different subselects for working channels and broken ones against the uyuni database with psql to see what is failing to work. Assuming the proper associations are in place, that should make it possible to figure out what change in the query is needed. The subselect SELECT 1 FROM rhnChannel c JOIN rhnTrustedOrgs tr ON c.org_id = tr.org_id WHERE c.parent_channel IS NULL AND c.channel_access = 'public' AND c.label = :channel_label AND tr.org_trust_id = :org_id LIMIT 1 which involves the trusted orgs table seems to be likely the one that needs to work/be fixed. Or maybe you need to change the channel_access to public.

But after trying to find out what query parts work and what don't and finding no difference between query results for the different channels, I just realized our error is critically different.

Our error in rhn_web_ui.log is a close but different error message: com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access

And that error appears to be because the client is not supplying a token in the request, failing the getTokenFromRequest in the DownloadPackage method in the same DownloadController.java module.

    if (checkTokens) {
        String token = getTokenFromRequest(request);
        validateToken(token, channelLabel, filename);
    }

so I've opened a discussion or a different ticket instead. @avshiliaev, if you or cbosdo could take a look at https://github.com/uyuni-project/uyuni/discussions/7710 and answer some of my questions, I would really appreciate the help. At this point I don't know if it's a bug or a configuration issue, but I really need to find a solution ASAP.

Good luck with your problem hbrown.

mcalmer commented 1 year ago

@parlt91 this sounds like the issue we solved yesterday. Do you have already a PR for uyuni which can be linked here?

parlt91 commented 1 year ago

@parlt91 this sounds like the issue we solved yesterday. Do you have already a PR for uyuni which can be linked here?

PR to address the issue is here: https://github.com/uyuni-project/uyuni/pull/7711 we'll merge once we received positive feedback that it'll fix the issue.

parlt91 commented 1 year ago

PR is merged now

hbrown-uiowa commented 11 months ago

I believe this is working for us now after upgrading to 2023.10

marmau06 commented 11 months ago

Hello, This morning, I updated Uyuni server to 2023.10, Debain do not work, again... I upgraded venv-salt-minion to last version venv-salt-minion_3006.0-21.8.uyuni_amd64.deb on uyuni Debian 11 client. Do not work again... I removed and re-registered the client, the same...

I still have this kind of message in rhn_web_ui.log:

2023-11-30 11:42:33,192 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libxml2_2.9.10 dfsg-6.7 deb11u4.amd64-deb.deb 2023-11-30 11:42:33,202 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-6] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/bind9-dnsutils_1:9.16.44-1~deb11u1.amd64-deb.deb 2023-11-30 11:42:33,206 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-9] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/bind9-libs_1:9.16.44-1~deb11u1.amd64-deb.deb 2023-11-30 11:42:33,211 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-5] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/bind9-host_1:9.16.44-1~deb11u1.amd64-deb.deb 2023-11-30 11:42:33,215 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-10] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/file_1:5.39-3 deb11u1.amd64-deb.deb 2023-11-30 11:42:33,220 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-7] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libmagic1_1:5.39-3 deb11u1.amd64-deb.deb 2023-11-30 11:42:33,225 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-8] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libmagic-mgc_1:5.39-3 deb11u1.amd64-deb.deb 2023-11-30 11:42:33,228 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/dnsutils_1:9.16.44-1~deb11u1.all-deb.deb 2023-11-30 11:42:33,233 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-2] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libaprutil1_1.6.1-5 deb11u1.amd64-deb.deb 2023-11-30 11:42:33,237 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-3] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libapr1_1.7.0-6 deb11u2.amd64-deb.deb 2023-11-30 11:42:33,242 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO com.suse.manager.webui.controllers.DownloadController - Forbidden: You need a token to access /manager/download/debian-11-amd64-main-security-uyuni/getPackage/libnss3_2:3.61-1 deb11u3.amd64-deb.deb

Does any body fixed this issue with 2023.10? What did I forget ?

Thanks in advance, it's getting more urgent for us now, we are supposed to patch servers once a month... And we are already two monthes late....

marmau06 commented 11 months ago

If this can help, "wget" do not work on a simple package, we get 403 error...

wget https://myserver.france.mydomain:443/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb --2023-11-30 14:19:48-- https://myserver.france.mydomain/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb Resolving proxy.france.mydomain (proxymop.france.rexel)... 10.3.204.8 Connecting to proxy.france.mydomain (proxy.france.mydomain)|10.3.204.8|:8080... connected. Proxy request sent, awaiting response... 403 403 2023-11-30 14:19:48 ERROR 403: 403.

But "curl" works fine on the same package...

curl -o /tmp/herve https://myserver.france.mydomain/rhn/manager/download/debian-11-pool-amd64-uyuni/getPackage/shim-helpers-amd64-signed_1%2b15.7%2b1%7edeb11u1-X.amd64-deb.deb % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4089 100 4089 0 0 159k 0 --:--:-- --:--:-- --:--:-- 159k

santeri3700 commented 11 months ago

There is a separate issue related to partially invalid repodata on DEB channels which causes similar token/403 errors. More details on https://github.com/uyuni-project/uyuni/issues/7788.

mcalmer commented 8 months ago

seems the original issue is solved and the second has its own issue. Closing