search

fedora-iot / greenboot

Generic Health Checking Framework for systemd
GNU Lesser General Public License v2.1
95 stars 29 forks source link

ostree.fedoraproject.org/iot/ Returns Forbidden, causing 01_update_platforms_check.sh failing #68

Open fancl20 opened 2 years ago

fancl20 commented 2 years ago

I'm not sure the issue should be created here or it's a reverse proxy configuration issue. The way it checks url valid is

HTTP_STATUS=$(curl -o /dev/null -Isw '%{http_code}\n' "$UPDATE_PLATFORM_URL" || echo "Unreachable")
if ! [[ $HTTP_STATUS == 2* ]] && ! [[ $HTTP_STATUS == 3* ]]; then
    URLS_WITH_PROBLEMS+=( "$UPDATE_PLATFORM_URL" )
fi

And access https://ostree.fedoraproject.org/iot/ will return 403. Although ostree updating is still working correctly.

Iolaum commented 2 years ago

When logging in to my rpi4 running fedora-iot I also noticed this issue:


Script '01_update_platforms_check.sh' FAILURE (exit code '1'). Continuing...
Boot Status is GREEN - Health Check SUCCESS
``
tonyjames commented 2 years ago

I am seeing this same failure. The script fails when the system is booting but runs successfully when the system is online.

$ sudo /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
We can connect to all update platforms
$ sudo rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
                   Version: 35.20220313.0 (2022-03-13T18:22:55Z)
                    Commit: 1571de66e257eb04bfe924611c3814ea8df847e8d5001022dc1b86de25603efb
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
LorbusChris commented 2 years ago

Maybe this runs before network is up? Could one of you check the journal to see where exactly the script fails? (there should be some echo output)

LorbusChris commented 2 years ago

Also, what hardware do you run this on? Does it have a RTC?

tonyjames commented 2 years ago

I am currently testing this on a Raspberry Pi 4 but I also see this same failure on a Supermicro system in my environment that has an Intel processor. Here is the relevant entry from the journal:

Nov 14 19:00:30 rpi.home.arpa 01_update_platforms_check.sh[853]: There are problems connecting with the following URLs:
Nov 14 19:00:30 rpi.home.arpa 01_update_platforms_check.sh[853]: https://ostree.fedoraproject.org/iot https://ostree.fedoraproject
.org/iot/mirrorlist
Nov 14 19:00:30 rpi.home.arpa greenboot[817]: Script '01_update_platforms_check.sh' FAILURE (exit code '1'). Continuing...

I see NetworkManager entries above this failure in the journal which seems to indicate that networking is online when this script runs.

fancl20 commented 2 years ago

I'm using it on J4125. It triggers when I ssh to the device and network is working properly.

> ssh fancl20@192.168.1.2
Script '01_update_platforms_check.sh' FAILURE (exit code '1'). Continuing...
Boot Status is GREEN - Health Check SUCCESS
Last login: Sat Mar 19 00:32:11 2022 from 192.168.1.171

> curl https://ostree.fedoraproject.org/iot/
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr>
<address>Apache Server at ostree.fedoraproject.org Port 443</address>
</body></html>

Also https://ostree.fedoraproject.org/iot/ returns 403 on any of my devices. I doubt this may be a regional issue caused by reverse proxy configuration.

fancl20 commented 2 years ago

Found curl https://ostree.fedoraproject.org/iot will return 301, which redirects to https://ostree.fedoraproject.org/iot/ and returns 403. Change the URL to https://ostree.fedoraproject.org/iot can pass the health check. Is this the expected behaviour?

LorbusChris commented 2 years ago

The URL seems to either be broken, or not correct. I'd expect it to be reachable though, but I guess rpm-ostree really only needs access to the mirrorlist to get access to the contents. @nullr0ute what say you? Should we just change the test to check availability of https://ostree.fedoraproject.org/iot/mirrorlist, or is there something to be fixed on the infra side?

In the meantime, what are the outputs of cat /etc/ostree/remotes.d/fedora-iot.conf and ostree remote show-url fedora-iot?

It triggers when I ssh to the device and network is working properly.

Note that the test isn't run when logging in, but when upgrading. It's only the MotD message previously generated by the test that is shown upon login.

tonyjames commented 2 years ago 
$ cat /etc/ostree/remotes.d/fedora-iot.conf
[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist

$ ostree remote show-url fedora-iot
https://ostree.fedoraproject.org/iot
fancl20 commented 2 years ago 
> cat /etc/ostree/remotes.d/fedora-iot.conf
[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist

> ostree remote show-url fedora-iot
https://ostree.fedoraproject.org/iot/
LorbusChris commented 2 years ago

For comparison, in the Fedora CoreOS case, both https://ostree.fedoraproject.org and https://ostree.fedoraproject.org/mirrorlist are accessible

sgallagher commented 2 years ago

For comparison, in the Fedora CoreOS case, both https://ostree.fedoraproject.org and https://ostree.fedoraproject.org/mirrorlist are accessible

Actually, it seems that https://ostree.fedoraproject.org is also not accessible. FTR, I'm seeing this issue with a recently-installed RPi 400.

Change the URL to https://ostree.fedoraproject.org/iot can pass the health check.

That sounds like a hack workaround rather than a fix. It's abusing the fact that the check treats a redirect response as successful rather than that it's actually returning valid data.

EZForever commented 1 year ago

Can confirm the issue still exists with a fresh install of Fedora IoT 37. The connection to the update servers are 100% fine (without any reverse proxy or other stuffs) but the check always fails.

[testuser@testhost ~]$ rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20221216.0 (2022-12-16T16:21:55Z)
               BaseCommit: d22ecec3117bf0a62dce309b62f386e1bfef6d29f1953087a187448ede2523e8
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: avahi cockpit cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-ostree cockpit-pcp cockpit-podman
                           nss-mdns pcp toolbox tuned

  fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20221216.0 (2022-12-16T16:21:55Z)
               BaseCommit: d22ecec3117bf0a62dce309b62f386e1bfef6d29f1953087a187448ede2523e8
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: avahi cockpit cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-ostree cockpit-pcp cockpit-podman
                           nss-mdns pcp toolbox tuned
[testuser@testhost ~]$ ls /etc/ostree/remotes.d/
fedora-iot.conf
[testuser@testhost ~]$ cat /etc/ostree/remotes.d/fedora-iot.conf 
[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist
[testuser@testhost ~]$ curl -I https://ostree.fedoraproject.org/iot/
HTTP/2 403 
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: same-origin
content-type: text/html; charset=iso-8859-1
date: Mon, 19 Dec 2022 15:27:06 GMT
server: Apache

[testuser@testhost ~]$ curl https://ostree.fedoraproject.org/iot/config
[core]
repo_version=1
mode=archive-z2
[testuser@testhost ~]$ /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh 
There are problems connecting with the following URLs:
https://ostree.fedoraproject.org/iot/
https://ostree.fedoraproject.org/iot/mirrorlist
[testuser@testhost ~]$

And, indeed, browsing to https://ostree.fedoraproject.org/iot/ gives a "Forbidden" error page.

This bug effectively makes the check bogus, and since there's no config option to skip any checks, we're now stuck with a error message that appears every time the system boots. Modify the script to check for https://ostree.fedoraproject.org/iot/config, as mentioned in #71, should fix the issue.

natterangell commented 1 year ago

Just chiming in to note that this is still an issue on a F37 IoT installation I deployed yesterday. I'm greeted with the same error as @Iolaum in comment #2 when connecting trough ssh.

Here's my installation as of now:

[user@localhost ~]$ sudo rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
                  Version: 37.20230302.0 (2023-03-02T19:39:29Z)
                   Commit: 624c983412c88e8d0f19f615ba87c01af47433d29cf24a2d8a0a0851ebb44d63
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A

  fedora-iot:fedora/stable/aarch64/iot
                  Version: 37.20221118.0 (2022-11-18T17:32:21Z)
                   Commit: d06a0d1d93cd2e3ebe1b744bb16c23087a001f3116579bdb0f22dedcf755250f
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
[user@localhost ~]$ sudo /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
There are problems connecting with the following URLs:
https://ostree.fedoraproject.org/iot/
[user@localhost ~]$

There is something weird with the URL in the mirror repo though, https://ostree.fedoraproject.org/iot/ returns 403:

[user@localhost ~]$ curl -I https://ostree.fedoraproject.org/iot/
HTTP/2 403 
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: same-origin
content-type: text/html; charset=iso-8859-1
date: Fri, 03 Mar 2023 09:32:16 GMT
server: Apache

while https://ostree.fedoraproject.org/iot/config returns 200

[user@localhost ~]$ curl -I https://ostree.fedoraproject.org/iot/config
HTTP/2 200 
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: same-origin
last-modified: Thu, 02 Jun 2022 01:14:11 GMT
etag: "26-5e06cb932551d"
accept-ranges: bytes
content-length: 38
apptime: D=185
x-fedora-proxyserver: proxy02.fedoraproject.org
x-fedora-requestid: ZAG-m2C6BxwWjJWdPy6mfAAAUQQ
date: Fri, 03 Mar 2023 09:32:11 GMT
server: Apache

So, to me it seems like the recent commit based on #93 doesn't work @miabbott, unless this is unrelated to that. My bash-skills are too limited to debug this.

miabbott commented 1 year ago

@natterangell I'm not able to reproduce this:

$ sshq -l root 192.168.122.140
Warning: Permanently added '192.168.122.140' (ED25519) to the list of known hosts.
Boot Status is GREEN - Health Check SUCCESS
Last login: Fri Mar  3 13:40:30 2023 from 192.168.122.1
[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20230302.0 (2023-03-02T19:41:35Z)
               BaseCommit: b5ccde3356e41fe5969269245b7be45c43c2faf77f947249566eb73db6b26381
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: audit strace

  fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20230217.0 (2023-02-17T14:30:01Z)
               BaseCommit: 8dde1800d7d3aeaf71e0737c9551e550114a20dd73046ba2e3b4d8aec3873f2e
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: audit
[root@localhost ~]# rpm -q greenboot
greenboot-0.15.4-1.fc37.x86_64
[root@localhost ~]#  /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
We can connect to all update platforms

Is the problem repeatable in your environment?

I agree that the infrastructure of the ostree repo is setup in such a way where you can get misleading HTTP error codes if you don't hit the right endpoint. I've thought about how to make the 01_update_platforms_check.sh more robust and created a Jira card for our team to investigate (https://issues.redhat.com/browse/THEEDGE-3108); I think I'll mirror the content of that card here for greater visibility.

miabbott commented 1 year ago

I think I'll mirror the content of that card here for greater visibility.

https://github.com/fedora-iot/greenboot/issues/98

natterangell commented 1 year ago

@natterangell I'm not able to reproduce this: Is the problem repeatable in your environment?

Well, this is my first and only client on Fedora IoT. It's reproducible in the sense that it happens consistently on every reboot and after a reinstall.

natterangell commented 1 year ago 
[user@localhost ~]$ sudo systemctl status greenboot-healthcheck.service
● greenboot-healthcheck.service - greenboot Health Checks Runner
     Loaded: loaded (/etc/systemd/system/greenboot-healthcheck.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/greenboot-healthcheck.service.d
             └─10-network-online.conf
     Active: active (exited) since Fri 2023-03-03 13:32:38 CET; 1 day 8h ago
    Process: 858 ExecStart=/usr/libexec/greenboot/greenboot check (code=exited, status=0/SUCCESS)
   Main PID: 858 (code=exited, status=0/SUCCESS)
        CPU: 1.249s

Mar 03 13:32:34 rockpro64 greenboot[858]: Script '00_required_scripts_start.sh' SUCCESS
Mar 03 13:32:35 rockpro64 01_repository_dns_check.sh[868]: All domains have resolved correctly
Mar 03 13:32:35 rockpro64 greenboot[858]: Script '01_repository_dns_check.sh' SUCCESS
Mar 03 13:32:38 rockpro64 greenboot[858]: Script '02_watchdog.sh' SUCCESS
Mar 03 13:32:38 rockpro64 greenboot[858]: Running Wanted Health Check Scripts...
Mar 03 13:32:38 rockpro64 greenboot[858]: Script '00_wanted_scripts_start.sh' SUCCESS
Mar 03 13:32:38 rockpro64 greenboot[858]: Script '01_update_platforms_check.sh' FAILURE (exit code '1'). Continuing...
Mar 03 13:32:38 rockpro64 greenboot[858]: Running Required Health Check Scripts...
Mar 03 13:32:38 rockpro64 greenboot[858]: Running Wanted Health Check Scripts...
Mar 03 13:32:38 rockpro64 systemd[1]: Finished greenboot-healthcheck.service - greenboot Health Checks Runner.
[user@localhost ~]$
miabbott commented 1 year ago

I just did a fresh install of Fedora IoT 37 and I'm still not able to reproduce this issue:

$ sshq -l root 192.168.122.92
Warning: Permanently added '192.168.122.92' (ED25519) to the list of known hosts.
Boot Status is GREEN - Health Check SUCCESS
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20221116.0 (2022-11-16T17:44:48Z)
                   Commit: 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
[root@localhost ~]# rpm -q greenboot
greenboot-0.15.2-2.fc37.x86_64
[root@localhost ~]# /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
We can connect to all update platforms

Even after upgrading to the latest commit, I'm still not able to reproduce this:

$ sshq -l root 192.168.122.92
Warning: Permanently added '192.168.122.92' (ED25519) to the list of known hosts.
Boot Status is GREEN - Health Check SUCCESS
Last login: Mon Mar  6 15:36:25 2023 from 192.168.122.1
[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20230303.0 (2023-03-03T21:23:06Z)
                   Commit: 03c69197e441419bb0633b464981079b1c579ff82f08e251a81aeb568d4bf117
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A

  fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20221116.0 (2022-11-16T17:44:48Z)
                   Commit: 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
[root@localhost ~]# rpm -q greenboot
greenboot-0.15.4-1.fc37.x86_64
[root@localhost ~]# /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
We can connect to all update platforms

Could you copy the script to a different system and see if the script works there? I wonder if there is something specific to your environment.

sgallagher commented 1 year ago

I can confirm that this error message no longer appears as of the update I performed today.

EZForever commented 1 year ago

Unfortunately the update does not work for my case, at least partially. Just updated my Fedora IoT 37, and got the latest version of 01_update_platforms_check.sh (checked with GitHub master branch), but the check still fails. However the problem list is shorter:

[testuser@testhost ~]$ rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20230405.0 (2023-04-05T14:31:37Z)
               BaseCommit: f55ecede16ee422c471fd723802313d660c1dac59721ee0712a392f0cee3c246
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: avahi cockpit cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-ostree cockpit-podman nss-mdns toolbox
                           tuned

  fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20230203.0 (2023-02-03T21:35:20Z)
               BaseCommit: 0e379e2798983a0ec8731611682dcf73648394535bdc57513f16ab8da1009426
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: avahi cockpit cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-ostree cockpit-podman nss-mdns toolbox
                           tuned
[testuser@testhost ~]$ cat /etc/ostree/remotes.d/fedora-iot.conf
[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist
[testuser@testhost ~]$ curl -I https://ostree.fedoraproject.org/iot/
HTTP/2 403 
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: same-origin
content-type: text/html; charset=iso-8859-1
date: Wed, 05 Apr 2023 16:37:53 GMT
server: Apache

[testuser@testhost ~]$ curl https://ostree.fedoraproject.org/iot/config
[core]
repo_version=1
mode=archive-z2
[testuser@testhost ~]$ /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
There are problems connecting with the following URLs:
https://ostree.fedoraproject.org/iot/
[testuser@testhost ~]$ rpm -q greenboot
greenboot-0.15.4-1.fc37.x86_64
[testuser@testhost ~]$

I don't really understand what's going on right now, and I can only assume that we have different formats for fedora-iot.conf, of which mine (shown above) contains the offending URL. If this really is the case, it might explain why some people are experiencing this problem while others aren't - those who upgraded from an older version of Fedora IoT might have older versions of fedora-iot.conf that are clear of problems.

miabbott commented 1 year ago

@EZForever I think what you are encountering is discussed in #98

madsl commented 1 year ago

Seen from the outside, this seems like a real non-issue. What's holding this up? Tried a freshly downloaded Fedora IOT ISO this weekend, and I still get this same 403 error, both before and after rpm-ostree upgrade. It makes the whole IOT spin look like alpha software, and gives the impression that this is something Fedora wont commit to if even such a (for the outsider) basic looking bug as this can stay unfixed for the most part of a year.

I "fixed" it by editing the fedora-iot.conf file and amending the url line to end with /iot/config - if that is the correct fix, why isn't it part of greenboot already?

adrienthebo commented 12 months ago

I just did a fresh install of Fedora IoT 37 and I'm still not able to reproduce this issue:

$ sshq -l root 192.168.122.92
Warning: Permanently added '192.168.122.92' (ED25519) to the list of known hosts.
Boot Status is GREEN - Health Check SUCCESS
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20221116.0 (2022-11-16T17:44:48Z)
                   Commit: 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
[root@localhost ~]# rpm -q greenboot
greenboot-0.15.2-2.fc37.x86_64
[root@localhost ~]# /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
We can connect to all update platforms

Even after upgrading to the latest commit, I'm still not able to reproduce this:

$ sshq -l root 192.168.122.92
Warning: Permanently added '192.168.122.92' (ED25519) to the list of known hosts.
Boot Status is GREEN - Health Check SUCCESS
Last login: Mon Mar  6 15:36:25 2023 from 192.168.122.1
[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20230303.0 (2023-03-03T21:23:06Z)
                   Commit: 03c69197e441419bb0633b464981079b1c579ff82f08e251a81aeb568d4bf117
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A

  fedora-iot:fedora/stable/x86_64/iot
                  Version: 37.20221116.0 (2022-11-16T17:44:48Z)
                   Commit: 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
[root@localhost ~]# rpm -q greenboot
greenboot-0.15.4-1.fc37.x86_64
[root@localhost ~]# /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh
We can connect to all update platforms

Could you copy the script to a different system and see if the script works there? I wonder if there is something specific to your environment.

I've isolated the reproduction case - it's specifically the trailing slash on the url that's causing this to fail. Examples follow:

Success case - no trailing slash ```ini [remote "fedora-iot"] url=https://ostree.fedoraproject.org/iot gpg-verify=true gpgkeypath=/etc/pki/rpm-gpg/ contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist ``` ```console [root@k4 ~]# bash -x /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh + set -e + REPOS_DIRECTORY=/etc/ostree/remotes.d + URLS_WITH_PROBLEMS=() + [[ ! -d /etc/ostree/remotes.d ]] + get_update_platform_urls + mapfile -t UPDATE_PLATFORM_URLS ++ grep -P -ho 'http[s]?.*' /etc/ostree/remotes.d/fedora-iot.conf + [[ 2 -eq 0 ]] + assert_update_platforms_are_responding + for UPDATE_PLATFORM_URL in "${UPDATE_PLATFORM_URLS[@]}" ++ curl -o /dev/null -Isw '%{http_code}\n' https://ostree.fedoraproject.org/iot + HTTP_STATUS=301 + [[ 301 == 2* ]] + [[ 301 == 3* ]] + for UPDATE_PLATFORM_URL in "${UPDATE_PLATFORM_URLS[@]}" ++ curl -o /dev/null -Isw '%{http_code}\n' https://ostree.fedoraproject.org/iot/mirrorlist + HTTP_STATUS=200 + [[ 200 == 2* ]] + [[ 0 -eq 0 ]] + echo 'We can connect to all update platforms' We can connect to all update platforms + exit 0 ```
Failure case - trailing slash ```ini [remote "fedora-iot"] url=https://ostree.fedoraproject.org/iot/ gpg-verify=true gpgkeypath=/etc/pki/rpm-gpg/ contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist ``` ```console [root@k4 ~]# bash -x /usr/lib/greenboot/check/wanted.d/01_update_platforms_check.sh + set -e + REPOS_DIRECTORY=/etc/ostree/remotes.d + URLS_WITH_PROBLEMS=() + [[ ! -d /etc/ostree/remotes.d ]] + get_update_platform_urls + mapfile -t UPDATE_PLATFORM_URLS ++ grep -P -ho 'http[s]?.*' /etc/ostree/remotes.d/fedora-iot.conf + [[ 2 -eq 0 ]] + assert_update_platforms_are_responding + for UPDATE_PLATFORM_URL in "${UPDATE_PLATFORM_URLS[@]}" ++ curl -o /dev/null -Isw '%{http_code}\n' https://ostree.fedoraproject.org/iot/ + HTTP_STATUS=403 + [[ 403 == 2* ]] + [[ 403 == 3* ]] + URLS_WITH_PROBLEMS+=("$UPDATE_PLATFORM_URL") + for UPDATE_PLATFORM_URL in "${UPDATE_PLATFORM_URLS[@]}" ++ curl -o /dev/null -Isw '%{http_code}\n' https://ostree.fedoraproject.org/iot/mirrorlist + HTTP_STATUS=200 + [[ 200 == 2* ]] + [[ 1 -eq 0 ]] + echo 'There are problems connecting with the following URLs:' There are problems connecting with the following URLs: + echo https://ostree.fedoraproject.org/iot/ https://ostree.fedoraproject.org/iot/ + exit 1 ```

This is extra tricky because there's a redirect from .../iot to .../iot/ so if you have the misfortune to check this in a browser you'll get an HTTP 304, and if you have a trailing slash you'll get an HTTP 403, so casual inspection makes this looks like both URLs fail when it's only .../iot/ that's failing.

If you're running into this, strip the trailing slash off the url in /etc/ostree/remotes.d/fedora-iot.conf and you won't need to do anything else.

If there's any way to reach the folks running ostree.fedoraproject.org, the combination of the redirection and the 403 on .../iot/ is a bit of a footgun; it would be appreciated if both of those pages could return something else likely to trigger failures.