(Status: error) for site behind Authelia

[cinderblockgames]

Does the site redirecting for authentication keep jump from being able to pick up that the site is up? I feel like 2XX and 3XX should be considered up, 4XX and 5XX considered down. (Or maybe have the ability to specify which status codes are considered up?)

[cinderblockgames]

Looks like this is because the HEAD method is returning a 405 Method Not Allowed.

[cinderblockgames]

Asking on Authelia's repo: https://github.com/authelia/authelia/discussions/4998

[daledavies]

Yep essentially 4xx or 5xx HTTP response codes are considered down, as per the underlying guzzle library. When I am able I do plan on adding some more options to the status checking feature, maybe an alternate URL or as you suggest a list of acceptable responses.

Thanks for reaching out, keep me posted on the response from authelia 😀

[cinderblockgames]

@daledavies Seems like the easiest approach (for here anyway) would be a way to override the request type per site. So, something like

        {
            "name": "Recipes",
            "url": "https://recipes.example.com",
            "description": "Recipes and meal planning",
            "icon": "tandoor.svg",
            "verb": "GET"
        }

[cinderblockgames]

@daledavies Don't suppose you want to add to the discussion?

[cinderblockgames]

Based on the discussion, it sounds like maybe the best thing to do is to allow the list of acceptable responses while also allowing a specification of if redirects should be followed? From my viewpoint, if the site is returning a 303 See Other, it's up for my use case. That's much more simplistic than actually allowing full authentication to be able to get into the site.

[daledavies]

Yes this feature was only intended to be a simple status check, I'm not going to implement authentication and passing a request body etc.

I'm happy to implement additional request methods and a list of acceptable response codes if I can though.

One other thing though, I had set the gizzle client options to allow redirects, so this shouldn't be a problem.

[daledavies]

Aside from this, would an alternate status URL help too? I think it might be beneficial for some cases.

[cinderblockgames]

@daledavies I do think that would help - being able to point to one place for the link and another for the status could let me try to set up a health check endpoint that's not protected by authentication.

[daledavies]

I'll see about implementing all the above then :)

[daledavies]

This will probably help with issue https://github.com/daledavies/jump/issues/53 and possible issue https://github.com/daledavies/jump/issues/56 too.

[daledavies]

I've added a change to allow for per-site status options as discussed above. I'll push a new release of Jump soon but just want to look at a few other issues first.

[daledavies]

This has been included in the latest v1.3.1 release today, you can pull this from Docker Hub using either the daledavies/jump:latest or daledavies/jump:v1.3.1 tags.

Thanks for reporting this, I'll close the issue for now but feel free to repoen or open a new one if needed :)

[cinderblockgames]

@daledavies Trying to test this - I added "request_method": "GET" to the sites.json config for this site, but I'm still getting an error. Are there logs that I can check to see what's happening? If I curl directly with GET, I do get a 200 OK back, so not sure what's going on.

[daledavies]

There aren't any verbose logs specific to the status checking, you'll get a mixture of nginx and php logs via docker logs. I can make it give us more logs though, I'll hack something into a test image tomorrow if I can to give us an idea of the response jump is getting.

Other than that remember docker has its own internal network so the true test would be to exec into the jump container and curl from there, if you haven't already.

[daledavies]

I've reopened this issue until we get this sorted, I appreciate your persistence 😀

[cinderblockgames]

Relevant bits:

/var/www/html # curl -v "https://recipes.example.com/invite/[redacted]"

< HTTP/1.1 302 Found
<a href="https://login.example.com/?rd=https%3A%2F%2Frecipes.example.com%2Finvite%2F[redacted]&amp;rm=GET">302 Found</a>

/var/www/html # curl -v "https://login.example.com/?rd=https%3A%2F%2Frecipes
.example.com%2Finvite%2F[redacted]&rm=GET"

< HTTP/1.1 200 OK

EDIT: From inside the Jump container.

[daledavies]

All my testing suggests that Jump correctly sends a GET request and also follows redirects, the status returned is from the final destination. I think though that by default guzzle will only follow 5 redirects so that could possibly be your problem.

Could you test with curl like before, but also paste the location header rather than the response body?

[daledavies]

It might also help if I set guzzle to add the referrer when redirecting, by default it does not and this works for everything I've tested with, but maybe authelia needs it.

[daledavies]

While testing with curl you could also tell it to follow redirects and limit the number to 5, will make it similar to what guzzle is joing in Jump...

curl -vL --max-redirs 5 "https://recipes.example.com/invite/[redacted]"

[daledavies]

I've pushed an image to Docker Hub you can test with, it should give some more information in docker logs. Specifically the eventual status code, the redirect history and the status code at each redirect.

Could you pull daledavies/jump:redirecttest and give it a spin? :)

[cinderblockgames]

So, this is interesting. Here's it with curl, which looks right:

curl -vL --max-redirs 5 "https://recipes.example.com/invite/[redacted]"
*   Trying [redacted]:443...
* Connected to recipes.example.com ([redacted]) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=example.com
*  start date: Mar  4 07:17:44 2023 GMT
*  expire date: Jun  2 07:17:43 2023 GMT
*  subjectAltName: host "recipes.example.com" matched cert's "*.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /invite/[redacted] HTTP/1.1
> Host: recipes.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< content-length: 154
< content-type: text/html; charset=utf-8
< date: Tue, 07 Mar 2023 16:28:16 GMT
< location: https://login.example.com/?rd=https%3A%2F%2Frecipes.example.com%2Finvite%2F[redacted]&rm=GET
< permissions-policy: interest-cohort=()
< referrer-policy: strict-origin-when-cross-origin
< set-cookie: authelia_session=[redacted]; expires=Tue, 07 Mar 2023 17:28:17 GMT; domain=example.com; path=/; HttpOnly; secure; SameSite=Lax
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=60;
< 
* Ignoring the response-body
* Connection #0 to host recipes.example.com left intact
* Issue another request to this URL: 'https://login.example.com/?rd=https%3A%2F%2Frecipes.example.com%2Finvite%2F[redacted]&rm=GET'
*   Trying [redacted]:443...
* Connected to login.example.com ([redacted]) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=example.com
*  start date: Mar  4 07:17:44 2023 GMT
*  expire date: Jun  2 07:17:43 2023 GMT
*  subjectAltName: host "login.example.com" matched cert's "*.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /?rd=https%3A%2F%2Frecipes.example.com%2Finvite%2F[redacted]&rm=GET HTTP/1.1
> Host: login.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 992
< content-security-policy: default-src 'self'; frame-src 'none'; object-src 'none'; style-src 'self' 'nonce-[redacted]'; frame-ancestors 'none'; base-uri 'self'
< content-type: text/html; charset=utf-8
< date: Tue, 07 Mar 2023 16:28:17 GMT
< permissions-policy: interest-cohort=()
< referrer-policy: strict-origin-when-cross-origin
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=60;
< 
<!DOCTYPE html>
<html lang="en">
<head>
  <base href="https://login.example.com/" />
  <meta property="csp-nonce" content="[redacted]" />
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta name="theme-color" content="#000000" />
  <meta name="description" content="Authelia login portal for your apps" />
  <link rel="manifest" href="./manifest.json" />
  <link rel="icon" href="./favicon.ico" />
  <title>Login - Authelia</title>
  <script type="module" crossorigin src="./static/js/index.fad9e36b.js"></script>
  <link rel="stylesheet" href="./static/css/index.40feef90.css">
</head>

<body
    data-basepath=""
    data-duoselfenrollment="false"
    data-logooverride="false"
    data-rememberme="true"
    data-resetpassword="true"
    data-resetpasswordcustomurl=""
    data-theme="light"
>
  <noscript>You need to enable JavaScript to run this app.</noscript>
  <div id="root"></div>

</body>
</html>
* Connection #1 to host login.example.com left intact

Here's are the PHP errors from the docker logs:

NOTICE: PHP message: PHP Warning:  Undefined property: stdClass::$default in /var/www/html/classes/Sites.php on line 92
2023/03/07 16:25:58 [error] 30#30: *3 FastCGI sent in stderr: "PHP message: PHP Warning:  Undefined property: stdClass::$default in /var/www/html/classes/Sites.php on line 92" while reading response header from upstream, client: 10.0.1.87, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm.sock:", host: "welcometo.lab.example.com"

NOTICE: PHP message: Site (https://recipes.example.com/invite/[redacted]) returned status of... 401
2023/03/07 16:26:00 [error] 30#30: *17 FastCGI sent in stderr: "PHP message: Site (https://recipes.example.com/invite/[redacted]) returned status of... 401" while reading response header from upstream, client: 10.0.1.87, server: _, request: "GET /api/status/94b68d17fb7a292d5c0f130303a0fd16fe363ef78f0e3ae1ef1456bf827d418e/ HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm.sock:", host: "welcometo.lab.example.com", referrer: "https://welcometo.example.com/"

So the app is seeing a 401 instead of the 302. Trying with "allowed_status_codes": [401]" instead.

[cinderblockgames]

That's working! Still don't know why the app is seeing just a straight 401 instead of the 302, but I'm happy. Thank you!

Edit: So nice to see greens across the board.

[daledavies]

Interesting, glad it's working for you. Out of interest can you share your full sites.json file? That first error is unrelated to this problem but I'm intrigued what caused it.

[cinderblockgames]

{
  "sites": [
    {
      "name": "Home",
      "url" : "https://welcometo.example.com",
      "description": "You are here",
      "icon": "jump.png"
    },
    {
       "name": "Media",
       "url" : "https://media.example.com",
       "description": "Watch media",
       "icon": "jellyfin.png"
    },
    {
       "name": "Requests",
       "url" : "https://requests.media.example.com",
       "description": "Request series and movies",
       "icon": "jellyseerr.svg"
    },
    {
       "name": "Recipes",
       "url": "https://recipes.example.com/invite/[redacted]",
       "description": "Recipes and meal planning",
       "icon": "tandoor.svg",
       "status": {
         "allowed_status_codes": [401]
        }
    },
    {
       "name": "Restaurants",
       "url" : "https://wiki.example.com/example/restaurants",
       "description": "Example  restaurant ratings",
       "icon": "wikijs.svg"
    },
    {
       "name": "Reset Password",
       "url" : "https://login.example.com/reset-password/step1",
       "description": "Reset your password",
       "icon": "authelia.jpg"
    }
  ]
}

[daledavies]

So the app is seeing a 401 instead of the 302. Trying with "allowed_status_codes": [401]" instead.

Perhaps a 401 might be correct, in terms of the request being unauthorized.

Also some differences between the curl and what Jump is doing, Curl wil be using the GET method where your configuration for Jump means it will use the default HEAD method for the request.

It looks like Authelia have implemented something to handle HEAD requests, but maybe they might respond differently depending on the request method.

[cinderblockgames]

If I do it with a HEAD request, I get:

HTTP/1.1 303 See Other
HTTP/1.1 405 Method Not Allowed

So still don't really know where the 401 comes from.

[daledavies]

NOTICE: PHP message: Site (https://recipes.example.com/invite/[redacted]) returned status of... 401
2023/03/07 16:26:00 [error] 30#30: *17 FastCGI sent in stderr: "PHP message: Site (https://recipes.example.com/invite/[redacted]) returned status of... 401" while reading response header from upstream, client: 10.0.1.87, server: _, request: "GET /api/status/94b68d17fb7a292d5c0f130303a0fd16fe363ef78f0e3ae1ef1456bf827d418e/ HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm.sock:", host: "welcometo.lab.example.com", referrer: "https://welcometo.example.com/"

If you we're not seeing anything in Jump's logs referring to redirect history or redirect status history then this suggests Authelia isn't responding with a redirect status at all, it's just straight up responding with a 401 Unauthorized code.

I'm trying to think why this might be. I've done some testing with guzzle in Jump and can see there are two differences in the request headers sent by Jump... guzzle doesn't set the Accept: */* header like curl and it uses a user agent of "GuzzleHttp/7" compared to curl's curl/7.81.0.

Anyway I can't help thinking that we might gain some insight from Authelia's own logs, can you see anything relevant in those when you do the curl vs Jump test?

[daledavies]

@cinderblockgames Did you find anything in the Authelia logs?

[cinderblockgames]

I haven't had a chance, but I'll try to look this week.