[BUG] Downstream proxy server is online with misconfigured settings

[mflagler]

Describe the bug Downstream Inductive Automation Server is working, but uptime monitor shows "Downstream proxy server is online with misconfigured settings"

To Reproduce Steps to reproduce the behavior:

1. Point HTTP Proxy to self-hosted Inductive Automation Gateway Server downloaded and installed from here: https://inductiveautomation.com/downloads/archive/8.1.39
2. Uptime monitor shows misconfiguration

Expected behavior Uptime monitor shows healthy running server

Screenshots

Host Environment (please complete the following information):

• Arch: Windows x64
  • Device: Windows VM for Testing
  • OS: Windows 11
  • Version: 23H2

Additional context Attached Wireshark capture with first part doing an edit/save of the HTTP proxy to force it to attempt to re-check the status/uptime. Starting at packet 43 I connected to the same URL (IP and port) as configured directly using Microsoft Edge to show how the browser packets look.

Ignition can be downloaded if wanted for testing by right-clicking the download and doing a save-as to bypass the registration form. A default install with standard edition and doing the initial setup (creating an admin user/password and specifying the ports to use) is all that's needed.

Zoraxy Uptime Monitor Capture.zip

[tobychui]

I have analysis your capture and found out that the capture didn't actually include the uptime monitor request. But anyway, from your screenshot, I think the downstream server is returning a status code in the 5xx range with the misconfigured as a badly chosen error message. I have updated the uptime monitor with better description and UI to reduce users confusion on the uptime status of their servers.

Will be closing this after next release.

[mflagler]

On my packet capture, packet 24 is the request from Zoraxy and in 25 it gets a 302 Found, but then Zoraxy sends on packet 28 a continuation which then triggers a response on packet 30 of a 400 Bad Request.

I suspect that the server is trying to set a cookie (JSESSIONID) as part of the 302 as seen in packets 11, 15, 19, and 23, but Zoraxy isn't sending the cookies with the redirect. On packet 53 when I do the same in Edge, it gets sent the cookie just the same, but on packet 54 of the redirect, Edge is sending the cookie and then finally gets a 200 OK response.

I think it's essentially getting stuck in a redirect loop because this session cookie isn't being set and it's expecting it to be.

[tobychui]

Hi @mflagler thanks for the further explanation!

I guess the packet 24 onward, although it maybe sent from Zoraxy but it is not the uptime checker. The uptime monitor will only report "Misconfigured" when the status code is in the 5xx range under the current implementation (3.0.4) https://github.com/tobychui/zoraxy/blob/2cdd5654ed898651482883ed08d9d9b79da095ef/src/web/components/uptime.html#L143

If you are sure the up-time is receiving a 4xx error, then the error code (e.g. Bad request) will be shown on the up-time monitor online status text. https://github.com/tobychui/zoraxy/blob/2cdd5654ed898651482883ed08d9d9b79da095ef/src/web/components/uptime.html#L148

Lastly, if your server reply 3xx, up-time will still assume it is online, so it clearly your screenshot does not match with what you have in the capture files. That is why I suspect that there are something running & connecting to your server but that "thing" is not the zoraxy up-time monitor checker.

[mflagler]

Here's another packet capture spanning several minutes showing the same result as the original packet capture. There's 2 checks it looks like by the uptime monitor in this capture, and both do a handful of redirect loops before sending a "Continuation" packet that causes the server to return a 400 bad request. I really do think the redirect loop wouldn't happen if cookies were passed that are received by the previous requests and all would be OK.

This is on v3.0.4 and I know the code doesn't say that's the status it will report, but that's how it's showing up on the uptime monitor.

Zoraxy Uptime Monitor Capture 2.zip

[tobychui]

Ok, I have checked your 2nd capture and trying to reproduce it with the following php script that emulates your capture response from your server.

<?php
// Set the necessary headers
header("HTTP/1.1 302 Found");
header("Date: Fri, 24 May 2024 21:33:56 GMT");
header("Referrer-Policy: strict-origin-when-cross-origin");
header("X-Content-Type-Options: nosniff");
header("X-Frame-Options: SAMEORIGIN");
header("X-XSS-Protection: 1; mode=block");
header("Location: http://192.168.1.120:8082/");
header("Transfer-Encoding: chunked");

// Optionally, you can include some content here, but usually for a 302 response, no content is necessary.
// To comply with the Transfer-Encoding: chunked, we need to output something even if it's empty.
echo "";
?>

For the location header, I tried to point it to sites with login redirect and normal static http site. Both shows up as "Online" in my environment. I double checked the code, except in a really rare case where the uptime indeed will return a 5xx error, is when the response received contains the http: server gave HTTP response to HTTPS client error.

https://github.com/tobychui/zoraxy/blob/6feb2d105dfd5789bfbe1f924a67e09124c80396/src/mod/uptime/uptime.go#L236

Updates

I guess I found the issue. This was from Golang net/http client request function. If CheckRedirect is nil, the Client uses its default policy, which is to stop after 10 consecutive requests. Which is exactly matching the capture file you posted earlier with 10 request and stopped. I will see if there are anything we can do.

[mflagler]

That's why I was thinking that because you're not passing the JSESSIONID cookie which the server is expecting, I think it's redirecting back to the URI that sets it. Not sure if there's other servers that do that, but I think just passing those cookies in the subsequent requests for that instance of the check that would stop the loop early on.

[tobychui]

@mflagler I think the back-end you are using is weird and as far as I know, it is not a common practice for how a cookie is set and used. If a user disable cookie in their browser, it will also create an infinite redirect loop which in general is a bad practice for website to have such design.

I have added a quick fix with a single-use cookie jar to make request with a one-time use cookie session. I guess your issue will be fixed in the v3.0.5 release.

[mflagler]

Yeah, I don't know how common it is either. The software running on the server is a popular industrial HMI software package used by a lot of companies.

[mflagler]

An alternative that might be more useful is I believe there's a health URI that's something like /Status/Ping (not certain on this as I'd have to look it up again), but could either check for 200 by default or a RegEx match or something that would essentially make it more capable of checking true service health rather than a status 200.

[tobychui]

@mflagler The issue has been fixed in v3.0.5. If you need advance features like custom health check path or RTT log, you can switch to some other projects that is designed specifically for tracking uptime like the Uptime Kuma. The uptime monitor inside Zoraxy is for a quick overview to see if there are any service is down (like, really offline and can't be connected), health check is kinda out of the scope of Zoraxy.

As always, Pull Requests are welcomed :)

[mflagler]

That fixed it. I did a wireshark capture again, and it doesn't get into the redirect loop at all now and uptime monitor shows online.