Localproxy fails to load web assets

[bryanlamtoo]

Connection reset by peer while reading response header from upstream, client 3.7.0.17.

Web Assets fail to load in cockpit when accessed directly through the localproxy on the source device

To Reproduce

Steps to reproduce the behavior:

1. Go to the AWS console and create an Iot tunnel. Add a service e.g WEB
2. On the destination device running Centos 8.3.2011 & cockpit-224.2-1.el8, configure cockpit to allow remote hosts by modifying the /etc/cockpit/cockpit.conf file's origins section
3. On the destination device , start the localproxy v2 in destination mode with WEB=9090 to forward requests to cockpit
4. On the source device running Ubuntu-20.04 with nginx=1.18.0-0ubuntu1 reverse proxying to port 5400, start the localproxy v2 in source mode with WEB=5400
5. Access cockpit from the browser in the source device.
6. Cockpit starts to load but the web assets fail to load.
7. Error logs below

Expected behavior

Cockpit loads with no errors and all assets loaded correctly

Actual behavior

Cockpit login page loads while the web assets i.e. css & javascript files fail to load

Logs Source localproxy logs

Destination localproxy logs

Nginx logs

Environment: Source Device

• OS Ubuntu (docker)
• Version 20.04
• Architecture: [x86-64]
• Nginx 1.18.0-0ubuntu1

Destination Device

• OS: Centos

• Version 8.3.2011

• Cockpit 224.2-1.el8

• Localproxy commit: c46dc0ea9bb7825b559c8d64ac2c57c262536b5b

Additional context Everything works if an SSH connection is set up to the destination device via the localproxy and a reverse tunnel to cockpit is created instead of accessing cockpit directly. I suppose this is because the connection to cockpit is masked by the encrypted SSH connection.

I suspect a server in AWS is not set up to handle the direct requests for the assets properly.

NOTE: Cockpit loads fine when accessed locally on the destination device or remotely via the device hostname

[kareali]

@bryanlamtoo Just to rule out Nginx configuration issues, what happens when you try to access cockpit via the source localproxy directly? For example, When you visit the web page from port 5400 instead of hitting the port to which Cockpit is listening.

[bryanlamtoo]

Good suggestion. So I did exactly that but the assets still fail to load

Here is a screen shot

[kareali]

I see, I'll try to see if I can reproduce this, I haven't used cockpit before though, do you mind sharing cockpit configurations file?

[bryanlamtoo]

Yeah sure! Here you go;

[WebService]
Origins = https://[your-origin-here] wss://[your-origin-here]

ProtocolHeader = X-Forwarded-Proto 

AllowUnencrypted = true 

You must specify the origin server where your requests will be coming from e.g https://[ip-address]:[port] or https://domain-name

[kareali]

I was able to get Cockpit to work without localproxy and then reproduced the same behavior after adding localproxy (without Nginx), I don't see indications for package/loss or being dropped either by localproxy or the cloud service itself 2 things looks suspicious to me

1. I noticed that the login page takes a lot to load and the cockpit page send a few libs/files in short duration so it's likely to try to exceed that tunnel bandwidth limit, it could be the case that cockpit has issues working with low bandwidth (a tunnel has a bandwidth limit of 100 KB/sec).
2. It seems that cockpit has auth mechanism and that includes validating the Host header and because when you access it via localproxy the host header is not the real one, cockpit either resets the connection or respond with 403 when I retry request a few seconds later(I'm guessing the cookie expires somehow) but when I copy the jquery.js network call as cURL (if I do it quickly before it expires) and send it via the terminal without the Host header (in the browser it's atomically set in my case it was set to localhost:5555). So maybe you could try using Nginx to re-write the headers appropriately so that Cockpit recognizes it.

[bryanlamtoo]

Thanks @kareali for going through this exercise.

Referring to the statement "reproduced the same behavior", do you mean the failure in loading web assets or the working behavior when localproxy is not involved.

Nonetheless, I'll go ahead and use Nginx to re-write the headers and see what happens though I think I did this before.

[kareali]

@bryanlamtoo When localproxy is not involved at all, everything works fine (The login page loads, I enter my credentials, and I go to the landing page)

When I add localproxy, the web assets load for the login page (but they take way too long which why I'm suspecting it's possible that cockpit is having trouble with the limited bandwidth), I enter my credentials, press enter, then it takes a few seconds until the web assets finally fail. The reason I'm suspecting the auth mechanism is because if I were to copy the jQuery request as cURL from chrome (I didn't pick jQuery request for any specific reason), and try it on my terminal, I get different results:

1. If use the same cURL request I copied without modifications, while the page is still loading, I get the reset connection error.
2. If use the cURL request I copied but update the host header to use the host DNS name instead of (localhost:5555), while the page is still loading, I get the reset connection error.
3. If I wait a few seconds until the chrome page fails, and try the same cURL in my terminal, I get 4xx error (Not Authorized)

[bryanlamtoo]

@kareali Can you confirm that the bandwidth limit is 100KB/sec and not 800KB/sec? I recently saw 800KB/sec in the AWS documentation.

And if it is 100KB/sec, can this be increased or is it fixed?

[kareali]

@bryanlamtoo It's 800 kbps (kilo bit per seconds) or 100 KB/sec (kilo byte per second), 1 byte = 8 bit (https://docs.aws.amazon.com/general/latest/gr/iot_device_management.html#limits_iot_secure_tunneling). Currently it's a fixed limit.

[bryanlamtoo]

Got it. Hadn't realized it's kbps and not KB/sec.

[kareali]

I ruled out bandwidth issues by testing in an environment where I could increase the bandwidth. Also I checked the logs for both the client and cloud service, no traffic is being dropped.

[bryanlamtoo]

Thanks @kareali. That's very helpful. I'll investigate cockpit a little bit more to see if I can find something.