[BUG] Pages unable to pull remote server config.

tuxpowered commented 2 months ago

Environment

Self-Hosted (Docker)

System

docker

Version

current

Describe the problem

Trying to add a page from a remote dashy install, unfortunately it errors out. with the dialog

Dashy has failed to load correctly due to a configuration error.

Ensure that
The configuration file can be found at the specified location
There are no CORS rules preventing client-side access
The YAML is valid, parsable and matches the schema
Error Details
Unable to load config from 'https://username:password@dashy.DOMAIN.COM/conf.yml'
Next Steps
Check the browser console for more details ([see how](https://github.com/Lissy93/dashy/blob/master/docs/troubleshooting.md#how-to-open-browser-console))
View the [Troubleshooting Guide](https://github.com/Lissy93/dashy/blob/master/docs/troubleshooting.md) and [Docs](https://dashy.to/docs/)
If you've verified the config is present, accessible and valid, and cannot find the solution in the troubleshooting, docs or GitHub issues, then [open a ticket on GitHub](https://github.com/Lissy93/dashy/issues/new/choose)
Click 'Ignore Critical Errors' below to not show this warning again

Looking at the browser console, I get this error, however....

Access to XMLHttpRequest at 'https://username:password@dashy.DOMAIN.COM/conf.yml' from origin 'http://172.31.10.227:8004' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.Understand this error
CoolConsole.js:11

The remote dashy is protected by HTTP Basic handled by Nginx Proxy Manager (NPM). When I tried passing the credentials in the url https://username:password@dashy.DOMAIN.COM/conf.yml from an unauthenticated browser it works fine no problem.

* Host dashy.DOMAIN.COM:443 was resolved.
* IPv6: (none)
* IPv4: X.X.X.X
*   Trying X.X.X.X:443...
* Connected to dashy.DOMAIN.COM (X.X.X.X) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=dashy.DOMAIN.COM
*  start date: Sep  9 16:34:20 2024 GMT
*  expire date: Dec  8 16:34:19 2024 GMT
*  subjectAltName: host "dashy.DOMAIN.COM" matched cert's "dashy.DOMAIN.COM"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
* using HTTP/2
* Server auth using Basic with user 'username'
* [HTTP/2] [1] OPENED stream for https://username:password@dashy.DOMAIN.COM/conf.yml
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: dashy.DOMAIN.COM]
* [HTTP/2] [1] [:path: /conf.yml]
* [HTTP/2] [1] [authorization: Basic BASICAUTHHASH]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /conf.yml HTTP/2
> Host: dashy.DOMAIN.COM
> Authorization: Basic BASICAUTHHASH
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< server: openresty
< date: Fri, 20 Sep 2024 18:02:32 GMT
< content-type: text/yaml; charset=UTF-8
< content-length: 11886
< x-powered-by: Express
< accept-ranges: bytes
< cache-control: public, max-age=0
< last-modified: Fri, 20 Sep 2024 17:15:59 GMT
< etag: W/"2e6e-192106ef165"
< x-served-by: dashy.DOMAIN.COM
<
appConfig:
  theme: lissy
...

The above curl was ran from the local docker server running dashy.

Additional info

No response

Please tick the boxes

[X] You have explained the issue clearly, and included all relevant info
[X] You are using a supported version of Dashy
[X] You've checked that this issue hasn't already been raised
[X] You've checked the docs and troubleshooting guide
[X] You agree to the code of conduct

tuxpowered commented 2 months ago

I did a random test using the endpoint https://snippet.host/tvcw/raw used in the documentation and the same errors happen.

So I am thinking there is nothing wrong with the proxy settings, and this is a dashy issue.

hockwill commented 2 months ago

The error message from the browser console says it did not fetch the remote resource at all as the page that was requesting the resource and the resource are from different origins. Browsers check the Access-Control-Allow-Origin header before requesting content.

While using https you need proper configuration of CORS.

tuxpowered commented 2 months ago

@hockwill So dashy is behind Nginx Proxy Manager. What I do not understand is if you go to the URL with curl or a web browser it absolutely loads the conf.yml file for you. But when another dashy instance tries it it does not work, only Dashy complains but no other browser, curl, or wget has an issue.

Additionally I do not understand why testing with the sample https://dashy.to/docs/pages-and-sections/ using the https://snippet.host/tvcw/raw again, dashy fails to load the page but browsers, curl and wget have no problem retrieving it.

If this was really a CORS issue, than shouldn't every browser and CLI tool be erroring out as well? It seems very poor to think that an application has to make the specific check but everyone else can access it.

hockwill commented 2 months ago

I am getting different results with different browsers using the https://snippet.host/tvcw/raw.

Curl has the access-control-allow-origin: * in the response. Excellent.
Firefox does not and complains about it being missing. Dashy fails to load correctly.
Chrome has it in the response. Dashy loads correctly.

After some searching about for why this was. I had Firefox clear the site data for my test Dashy and it worked.

CORS is handled in the browser when one site tries to include resources from another. Fetching a single item with curl does not trigger CORS.

tuxpowered commented 2 weeks ago

Sorry for loonnnng time away. So if I understand you correctly this is a browser issue?

During this time I migrated services off nginx proxy manager and moved to Caddy. Yes the Access-Control-Allow-Origin is set in caddy

headers / {
  Access-Control-Allow-Origin *
}

So while previously I checked with curl and things worked I checked again and still failure. So I took the time and added the https://snippet.host/tvcw/raw snippet as a remote page.

I get the same error, "Unable to load config from: https://snippet.host/tvcw/raw

I tried this in multiple browsers. (Chrome, Safari, Opera)

After some more testing, it appears that the issue may be related to Dashy's handling of authenticated pages. while passing HTTPBasic in the URL string works fine in a browsers, it seems to not work in dashy's remote pages.

I verified this by starting an incognito window with RAW working, and with the destination behind a basic auth page it fails.

hockwill commented 2 weeks ago

Yes CORS is a browser/https thing. The https server hosting the config files it the one that needs to provide the headers.

I poked about to make my own instance load a config file from a basicauth https host. I needed to provideAccess-Control-Allow-Origin $http_origin, Access-Control-Allow-Methods 'GET, POST, OPTIONS', Access-Control-Allow-Credentials 'true'. If Allow-Origin is provided with '*' while trying to perform basic authentication you will get a follow up error saying * cannot be used with authentication and it needs to specify the allowed origin.

tuxpowered commented 1 week ago

@hockwill ... So when using basicauth, the Access-Control-Allow-* methods need to be manually populated to specific hosts? Would you happen to have a copy of a caddy config to set the headers right?

This is super odd still because if I take the same URL string put into dashy with the basicauth added to the URL, and placed into ANY browser it instantly connects and gives the conf. No needing to modify anything in the server side or client side.

It only seems to fail when dashy makes the request specifically. If this was actually an HTTPS/browser thing as you suggest, then that failure should happen on any browser.

Remote conf.yal

If you look at the headers from a chrome session using the same connection string, you can see it clearly loads without any issues. And NONE of those headers are required.

If in fact the "Browser" required the Access-Control-Access-* headers than it should have errored which it does not. It simply does as expected and is to load the content.

hockwill commented 1 week ago

https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

I do not use Caddy anywhere can cannot provide header setups for it.

Lissy93 / dashy