"Too many redirects" for css and js files on the new infrastructure

RyuLindow commented 3 years ago

Issue description

Unfortunately, this seems to be quite random and it doesn't happen for every visitor on the page but rather the ones who visited the site before.

Sometimes new visitors can also get the same error.

I've seen it happen on a few projects now.

I don't have any steps to reproduce it.

The solution which worked every time involves changing the path of the css/js file, so either change the name of the folder the file is in or change the name of the file itself.

Bumping the CDF doesn't seem to help btw.

I'll try to provide some screenshots and more data the next time I see it.

FransdeJong commented 3 years ago

I've ran into this issue twice now. I don't have a clear reproduction scenario but what happened on the most recent site is the following:

The physical path to the css file on this site is \Css\site.min.css The reference to the file in the master was ~/Css/site.min.css

When we went live and added a custom Url everything worked like it worked before we added the hostname. 4 hours after we went live I checked the site and I ran into this issue. When I checked what happend I noticed that the Url "/Css/site.min.css" was not working and "/css/site.min.css" was.

After I noticed that I logged in to Umbraco and changed the master template to ~/css/site.min.css. I didn't change anything else. So changing the path isn't necessary.

After debugging this with @Aaen from support I looked at the Url's again (again 4-5 hours later) and now both "/Css/site.min.css" and "/css/site.min.css" were working fine.

I still didn't change the path or filename.

Could this have something to do with caching in Cloudflare? In both cases it only happened to us after adding a custom domain.

FransdeJong commented 3 years ago

My colleague remembered what site we had the issue on before and on that site the error fixed itself to. So my first idea would be to look at static file caching in Cloudflare.

RyuLindow commented 3 years ago

Heya @FransdeJong.

We don't cache anything in Cloudflare. Yet. There are plans to implement the CDN on Cloud later and it's also on our roadmap.

I'll be keeping a close eye on this issue, as I've seen it on 3 projects and will post all the updates here when/if I get any :)

FransdeJong commented 3 years ago

Hmm.. than I'd look at V-next. It is something that is introduced recently. We've never seen this behaviour before and now twice in two weeks.

QaisarIshaq commented 3 years ago

We have had the same problem on one of our website and renaming the files fixed it.

RyuLindow commented 3 years ago

I think that it's caused the HTTP>HTTPS redirect config transform which we have for all live environments on Cloud.

Two theories:

Cloudflare's HTTPS redirect + config transform HTTPS redirect + additional HTTPS redirect added manually be the dev to the web.config = "Too many redirects"
Cloudflare's HTTPS redirect + config transform HTTPS redirect = "Too many redirects"

Still not sure why it's only happening on the new infra. Cloudflare setup should be the same...

Will get back with more info by the end of the week :)

FransdeJong commented 3 years ago

@RyuLindow I see one flaw in this theory, why does the issue present itself after a while and resolve itself over time? That doesn't make sense to me...

RyuLindow commented 3 years ago

Oh, I fully agree. The part about it being random really annoys me because I can't even reproduce this issue :(

Just trying to get some leads. Maybe one of them will be the right one.

FransdeJong commented 3 years ago

I'm not sure it is random. It starts happening with a delay and resolves itself later on. It's almost like cloudflare adds something that breaks it and later resolves it or adds something that fixes the issue again.

Also in my experience it only happens after you add a custom domain.

c9mb commented 3 years ago

@RyuLindow - Out of curiosity - do you also have the umbraco.io => custom-domain config-transform redirect in place ?

I reported a similar issue with cloud-support when the cloudflare-dns started to roll out and I started to deploy new sites with it.

It would get stuck in a redirect loop as cloudflare would redirect a custom-domain request to umbraco.io and the config-redirect would send it back to the custom-domain, which cloudflare would redirect back to umbraco.io... etc etc.

Support was able to repro and came up with a hacky workaround, involving some custom server variables to track the original requested host, so I could re-write the rule, but then it magically 'fixed' itself, and was no longer needed, so I removed the workaround from my sites.

When I followed this up, so find out if it have been 'fixed' or had a better solution, it was a case of "problem? what problem?" as support was unable to repro again, and I suspect figured it was just another day, and another dumb user.

I was never really comfortable it was actually fully resolved... but it's hard to ague against when you can't repro the issue.

hfloyd commented 3 years ago

We are seeing this issue also. The only thing which seems to "resolve" it is turning Debug Mode ON for the live site... not an ideal solution. The question is - what is it about "non-Debug" mode which interferes with CDF processing/asset loading?

I found this, which sounds like the same issue, but the suggestions provided don't really help https://forums.asp.net/t/2131069.aspx?MVC+Bundle+Not+Working+with+Release+Configuration+Debug+is+False+css+and+js+not+loading

So, debug=true prevents caching and compression - which would explain why it fixes the problem - and points to caching/compression as being the root of the issue.

RyuLindow commented 3 years ago

Here's the promised update :)

I've looked into this (and while I still don't know why it's random and appears and then fixes itself) I believe that it's related to - https://community.cloudflare.com/t/community-tip-fixing-err-too-many-redirects/42335

We have the Always Use HTTPS setting enabled and if you read - https://support.cloudflare.com/hc/en-us/articles/204144518-SSL-FAQ#h_a61bfdef-08dd-40f8-8888-7edd8e40d156 you'll notice this section:

While protecting your site via Cloudflare, it is not recommended to perform redirects at your origin web server:

Page Rule redirects are processed at the Cloudflare edge resulting in quicker response and reduced requests to your server. Origin web server redirects can cause redirect loop errors - https://support.cloudflare.com/hc/en-us/articles/115000219871

We have a web.config transform for the live env with an HTTP>HTTPS redirect on Cloud which is a pre Cloudflare thing. This would fall under "it is not recommended to perform redirects at your origin web server"

I think it would make sense to just remove that file as it's unnecessary since we have that redirect on the Cloudflare level.

I've passed all of this info over to the team for them to look into it, as this is most likely as far as I can get on my own.

hfloyd commented 3 years ago

Thanks so much, Darek! So, should we just remove the redirect in question? Or wait to hear confirmation from you after consulting your colleagues? ~Heather

RyuLindow commented 3 years ago

You can definitely go ahead and remove the Https.Web.live.xdt.config file from your repository and they deploying that change to Cloud, as that redirect is handled on the Cloudflare level.

We don't have a project with this issue and we can't reproduce it, so it's not something I confirmed to work but it's definitely worth trying :)

I'll check with the team next week and hopefully, we'll get some updates on this.

It would be great if you could give this a go on your project and let me know if that helped, though.

hfloyd commented 3 years ago

So, in our project, the file was named Latch.Web.live.xdt.config. I did remove the file. I also checked our other web.config transforms and removed any other https-related redirect rules, and made sure any references to the current site's domain was updated to match the new *.euwest01.umbraco.io format.

After pushing those changes through to Live, and even restarting the environment, the issue is still occurring.

This seems to be the main issue: getting a 404 here:

RyuLindow commented 3 years ago

Thanks a lot for testing that out, Heather :)

I passed the info over to the team and we'll look further into it.

Also, I checked your s1 app and it has the same console errors there as well, so I'm not sure if that's related, though 🤔

There doesn't seem to be a /bundles folder there.

bhavikvaishnani commented 3 years ago

We are seeing this issue also. The only thing which seems to "resolve" it is turning Debug Mode ON for the live site... not an ideal solution. The question is - what is it about "non-Debug" mode which interferes with CDF processing/asset loading?

I found this, which sounds like the same issue, but the suggestions provided don't really help https://forums.asp.net/t/2131069.aspx?MVC+Bundle+Not+Working+with+Release+Configuration+Debug+is+False+css+and+js+not+loading

So, debug=true prevents caching and compression - which would explain why it fixes the problem - and points to caching/compression as being the root of the issue.

Yes, Having the same issue, but surprisingly it's working for partial code, As our project uses uSkinned so respective CSS is getting compress and showing the output (DependencyHandler.axd..) but the newly added css are not getting rendered with debug mode set to Off. (and same is working for debug mode set to On)

so any advice on it?

RyuLindow commented 3 years ago

Hi @bhavikvaishnani Can you please reach out to support and provide them with a link(s) to the file(s) that you're seeing the issue with?

We're currently gathering information so that we have some details that we can send to Cloudflare, as we'll be creating a support ticket with them shortly.

It would be great if you could reference this issue as well :)

Aaen commented 3 years ago

@RyuLindow I have links if you need it for the issue that @bhavikvaishnani are seen let me know if you want them.

meyntony commented 3 years ago

I have reported the issue with Cloudflare, and have asked for assistance in investigating the issue, when was the last time this has occurred?

hfloyd commented 3 years ago

This is still an active issue for one site I work on. Same behavior on any Umbraco Cloud environment - (dev/staging/live) if I turn ON "debug mode", the front of the site renders just fine. If I turn OFF "debug mode", the display is all messed up.

meyntony commented 3 years ago

Hi @hfloyd , Thanks for that information. When you say any Umbraco Cloud environments? Could you let me know if its for a certain version of Umbraco v7/v8/v9 etc. We are currently collecting information to investigate the issue before we can find a fix. :) So all the help will be useful.

hfloyd commented 3 years ago

This is specifically a v8 site. It was happening on v 8.14.4 (when the infrastructure change happened). Yesterday I updated it to 8.17.0, but the issue persists. There are a few support cases for this.

meyntony commented 3 years ago

Hi @hfloyd , Thanks for that. I just created a cloud site on the new infrastructure with 8.17.0 The is what I get for a simple page with the bundled css With debug off: With debug on:

I also have gone through some of the support cases, seems like this files which had the issue loading works now, so if you have a cloud solution where this can be reproduced by toggling debug mode on/off, Could you reach out to support with the details on how we could reproduce the issue? Also please let them know of this ticket, so that they reach out to me.

Thanks in advance.

hfloyd commented 3 years ago

There are a few support cases already setup for this issue for our website in question, and i have referenced this github issue in those as well by URL. So, it seems that the "DependencyHandler.axd" file could be the issue?

meyntony commented 3 years ago

Hi @hfloyd ,

I noticed that for the project you mentioned, the issue happens even locally when I switch the debug off via the web.config. Could you confirm this?

Thanks in advance.

hfloyd commented 3 years ago

@meyntony Testing locally, you are correct. So, that points away from it being a Cloudflare issue, and towards...? A more general search (https://www.google.com/search?q=dependencyhandler+debug&rlz=1C1PRFI_enUS760US760&oq=De&aqs=chrome.3.69i60j69i57j35i39l2j69i60l4.4577j1j7&sourceid=chrome&ie=UTF-8) turns up a bunch of similar issues, so it seems I have my work cut out for me.

This used to work fine (the site has been live for over a year) and the error started around the time we got the infrastructure switch, so assumed it was related. Of course "correlation ≠ causation", and this is a reminder of that truism. I will report back if I find any solution, and if you have any other ideas, I'd be happy to hear them. Thanks!

RyuLindow commented 2 years ago

Hello :)

I wanted to post some solutions that resolved this on the project they were tried on.

"Too many redirects" error

Possible cause: Conflict between Cloudflare's forced HTTP to HTTPS redirection and the pre-Cloudflare HTTP to HTTPS config transform on the origin server which is UC.

Solution: Remove HTTP to HTTPS config transform from the project's repo. Based on - https://community.cloudflare.com/t/community-tip-fixing-err-too-many-redirects/42335

Unstyled pages

Possible cause: Some Cloudflare endpoints have stored a 404 response instead of app.css. We suspect this could be due to this file having been requested by an end-user or a crawler during deployment, while the resource on Cloud was temporarily unavailable. The resulting response was cached by some Cloudflare endpoints and is being served to some users.

Solution: Version control static assets (CSS, JS) using a query string which is modified if the file changes, e.g. using a technique such as Cache busting with ASP.Net MVC .

c9mb commented 2 years ago

FWIW - My sites are all using the common IIS rule to redirect from umbraco.io to their custom-domain, with some exclusions. However, this created a problem when migrated to the new infrastructure and its default fileprovider configuration, resulting in a 'too many redirects' error for image and resource requests.

The workaround provided to get migrated sites running was an additional exclusion for any request with a file extension. e.g. <add input="{REQUEST_FILENAME}" negate="true" pattern="(.*?)\.[a-zA-Z1-5]{1,5}$" />

However, I suspect this sledgehammer solution might be a bit of a blanket that masks an underlaying problem, and that is possibly has more to do with Cloudflare and the way the fileprovider handles file requests, perhaps indicating the source of the problem. The new azurewebapp sites use a base domain of azurewebsites.net with a GUID host name, and everything appears to be redirected up and down from there.

I'm also running a default IIS rule to redirect from a //domain request to a //www.domain request, and this also caused issues on the new infrastructure with request coming from azurewebsites.net (e.g. KeepAlive task) and another exclusion was required for that rule to stop it spewing out 'failed to resolve' errors in the log. <add input="{HTTP_HOST}" negate="true" pattern="^(.*)?\.azurewebsites\.net$" />

FransdeJong commented 2 years ago

I just ran into a situation where a client's site was completely unreachable due to too many redirects. Debug mode didn't help. Somehow changing the culture and hostnames did... I don't understand this at all since it makes no sense how this should've fixed this but I wanted to share this here.

The issue is handled by Prathees this morning if you need more information.

c9mb commented 2 years ago

@FransdeJong does the client site have any IIS redirection rules in the config.web or config.web transformations ?

FransdeJong commented 2 years ago

Just the non www to www and umbraco.io to real URL like all our other sites. Also nothing changed that triggered this issue.The site was running fine as is for half a year and last change was from the 24th of November

c9mb commented 2 years ago

As per my earlier post, my sites also have the standard umbraco.io to custom-domain and the non-www to www redirect rules, and both caused problems when Umbraco migrated the site to their new Azure-PaaS infrastructure. Both rules required additional exclusions to be added - and specifically the exclusion in the umbraco.io to custom-domain rule, to allow any request with a file extension to bypass it - i.e. images, scripts, css, etc.

sitereactor commented 2 years ago

Does your redirect section in web config start with <clear/> by any chance?

meyntony commented 2 years ago

Does your redirect section in web config start with <clear/> by any chance?

No, that does not seem to be the case for the projects that we see this issue in.

meyntony commented 2 years ago

We suspect that the 301 redirect is getting cached in Cloudflare which is part of the current infrastructure at Umbraco Cloud , because it contains a header in the response Cache-Control: max-age=3600 This happens when resources are requested over http , so for example if you have a static resource https://xyz.com/index.js and you request it over http

You can do this on your terminal with curl curl -sD- http://xyz.com/index.js You should be able to see the Cache-Control header.

We now ensure that 3xx-5xx responses are not cached by default. But if the website sends cache-control headers they will be respected and that is how it would be cached. If you are unsure if certain response codes are sending cache control headers please can check via the Chrome console.

umbrabot commented 2 years ago

Hiya @RyuLindow,

Just wanted to let you know that we noticed that this issue got a bit stale and might not be relevant any more.

We will close this issue for now but we're happy to open it up again if you think it's still relevant (for example: it's a feature request that's not yet implemented, or it's a bug that's not yet been fixed).

To open it this issue up again, you can write @umbrabot still relevant in a new comment as the first line. It would be super helpful for us if on the next line you could let us know why you think it's still relevant.

For example:

@umbrabot still relevant This bug can still be reproduced in version x.y.z

This will reopen the issue in the next few hours.

Thanks, from your friendly Umbraco GitHub bot :robot: :slightly_smiling_face:

umbraco / Umbraco.Cloud.Issues