Empty file for bundles assets

LodewijkSioen commented 11 years ago

We're using Cassette 2.0 and every now and then, the bundled assets in Cassette return an empty file. We're seeing this issue at multiple clients. We need to 'touch' a javascript or stylesheet in order to force cassette to create a new bundle. We have no idea why this is happening, but this is a serious issue.

These posts in the mailing list are similar to the problem we're having: https://groups.google.com/forum/#!searchin/cassette/empty$20file/cassette/Kj0NHKwVpE8/crI4xe0yIkIJ https://groups.google.com/forum/#!searchin/cassette/empty$20file/cassette/k3L7lLWtxGk/mld_UekcgR8J

andrewdavey commented 11 years ago

Is it also returning a HTTP status code of 304 not modified with these empty responses? Another developer was seeing a similar behaviour this week.

LodewijkSioen commented 11 years ago

That's correct. Clearing the cache didn't fix the problem though. I'll keep an eye out if it happens again, but we'll be putting debug=false in our configs on our next deployment until this is fixed...

ChristianMoser commented 11 years ago

Hi Andrew,

after cassette did a really good job for months, we are facing a serios problem:

For one of our script bundles, cassette returns HTTP 302, even though we press CTRL + F5. The locally cached version has a size of 0 bytes.

The strange thing is, that we could only reproduce it in Firefox. In IE9 it worked.

Here is a snapshot of the HTTP headers:

Request with Firefox:

GET /cassette.axd/script/9a6HnUu2xOk01mW6XsHqE-MYaTg=/Scripts/Common HTTP/1.1 Host: www.myurl.com User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1) Accept: / Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive Referer: https://www.myurl.com/Admin/Users Cookie: ASP.NET_SessionId=uypo3iokkl2niajxjbajgqar; .ASPXAUTH=E3A3B4ED149084ECFBF75AB5BC83CB5EFBB533C5FDDBD313F739737C0638F4EC530C7AA4BEF817DE4C3FB812FEBB8DE29CB58252CF210FA64ECECD84B8A99EEDD7729242F51965E46964299B3DA5EE40E0F695468C2EE2DFA5E9D1E2081F9E79 Pragma: no-cache Cache-Control: no-cache

Response: HTTP/1.1 304 Not Modified Cache-Control: public Expires: Tue, 07 Jan 2014 06:42:17 GMT Etag: "f5ae879d4bb6c4e934d665ba5ec1ea13e3186938" Server: Microsoft-IIS/7.5 X-Powered-By: ASP.NET X-UA-Compatible: IE=edge Date: Mon, 07 Jan 2013 11:25:54 GMT

Same with Internet Explorer 9

Request GET /cassette.axd/script/9a6HnUu2xOk01mW6XsHqE-MYaTg=/Scripts/Common HTTP/1.1 Accept application/javascript, /;q=0.8 Referer https://www.myurl.com/MyPath Accept-Language en-US,de-CH;q=0.5 User-Agent Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) Accept-Encoding gzip, deflate Host www.myurl.com Connection Keep-Alive Cache-Control no-cache Cookie utma=6325235.925635537.1346063365.1346446180.1346450024.8; utmz=6325235.1346063365.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); ASP.NET_SessionId=k35qd0ybownv0vpdo3xg5lwj; .ASPXAUTH=0B59B51EAB9EE337A7A9E341559F87C3347594AB599AA4F504C5884BD07E5090534E016D3731DD86E133BA87CA4C7B58521EB74CA8B5162D312F3B77FAC07E42E5B6CA656B3F3180236F5C573A58C8D4B108BC22297037FA64488ED5E000CF7E; utma=33200041.701503579.1346061556.1346061556.1346061556.1; utmz=33200041.1346061556.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)

Key Value Response HTTP/1.1 200 OK Cache-Control public Content-Type text/javascript Expires Tue, 07 Jan 2014 11:26:14 GMT ETag "f5ae879d4bb6c4e934d665ba5ec1ea13e3186938" Server Microsoft-IIS/7.5 X-AspNet-Version 4.0.30319 X-Powered-By ASP.NET X-UA-Compatible IE=edge Date Mon, 07 Jan 2013 11:26:14 GMT

(The real url has been replaced by www.myurl.com for privacy reasons).

I hope this helps you to narrow down the problem

Greetings

Christian http://www.wpftutorial.net

LodewijkSioen commented 11 years ago

This is exactly the issue I described. We had it in various versions of IE, so it's not just FireFox. Also, the issue only appears at random clients after restarting the application after updates. And it's just one of the bundles that's empty. It's as if Cassette sends down an empty bundle to the client and there is nothing we can do to force the client to re-fetch the bundle.

EnricoKestenholz commented 11 years ago

We saw the same issue too.

The only way to force the client to re-fetch is to change the css so that a new has is generated.

It seems to only happen when a new .css or .js file is added, but it doesn't happen every time so I can't be sure that it is actually the reason why it genereates an empty boundle.

andrewdavey commented 11 years ago

This could be caused by an IIS issue: http://serverfault.com/questions/117970/iis-7-returns-304-instead-of-200

nsbingham commented 11 years ago

I'm seeing something similar using Cassette 2.0 on Windows Azure (IIS7.5 on Windows Server 2008 R2). We occasionally receive empty 200 responses from our servers. We aren't building the Cassette bundles beforehand, so they're being created on first load. We don't tend to see this right after a deployment, but usually a day or two later and only one of the servers. A reboot temporarily fixes it. It does seem very similar to the bug in the serverfault link above. It's only happening with our Cassette JavaScript bundles though and not any other assets being served by the server including our CSS bundles being handled by Cassette. We have seen this behaviour across multiple browsers (Chrome, Safari, IE 10, Firefox), even browsers with a clear cache, so it does point to it being an issue server-side. I'm continuing to investigate, but wanted to provide some additional information.

tobiaszuercher commented 11 years ago

what is the state about this issue? we still have this problem and can't use it in production :(

rodmjay commented 11 years ago

Could it have something to do with the encoding of the files, or encoding mismatch in the bundle? Is there a specific encoding that works better?

thelalle commented 10 years ago

We had this issue, it was a nightmare to find out what was causing this but in the end the solution was:

 <system.web>
    <caching>
      <outputCache enableOutputCache="false" />
      <outputCacheSettings>
        <outputCacheProfiles />
      </outputCacheSettings>
    </caching>
 </system.web>

iis2696 commented 10 years ago

Has anyone come up with a different solution to this? We're getting this issue on our server but would rather not turn debug on, output caching off, or not use cassette.

nvivo commented 9 years ago

Any news on this? After months in development and a couple days in production started having this problem out of nothing. The "quick fix" was to turn debugging on.

ratheo commented 9 years ago

We're also experiencing this. Instead of disabling ouput cache completely, you can disable it for .axd files only.

<system.webServer>
    <caching enabled="true" enableKernelCache="true">
      <profiles>
        <add extension=".axd" policy="DisableCache" kernelCachePolicy="DisableCache" />
      </profiles>
    </caching>
</system.webserver>

jerrosenberg commented 9 years ago

I can reproduce this consistently, only on Windows Server 2008 R2 / IIS 7.5 -- NOT IIS 8, with debug false. We also use the following web.config cassette configuration in case it's relevant:

<cassette rewriteHtml="false" cacheDirectory="asubfolder" />

Steps are:

Using IE 10 or IE 11 (I believe earlier versions, too, but these are what I tested right now), do a hard refresh (Ctrl+F5) on a page referencing cassette script bundles.
Do a soft refresh (F5) on the same page. Don't do anything else with IE at this point.
On the server, run the following command from a prompt: netsh http show cachestate Observe a 304 response in the cache list for the scripts.
Visit the same page in Chrome. Do a hard refresh (Ctrl+F5). Observe the 304 response despite having no If-None-Match header and having no-cache headers.

The steps must be done fairly quickly because after a certain timeout, I believe 2 minutes, the unused urls are evicted from the kernel mode cache, and it beings working again.

Once the problem starts happening, I can resolve it by using IE to do another hard refresh (Ctrl+F5). If you run netsh http show cachestate again, you'll see the response code back to 200 again, and other browsers work fine again.

Because of how our web.configs are setup and how we use location tags with inheritInChildApplications=false, I could not reliably get the kernel cache policy disabled with other people's suggestions using configuration.

Instead, we added an IUrlModifier that adds a static query string to the end of all bundle urls "?n". This seems to avoid the issue - not sure if it's avoiding the kernel cache this way or just causes a different code path that avoids the bug.

Can anyone else reproduce using these steps?

jeybonnet commented 8 years ago

Hi,

Is there any answer to this issue?

TheCloudlessSky commented 6 years ago

I know this is an old issue but I'm randomly experiencing a similar issue in one of my apps that still uses Cassette.

@jerrosenberg when you added the static query string, did you implement it literally with ?n like this?

public class HackCacheUrlModifier : IUrlModifier
{
    public string Modify(string url)
    {
        if (url.StartsWith("cassette.axd/script/") || url.StartsWith("cassette.axd/stylesheet/"))
        {
            return "/" + url + "?n";
        }
        else
        {
            return "/" + url;
        }
    }
}

I haven't been able to reproduce your steps, but I'm interested in your fix if it works.

jerrosenberg commented 6 years ago

I’m not at the same company anymore so unfortunately can’t tell you for sure. Shame on me for not posting the snippet. But if memory serves and based on my wording, yes it was literally “?n”. I believe any query string avoids the issue.

TheCloudlessSky commented 5 years ago

After an extensive investigation, we've found the root cause of this issue (2 bugs):

Bug # 1

Cassette sets the Vary: Accept-Encoding header as part of its response to a bundle since it can encode the content with gzip/deflate:

However, the ASP.NET output cache will always return the response that was cached first. For example, if the first request has Accept-Encoding: gzip and Cassette returns gzipped content, the ASP.NET output cache will cache the URL as Content-Encoding: gzip. The next request to the same URL but with a different acceptable encoding (e.g. Accept-Encoding: deflate) will return the cached response with Content-Encoding: gzip.

This bug is caused by Cassette using the HttpResponseBase.Cache API to set the output cache settings (e.g. Cache-Control: public) but using the HttpResponseBase.Headers API to set the Vary: Accept-Encoding header. The problem is that the ASP.NET OutputCacheModule is not aware of response headers; it only works via the Cache API. That is, it expects the developer to use an invisibly tightly-coupled API rather than just standard HTTP.

Bug # 2

When using IIS 7.5 (Windows Server 2008 R2), bug # 1 can cause a separate issue with the IIS kernel and user caches. For example, once a bundle is successfully cached with Content-Encoding: gzip, it's possible to see it in the IIS kernel cache with netsh http show cachestate. It shows a response with 200 status code and content encoding of "gzip". If the next request has a different acceptable encoding (e.g. Accept-Encoding: deflate) and an If-None-Match header that matches the bundle's hash, the request into IIS's kernel and user mode caches will be considered a miss. Thus, causing the request to be handled by Cassette which returns a 304:

BundleRequestHandler: https://github.com/andrewdavey/cassette/blob/a2d9870eb0dc3585ef3dd542287595bc6162a21a/src/Cassette.Aspnet/BundleRequestHandler.cs#L44

However, once IIS's kernel and user modes process the response, they will see that the response for the URL has changed and the cache should be updated. If the IIS kernel cache is checked with netsh http show cachestate again, the cached 200 response is replaced with a 304 response. All subsequent requests to the bundle, regardless of Accept-Encoding and If-None-Match will return a 304 response. We saw the devastating effects of this bug where all users were served a 304 for our core script because of a random request that had an unexpected Accept-Encoding and If-None-Match.

The problem seems to be that the IIS kernel and user mode caches are not able to vary based on the Accept-Encoding header. As evidence of this, by using the Cache API with the workaround below, the IIS kernel and user mode caches seem to be always skipped (only the ASP.NET output cache is used). This can be confirmed by checking that netsh http show cachestate is empty with the workaround below. ASP.NET communicates with the IIS worker directly to selectively enable or disable the IIS kernel and user mode caches per-request.

We were not able to reproduce this bug on newer versions of IIS (e.g. IIS Express 10). However, bug # 1 was still reproducible.

Our original fix for this bug was to disable IIS kernel/user mode caching only for Cassette requests like others mentioned. By doing so, we uncovered bug # 1 when deploying an extra layer of caching in front of our web servers. The reason that the query string hack worked is because the OutputCacheModule will record a cache miss if the Cache API has not been used to vary based on the QueryString and if the request has a QueryString.

Workaround

We've been planning to move away from Cassette anyways, so rather than maintaining our own fork of Cassette (or trying to get a PR merged), we opted to use an HTTP module to work around this issue.

public class FixCassetteContentEncodingOutputCacheBugModule : IHttpModule
{
    public void Init(HttpApplication context)
    {
        context.PostRequestHandlerExecute += Context_PostRequestHandlerExecute;
    }

    private void Context_PostRequestHandlerExecute(object sender, EventArgs e)
    {
        var httpContext = HttpContext.Current;

        if (httpContext == null)
        {
            return;
        }

        var request = httpContext.Request;
        var response = httpContext.Response;

        if (request.HttpMethod != "GET")
        {
            return;
        }

        var path = request.Path;

        if (!path.StartsWith("/cassette.axd", StringComparison.InvariantCultureIgnoreCase))
        {
            return;
        }

        if (response.Headers["Vary"] == "Accept-Encoding")
        {
            httpContext.Response.Cache.VaryByHeaders.SetHeaders(new[] { "Accept-Encoding" });
        }
    }

    public void Dispose()
    {

    }
}

I hope this helps someone 😄!

andrewdavey / cassette