Increasing size of cache for Hash Response in Symfony reverse proxy

[tomaszmadeyski]

There's file which stores cache for Hash Response (_fos_user_context_hash request). By default Vary is set to ['Cookie', 'Authorization']. Let's imagine we have 10 anonymous users with different session (Cookie). It will result in 10 entries with the same X-User-Hash. The issue is that old (stale) entries are not removed.

Here is unserialized array from cache for HashResponse (irrelevant data is removed for easier overview):

Array
(
    [0] => Array
        (
            [0] => Array
                (
                    [cookie] => Array
                        (
                            [0] => eZSESSID=4id3umajrf71hnn5pp2dufu8j7
                        )
                )
            [1] => Array
                (
                    [x-user-hash] => Array
                        (
                            [0] => 5945886524019aefc21a0e99486455ed340b913f23d342bdc6d7459b1a67eeef
                        )
                    [date] => Array
                        (
                            [0] => Tue, 06 Jun 2017 08:36:40 GMT
                        )
                    [cache-control] => Array
                        (
                            [0] => max-age=30, public
                        )
                )
        )
    [1] => Array
        (
            [0] => Array
                (
                    [cookie] => Array
                        (
                            [0] => eZSESSID=v43eimfa20v9sdgijeb2at4up2
                        )
                )
            [1] => Array
                (
                    [x-user-hash] => Array
                        (
                            [0] => b1731d46b0e7a375a5b024e950fdb8d49dd25af85a5c7dd5116ad2a18cda82cb
                        )
                    [date] => Array
                        (
                            [0] => Tue, 06 Jun 2017 08:36:01 GMT
                        )
                    [cache-control] => Array
                        (
                            [0] => max-age=30, public
                        )
                )
        )
    [2] => Array
        (
            [0] => Array
                (
                    [cookie] => Array
                        (
                            [0] => eZSESSID=nj7ru5lr10pt87jmfvgb1c5qf1
                        )
                )
            [1] => Array
                (
                    [x-user-hash] => Array
                        (
                            [0] => b1731d46b0e7a375a5b024e950fdb8d49dd25af85a5c7dd5116ad2a18cda82cb
                        )
                    [date] => Array
                        (
                            [0] => Tue, 06 Jun 2017 08:27:28 GMT
                        )
                    [cache-control] => Array
                        (
                            [0] => max-age=30, public
                        )
                )
        )
    [3] => Array
        (
            [0] => Array
                (
                    [cookie] => Array
                        (
                            [0] => eZSESSID=rcpnur318ub47rgeevi9mlq2f5
                        )
                )
            [1] => Array
                (
                    [x-user-hash] => Array
                        (
                            [0] => b1731d46b0e7a375a5b024e950fdb8d49dd25af85a5c7dd5116ad2a18cda82cb
                        )

                    [date] => Array
                        (
                            [0] => Tue, 06 Jun 2017 08:25:17 GMT
                        )

                    [cache-control] => Array
                        (
                            [0] => max-age=30, public
                        )
                )
        )
)

First entry is for logged in user so there's different x-user-hash then in other entries. All other entries are for anonymous users coming from different sessions resulting in same x-user-cache but as you can see entries [2] and [3] are stale (according to date and short max-age set to 30 seconds) and could / should be removed.

The result is that this file keeps growing together with number of users: after performing stress tests with ~5000 users file had ~7MB.

Would it be possible to add invalidation for stale cache entries in Hash Response cache?

[dbu]

i am not aware of such a file. what is the file name? and are you using varnish or the symfony cache kernel? could it be that this is something ez publish is doing?

[tomaszmadeyski]

The filename in my case is located in cache/prod/http_cache/md/63/a0/382348be76ec1c4c13102be3aff5a9d173f16ab8e9a14b292830f2baaeda. The file name is a result of 'md'.hash('sha256', $request->getUri()) (source) where $request->getUri() is http://v.box:8000/_fos_user_context_hash

I'm using Symfony Cache Kernel , not a Varnish. This stuff is happeninng purely in FosHttpCache / Symfony Cache - I debugged it and none of ez publish code is involved here.

The part of logic which is used here is mentioned here

[dbu]

ah i see. is this cache cleared at some point? 7 MB does not sound alarming to me, but if it just keeps growing forever, that will of course become a problem.

@andrerom would you know what to do in this case? i think you have more experience using the symfony cache...

@tomaszmadeyski did you see the note we have on http://foshttpcache.readthedocs.io/en/stable/user-context.html ? if you are able to not start a session for anonymous users, you can avoid the hash lookup... (that would be a good idea in general, otherwise you risk somebody storing information in the php session and that would not work well with caching)

[andrerom]

would you know what to do in this case?

I'm not that familiar with this part, that would be @lolautruche most likely.

if you are able to not start a session for anonymous users, you can avoid the hash lookup...

If the issue here is anonymous user, then indeed the clause in VCL hardcoding the hash solves this (until 2.0 where this logic has been disabled by default).

However, @tomaszmadeyski, wasn't this regarding logged in users? Basically the customer did stress test with tons of different users that would share user context hash, and since they all had different session id's this file grew constantly.

They are aware we (eZ), you (FOS), and Symfony recommend Varnsih, while they are not in a position to do so right now they have been told to switch to that for anything performance critical.

But issue is still valid hence us reporting it.

[lolautruche]

Hi

One of the main issues with SymfonyCache is that stale cache is never purged. This explains why the directory grows constantly. There's nothing much we can do about that, except maybe writing a different cache store, which can be pointless since Varnish is much more performant.

[andrerom]

directory grows constantly

it's not the directory, it's the file containing the user hash. So question is if we can find a way to split it over several files for instance.

[tomaszmadeyski]

ah i see. is this cache cleared at some point?

I didn't notice any clear of this cache and I didn't see in code any logic responsible for clearing it.

wasn't this regarding logged in users?

as far as I understood our client it was for anonymous users and I verified it for anonymous users as well. So not starting session for such users may be solution (or a workaround) indeed

[dbu]

This explains why the directory grows constantly. There's nothing much we can do about that

or have a cronjob that deletes caches older than something like 1 day. but not a great solution either.

If the issue here is anonymous user, then indeed the caluse in VCL hardcoding the hash solves this (until 2.0 where this logic has been disabled by default).

uh, good point. i will update the doc accordingly.

[lolautruche]

Having a cronjob is the only viable solution to me.

it's not the directory, it's the file containing the user hash

I don't get it. How can there be only one file? Shouldn't we have 2 files per hash response (metadata+entity)?

If the issue here is anonymous user, then indeed the caluse in VCL hardcoding the hash solves this (until 2.0 where this logic has been disabled by default).

This is one of the reasons I was against the hardcoded hash removal 😉

[andrerom]

Shouldn't we have 2 files per hash response (metadata+entity)?

@tomaszmadeyski did you also reproduce this? how many md files are generated? Customer was complaining that a given md file grew to 7mb and beyond.

Snippet of original report

Recently, we stress-tested our production environment automatically with approx. 5000 different users.
As one result, we discovered the following situation:
In the directory /ezpublish/cache/prod/http_cache/md/e3/f0 exists one cache file, named "3bfeee...." with a size of 7.314 KByte, which contains a serialized array with approx. 5000 elements.

As an alternative if there is no easy way to split the file up further, maybe we can strip out most http headers from the user context hash lookup so the meta data file grows slower.

[dbu]

This is one of the reasons I was against the hardcoded hash removal

i am afraid of potentially dangerous things being enabled by default. but if there is an anonymous request, this only leads to one single lookup which will then be cached, so not a big problem imho. its still possible to enable the hardcoded hash.

As an alternative if there is no easy way to split the file up further, maybe we can strip out most http headers from the user context hash lookup so the meta data file grows slower.

i would be ok with that (as long as we have a list of headers we remove and leave any other headers in)

[tomaszmadeyski]

how many md files are generated?

There's a bunch of md files being generated (one per request url) but issue is only with file related to _fos_user_context_hash route. It's the only file with Vary set to ['Cookie', 'Authorization'] and that's why this file keeps growing as each user is different which results in new entry in this specific cache file.

All other md files have Vary set to X-User-Hash and it works fine: entries are not duplicated for different users (assuming they have the same x-user-hash)

[dbu]

yeah that makes sense. i don't see how we could avoid that. maybe we could cache those responses in a special way in HttpCache so they go to separate files and can be cleaned out. or provide a symfony command that periodically rewrites that file and removes old entries.

the only real solution seems to be to avoid session cookies for anonymous users.

i can imagine that on a busy website, there could be quite some lock contention over that cache file (or even race conditions leading to entries being lost), when every server thread constantly wants to update the same file.

[andrerom]

the only real solution seems to be to avoid session cookies for anonymous users.

I don't think this is just about anonymous users, the original report said (full text above):

stress-tested our production environment automatically with approx. 5000 different users

And indeed this issue will hit you with any kind of users with sessions, logged in or not, if you have lots of them. If this was only anonymous users we would not have reported it.

So imho the report here is misleading, it's more severe than referring to anonymous implies.

[tomaszmadeyski]

Or maybe you can make use of max-age and date attributes which are already in there in serialized array (the one that keeps growing)? While writing to file a new entry you can check if any other entries is outdated (current timestamp, max-age, date) and remove such entry? In that case there's no need of cronjob.

[dbu]

true, for logged in users there is no way to avoid the issue. sure, you could also clean up the file while writing. but if its thousands of entries and lots of requests, this would be a huge performance waste as you would scan the entries all the time. and as php has no threads, you would also block the web request with scanning.

afaik drupal has this horrible poor mans cronjob thing that just randomly executes system tasks during some requests. i don't think we should go that way here. if anything then it has to be a separate cronjob.

btw, did anybody try if varnish really can handle this scenario of tons of variants of the same page gracefully?

[andrerom]

btw, did anybody try if varnish really can handle this scenario of tons of variants of the same page gracefully?

afaik it does, can't remember us testing 5000 users, but we tested at least 500, and from what I have seen elsewhere it should no be a problem.

So summary ?

• See if we can strip more headers for this one url to reduce the size per session
• Add command to prune the file here for old entries, it will have to avoid blocking writes to the file for long at a time (I assume requests do locking on it, everything else would be buggy since it all goes to one file and is not in a format that can be amended to whiteout locks)