Issue with getFormKey in catalog/product/view and catalog/category

[alexniculescu23]

Hello,

I'm having some issues with turpentine - everything works well with my main store www.bobandlush.com, but I'm having problems with www.bobandlush.de and www.bobandlush.ch . Catalog/product/view form breaks for me - please check attached screenshots. I'm using Magento 1.9.2.2.

Anyone knows what I can do in order to solve this issues? If I clear magento's cache folder, everything works fine for a while, then the problem comes back.

Thanks, Alex

[miguelbalparda]

@alexdan2302 what have you tried so far? Have you checked the wiki?

[miguelbalparda]

Also, you might want to try https://github.com/nexcess/magento-turpentine/pull/1115 and report back.

[alexniculescu23]

@miguelbalparda so far I tried using -p feature=+esi_ignore_other_elements (already added this from the start) , but the problem is still there. I'm updating app/code/community/Nexcessnet/Turpentine/Model/Core/Session.php right now, clear the cache and then report back. Thank you!

[miguelbalparda]

Perfect @alexdan2302, we could really use some feedback for that PR. Thank you!

[alexniculescu23]

@miguelbalparda Unfortunately I still have the same problem. Do you know what else I can try? My main website (.com website without a store code in the url) works fine. This issue appears only on my .de / .ch websites where store codes exist in the url.

[alexniculescu23]

To get rid of this problem, I'm clearing the cache. You can see exactly what is happening from my original screenshots. Please let me know if you have any other ideas about what I should try.

[aricwatson]

Just to be sure - the sites are all running from the same installation of Magento?

[alexniculescu23]

Yes they are.

[aricwatson]

Does this help (from https://github.com/nexcess/magento-turpentine/issues/36)

Posting here because it's the second time this has happened to me: If you're using a custom store view switcher, make sure to include the "_from_store" parameter as well, as the caching won't work properly without it. "?__store=en" won't work, but "?store=en&___from_store=fr" will.

[alexniculescu23]

I'm not changing stores, I'm navigating directly to my .de / .ch store and I have those cache issues with getFormKey being concatenated with the old url.

[alexniculescu23]

Do you know how can I stop that url from being replaced? I'm not seeing turpentine doing that for my main website (.com). That's the only problem I have. I don't understand why this problem happens only on my secondary websites.

[aricwatson]

Are you using the Use VCL Fix under the varnish general settings? If so, does it work on the first page load (when it should be passing the request through to Magento?)

[alexniculescu23]

Yes, I am using it, but it does not work on the first page load from incognito. This happens only in my de/ch stores

[miguelbalparda]

Is there any chance you were using an old version of Turpentine? We have seen some issues with older versions still using a local rewrite. Do you have anything inside app/code/local/Mage/Core/Model/?

[alexniculescu23]

I'm using 0.6.8 Nexcess_Turpentine and yes I do have something inside app/code/local/Mage/Core/Model/ - File/Uploader.php and Resource/Session.php (PHP7 compatibility fix from Inchoo )

[alexniculescu23]

You can view it here on github https://github.com/Inchoo/Inchoo_PHP7

[alexniculescu23]

Any other ideas guys about what I should try?

[aricwatson]

Is it possible to test w/out the PHP7 fix?

[alexniculescu23]

Our server has php7 and without that fix sessions won't work correctly. You said something earlier about if it works on the first page load - which it doesn't - when I visit a product page directly on my de/ch store from an incognito window the getFormKey issue is there.

Here is the link: http://www.bobandlush.de/de_de/60-ententrockenfutter.html

Can I do something to avoid that link being replaced? I don't see that happening on my .com store

.com link: http://www.bobandlush.com/premium-duck-kibble-with-potatoes-peas.html

[aricwatson]

It looks like there's some sort of Varnish/Turpentine configuration problem. Even if I set a varnish_bypass cookie and view your .de site I see ESI processing not enabled and other issues.

Also, it looks like Turpentine is not set up at all on your .com site - which might explain why you're not seeing any of the problems you're seeing on the other sites.

[ArjenMiedema]

I'm having the same issue as @alexdan2302, but with a slightly different URL. In my shop it shows this as my add to cart URL:

http://www.mydomain.com/checkout/cart/add/uenc/aHR0cDovL2RhdG9uYS5oeXBlcm5vZGUuaW8vZ2VyZWVkc2NoYXBwZW4,/product/1095/form_key/<esi:include src=" http:="" www.mydomain.com="" turpentine="" esi="" getformkey="" ttl="" 88400="" method="" scope="" global="" access="" private="" "="">/'

I found out that this has something to do with running on Varnish 4.0, but I have no idea how to fix this using the FAQ. So in fact it's not the same issue after all :-)

[alexniculescu23]

@ArjenMiedema I'm using varnish 4.0 as well - so you are saying that this is related to this specific version ?

[ArjenMiedema]

@alexdan2302 yes, it's only related to Varnish 4.x (which only works with the latest version of the Turpentine extension). In my case it has been fixed by our hosting provider, who updated the default Varnish configuration on the server. It's either that or downgrading to Varnish 3.x, but that's not something I'd suggest.

[miguelbalparda]

@ArjenMiedema would you mind adding here what was changed in the Varnish config?

[ArjenMiedema]

@miguelbalparda I have no idea, I'm not that good at server configuration. It has been fixed by Byte.nl on their Hypernode environment. The only thing I know is that they added +esi_ignore_other_elements to the config. If you need more info, maybe I can ask them.

[miguelbalparda]

Maybe @gwillem can help here :sos:

[gwillem]

@genisd can you have a look?

[genisd]

These are our varnish4 startup flags, I'm not sure if they're relevant or useful:

DAEMON_OPTS="-a :6081 \
             -p vcc_allow_inline_c=on \
             -p thread_pool_add_delay=2 \
             -p thread_pools=2 \
             -p thread_pool_min=25 \
             -p thread_pool_max=2000 \
             -p timeout_linger=50 \
             -p first_byte_timeout=300 \
             -p cli_buffer=65536 \
             -p syslog_cli_traffic=off \
             -p workspace_backend=64k \
             -p feature=+esi_disable_xml_check,+esi_ignore_other_elements \
             -T 127.0.0.1:6082 \
             -u varnish -g varnish \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,2675m"

@alexdan2302 maybe you can try these flags to see if they give you any different behaviour? in particular the two feature flags feature=+esi_disable_xml_check,+esi_ignore_other_elements

[alexniculescu23]

@genisd thank you, I just added +esi_disable_xml_check and I'm waiting to see if the problem comes back or not.

Meanwhile I was looking at what @aricwatson told me earlier and it is possible I have some sort of configuration problem. I'm attaching 2 files - first is with my startup flags and second is my default.vcl file. Please tell me if you see anything strange there. If someone opens an incognito window and tries to access my product page directly http://www.bobandlush.ch/de/60-ententrockenfutter.html , I still get those errors and I want to know what I did wrong. Thank you!

I also used http://www.isvarnishworking.com/ to test if varnish is running fine and it tells me "Yes! Sort of"

default.vcl.txt startup_flags.txt

[genisd]

It says on your page (http://www.bobandlush.ch/de/60-ententrockenfutter.html), twice the following:

ESI processing not enabled ESI processing not enabled

In varnish4 (I'm not sure about varnish3), esi processing must explicitly be enabled for certain urls in the VCL. Otherwise no ESI processing will occur. So the vcl maybe interessting to take a further look

[miguelbalparda]

Thanks a lot for the help @genisd and @gwillem!!!

[alexniculescu23]

Unfortunately that issue is still there, it came back this morning. @genisd can you please provide an example of how esi processing explicitly enabled for certain urls looks like? I'm using the default.vcl for varnish 4.0 that came with turpentine.

@miguelbalparda can you please have a look at my default.vcl ? I have uploaded my vcl in my previous post.

Thank you!

[sylvainraye]

I have similar issue.

It happens if access of an ESI block is private, if this block has urls with formKey and if the ESI block is cached in Magento. In my case, it was the block product_list in catalog_category_default handle. In this case, the ESI flag is 0 so Varnish doesn't interpret ESI includes. With access customer_group or public, the issue didn't happen.

It seems to be related to the registry turpentine_esi_flag which is set in two places: in the class Nexcessnet_Turpentine_Model_Core_Session::getFormKey or in \Nexcessnet_Turpentine_Model_Observer_Esi::injectEsi and defines the -X-Varnish-ESI flag. I think the candidate of the issue is in Nexcessnet_Turpentine_Model_Core_Session::getFormKey cause of Magento caching of the block.

The ESI flag must be 1 in this case and varnish will interpret ESI includes.

To reproduce the issue:

• have an ESI block with private access and having a cache lifetime not null and cache key defined.
• clear magento cache and restart varnish to purge it.
• open a browser and delete any session of the website and refresh it at least twice (first refresh load create a new session and varnish doesn't cache anything, the second call get the response from the backend server and varnish save it in cache, the third call comes from directly from varnish cache).
• open a second browser, delete any previous session and refresh at least twice. You may also use the first browser but you will have again to delete the frontend session.
• The problem will appear in the second browser.

Possible solutions:

If the block is private access, define the cache key of the block per session as the following:

    public function getCacheKeyInfo()
    {
        return parent::getCacheKeyInfo() + [
            'class' => get_class($this),
            'name' => $this->getNameInLayout(),
            'url' => $this->getRequest()->getPathInfo(),
            'query' => implode('|', $this->getRequest()->getParams()),
            'customer_group' => Mage::getSingleton('customer/session')->getCustomerGroupId(),
            'session' => Mage::getSingleton('core/cookie')->get('frontend')
        ];
    }

Second solution use a different ESI access like customer_group or public in the ESI option. It may be not relevant for some of you

Third Solution: set Cache Lifetime of the ESI block to null and let Varnish manage the caching of the block which is probable the easiest and better solution. Although it's solved in the observer/esi of Turpentine, it seems to have any effect.