ESI URL return 404 error

[rameshramachandran84-zz]

Hi Team I am having an issue related ESI URL . getting 404 error for ESI . These URL's were working fine and suddenly having this issue .

magento version 1.14.2.4 Turpentine extension version : 0.6.8 varnish version : 3.0.7

[miguelbalparda]

What have you tried so far to debug this issue? Any more info you can add?

[rameshramachandran84-zz]

I have checked the ESI urls while is throwing 404 and correct URL , both have difference in the number of characters ie, url length is different (Not sure that is the root cause)

I am not able to get an conclusion yet , when the URL got invalidated.

IS there any suggestion that you can provide to check this issue any tips to debug this issue .

404 url sample : turpentine/esi/getBlock/method/ajax/access/private/ttl/3600/hmac/f074fa30949db4de76a56e14cfd92fc9bec88854dd22c257274e46982ddf5f21/data/.nnZqKJS-Fjz3.WDnrCotxq9-q2P6pJDdDzrZkIKm4-FLtZ.PZVe4u6.rviH.uU-j2ytPzTi3uEWR.3ld9Ki9zyepB.PVOcUoCcwLDCCz.VCFZ-QlOejbMVlgUkUj0faffyKnF2W0nEz1LwoMwpcl2IqPqTZVuTXR4ADs1bJ2u6BEDZ6F96Q0e3B6qY8rbtEsdLMnCvTV5Jg0rjzIbgIyQ5aBpb1dm9cT7J8QstcNpnuB5J4w8opjzlJn7D0H3Sq7AQuCQ57n5tdRusLXjj8hXE2NwepvxyEEuMtS5M1z4-NqdqwngOxnh0Zk9GFXOaa7UtFpwcLUBUjNiEP-1H7aNIyQk0d8yLcrz0BgtcMXicpEXB6JW3dwX57Fq8Wl3Wlr35ovPvOwP6iMJJoI9bADOZsY.Tdv.KQ

correctly working URL sample turpentine/esi/getBlock/method/ajax/access/private/ttl/3600/hmac/9c8173112cace9cb6eec86d62d7382422755a5d0b5a40ac5f27d9400c52bc7bc/data/hl53o10wOGVWuoAIWIQkkQJx2r5GGjviVc.mZZon.d8iS-0puRCCEjrjZPquKaEKuK2WLOstSsfM0DypjhL1b3gwjMZ2XIGdVL47xaGlUK1dj2njNNy9DnjdTZetKmaesawbxSfK6OqvaBqo.Wu55ltMkDFuUmwdq-Be0BVX2RMTPwYT-cGxz0FYPOyS7tbQWEKeMFa5V2KN.bk-.EbtE4nXAurIywSMF70EgGEB8nbWyybtc9bROUJ399Rsy0MZrK0gmILRGYOY-ysPWjTR4fiSdgs30Y4GfQdWjCyPrHW99GSooyK6kNQqDtRC9hil5n2REn7G9PnZc2Rt5n8CUs0ZY6sFjmAZ4KGW9FkRpauLmiFnaJoMoOqii9r5pL5gt3wbklZjDm9T-.vwN94Q7pBW0UmKxv4oFEJ2ydxxekHbecWDRTDZ4SSAN-wg4BvfuCVFeAY2OAg=/

[miguelbalparda]

Have you made any changes that might have caused this in your server configs or .htaccess or code? A 404 is usually caused by a bad server config and not something related to Turpentine.

[rameshramachandran84-zz]

No changes made to turpentine module . and this issue is only for few products so i doubt if its a configuration issue then it will affect all products .

[miguelbalparda]

I've seen issues where the length of the URL was the cause of this, I suggest you to take a look at the open and closed issues plus the Wiki to see if you can find the exact server setting that is causing this.

[rameshramachandran84-zz]

actually i had check github ,but not able to find anybody reported this issue earlier . also i noticed that the ESI url does not ends with text '"=/" , so my guess is the url trimmed and may be that is causing 404.

[rameshramachandran84-zz]

can you able to recollect any issue reported the same .

[miguelbalparda]

I got this from the search.

[rameshramachandran84-zz]

no luck yet. One question regarding the ESI url, will the ESI url invalidated and recreated for certain products when the product got updated from admin .

[miguelbalparda]

I'll have to check that, but in the meantime please take a look at the code here and to the Wiki, there you can find things like the ESI decoder to see how/why your calls are failing.

[rameshramachandran84-zz]

thanks for the information . I have tried to decode the ESI url using ESI decoder and i am able to see the correct product info and while i browse the url got 404 error .

the decoded data as follows

array ( 'block_type' => 'Mage_Catalog_Block_Product_View', 'complex_registry' => array ( 'current_product' => array ( 'model' => 'Mage_Catalog_Model_Product', 'id' => '23415', ), ), 'dummy_blocks' => array ( ), 'esi_method' => 'ajax', 'flush_events' => array ( 0 => 'customer_login', 1 => 'customer_logout', ), 'layout_handles' => array ( 0 => 'THEME_frontend_enterprise_default', 1 => 'catalog_product_view', ), 'name_in_layout' => 'product.prices', 'simple_registry' => array ( ), 'store_id' => '1', )

[miguelbalparda]

Well, if the URL is valid I don't think your issue is related to Turpentine but to a server/Magento config.

[rameshramachandran84-zz]

thanks for the response : Can you please review the VCL file generated .

C{

include

include

include

include

static pthread_mutex_t lrand_mutex = PTHREAD_MUTEX_INITIALIZER; void generate_uuid(char buf) { pthread_mutex_lock(&lrand_mutex); long a = lrand48(); long b = lrand48(); long c = lrand48(); long d = lrand48(); pthread_mutex_unlock(&lrand_mutex); sprintf(buf, "frontend=%08lx%04lx%04lx%04lx%04lx%08lx", a, b & 0xffff, (b & ((long)0x0fff0000) >> 16) | 0x4000, (c & 0x0fff) | 0x8000, (c & (long)0xffff0000) >> 16, d ); return; } }C import std; backend default { .host = "127.0.0.1"; .port = "8080"; .first_byte_timeout = 300s; .between_bytes_timeout = 300s; } backend admin { .host = "127.0.0.1"; .port = "8080"; .first_byte_timeout = 21600s; .between_bytes_timeout = 21600s; } acl crawler_acl { "127.0.0.1"; } acl debug_acl { } / -- REMOVED sub generate_session { if (req.url ~ ".[&?]SID=([^&]+).") { set req.http.X-Varnish-Faked-Session = regsub( req.url, ".[&?]SID=([^&]+).", "frontend=\1"); } else { C{ char uuid_buf [50]; generate_uuid(uuid_buf); VRT_SetHdr(sp, HDR_REQ, "\030X-Varnish-Faked-Session:", uuid_buf, vrt_magic_string_end ); }C } if (req.http.Cookie) { std.collect(req.http.Cookie); set req.http.Cookie = req.http.X-Varnish-Faked-Session + "; " + req.http.Cookie; } else { set req.http.Cookie = req.http.X-Varnish-Faked-Session; } } sub generate_session_expires { C{ time_t now = time(NULL); struct tm now_tm = gmtime(&now); now_tm.tm_sec += 3600; mktime(&now_tm); char date_buf [50]; strftime(date_buf, sizeof(date_buf)-1, "%a, %d-%b-%Y %H:%M:%S %Z", &now_tm); VRT_SetHdr(sp, HDR_RESP, "\031X-Varnish-Cookie-Expires:", date_buf, vrt_magic_string_end ); }C } -- / sub vcl_recv { if (req.restarts == 0) { if (req.http.X-Forwarded-For) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } if(false) { set req.http.X-Varnish-Origin-Url = req.url; } if (req.http.Accept-Encoding) { if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } else if (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } } if (!true || req.http.Authorization || req.request !~ "^(GET|HEAD|OPTIONS)$" || req.http.Cookie ~ "varnish_bypass=1") { if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed).php/)?circagate") { set req.backend = admin; } return (pipe); } set req.url = regsuball(req.url, "([^:])//+", "\1/"); if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed).php/)?") { set req.http.X-Turpentine-Secret-Handshake = "1"; if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed).php/)?circagate") { set req.backend = admin; return (pipe); } if (req.http.Cookie ~ "\bcurrency=") { set req.http.X-Varnish-Currency = regsub( req.http.Cookie, ".\bcurrency=([^;]).", "\1"); } if (req.http.Cookie ~ "\bstore=") { set req.http.X-Varnish-Store = regsub( req.http.Cookie, ".\bstore=([^;]).", "\1"); } if (req.url ~ "/turpentine/esi/get(?:Block|FormKey)/") { set req.http.X-Varnish-Esi-Method = regsub( req.url, "./method/(\w+)/.", "\1"); set req.http.X-Varnish-Esi-Access = regsub( req.url, "./access/(\w+)/.", "\1"); if (req.http.X-Varnish-Esi-Method == "esi" && req.esi_level == 0 && !(false || client.ip ~ debug_acl)) { error 403 "External ESI requests are not allowed"; } } if (req.http.Cookie !~ "frontend=" && !req.http.X-Varnish-Esi-Method) { if (client.ip ~ crawler_acl || req.http.User-Agent ~ "^(?:ApacheBench/.|.Googlebot.|JoeDog/.Siege.|magespeedtest.com|Nexcessnet_Turpentine/.)$") { set req.http.Cookie = "frontend=crawler-session"; } else { return (pipe); } } if (true && req.url ~ "..(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") { unset req.http.Cookie; unset req.http.X-Varnish-Faked-Session; set req.http.X-Varnish-Static = 1; return (lookup); } if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed).php/)?(?:circagate|api|cron.php|circagate|admin|sales/guest/form|catalogsearch|quickview|catalog/quickview/minicartupdate|checkout/cart/delete/|claymore-medium-tapered-round-wall-washer-chd1624/|renwick-flush-mount-arn4004/|paulo-floor-light-tob1201/|darlana-small-lantern-chc2164/)" || req.url ~ "\?.__from_store=") { return (pipe); } if (true && req.url ~ "(?:?&(?=[&=]|$))") { return (pass); } if (true && req.url ~ "?&=") { set req.url = regsuball(req.url, "(?:(\?)?|&)(?:utm_source|utm_medium|utm_campaign|utm_content|utm_term|gclid|cx|ie|cof|siteurl)=[^&]+", "\1"); set req.url = regsuball(req.url, "(?:(\?)&|\?$)", "\1"); } if(false) { set req.http.X-Varnish-Cache-Url = req.url; set req.url = req.http.X-Varnish-Origin-Url; unset req.http.X-Varnish-Origin-Url; } return (lookup); } } sub vcl_pipe { unset bereq.http.X-Turpentine-Secret-Handshake; set bereq.http.Connection = "close"; } sub vcl_hash { if (true && req.http.X-Varnish-Static) { hash_data(req.url); if (req.http.Accept-Encoding) { hash_data(req.http.Accept-Encoding); } return (hash); } if(false && req.http.X-Varnish-Cache-Url) { hash_data(req.http.X-Varnish-Cache-Url); } else { hash_data(req.url); } if (req.http.Host) { hash_data(req.http.Host); } else { hash_data(server.ip); } hash_data(req.http.Ssl-Offloaded); if (req.http.X-Normalized-User-Agent) { hash_data(req.http.X-Normalized-User-Agent); } if (req.http.Accept-Encoding) { hash_data(req.http.Accept-Encoding); } if (req.http.X-Varnish-Store || req.http.X-Varnish-Currency) { hash_data("s=" + req.http.X-Varnish-Store + "&c=" + req.http.X-Varnish-Currency); } if (req.http.X-Varnish-Esi-Access == "private" && req.http.Cookie ~ "frontend=") { hash_data(regsub(req.http.Cookie, "^.?frontend=([^;]);.$", "\1")); } if (req.http.X-Varnish-Esi-Access == "customer_group" && req.http.Cookie ~ "customer_group=") { hash_data(regsub(req.http.Cookie, "^.?customer_group=([^;]);.$", "\1")); } return (hash); } sub vcl_hit { } sub vcl_fetch { set req.grace = 15s; set beresp.http.X-Varnish-Host = req.http.host; set beresp.http.X-Varnish-URL = req.url; if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed).php/)?") { unset beresp.http.Vary; set beresp.do_gzip = true; if (beresp.status != 200 && beresp.status != 404) { set beresp.ttl = 15s; return (hit_for_pass); } else { if (beresp.http.Set-Cookie) { set beresp.http.X-Varnish-Set-Cookie = beresp.http.Set-Cookie; unset beresp.http.Set-Cookie; } unset beresp.http.Cache-Control; unset beresp.http.Expires; unset beresp.http.Pragma; unset beresp.http.Cache; unset beresp.http.Age; if (beresp.http.X-Turpentine-Esi == "1") { set beresp.do_esi = true; } if (beresp.http.X-Turpentine-Cache == "0") { set beresp.ttl = 15s; return (hit_for_pass); } else { if (true && bereq.url ~ "..(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") { set beresp.ttl = 28800s; set beresp.http.Cache-Control = "max-age=28800"; } elseif (req.http.X-Varnish-Esi-Method) { if (req.http.X-Varnish-Esi-Access == "private" && req.http.Cookie ~ "frontend=") { set beresp.http.X-Varnish-Session = regsub(req.http.Cookie, "^.?frontend=([^;]);.$", "\1"); } if (req.http.X-Varnish-Esi-Method == "ajax" && req.http.X-Varnish-Esi-Access == "public") { set beresp.http.Cache-Control = "max-age=" + regsub( req.url, "./ttl/(\d+)/.", "\1"); } set beresp.ttl = std.duration( regsub( req.url, "./ttl/(\d+)/.", "\1s"), 300s); if (beresp.ttl == 0s) { set beresp.ttl = 15s; return (hit_for_pass); } } else { set beresp.ttl = 3600s; } } } return (deliver); } } sub vcl_deliver { if (req.http.X-Varnish-Faked-Session) { set resp.http.Set-Cookie = req.http.X-Varnish-Faked-Session + "; expires=" + resp.http.X-Varnish-Cookie-Expires + "; path=/"; if (req.http.Host) { if (req.http.User-Agent ~ "^(?:ApacheBench/.|.Googlebot.|JoeDog/.Siege.|magespeedtest.com|Nexcessnet_Turpentine/.*)$") { set resp.http.Set-Cookie = resp.http.Set-Cookie + "; domain=" + regsub(req.http.Host, ":\d+$", ""); } else { if(req.http.Host ~ "") { set resp.http.Set-Cookie = resp.http.Set-Cookie + "; domain="; } else { set resp.http.Set-Cookie = resp.http.Set-Cookie + "; domain=" + regsub(req.http.Host, ":\d+$", ""); } } } set resp.http.Set-Cookie = resp.http.Set-Cookie + "; httponly"; unset resp.http.X-Varnish-Cookie-Expires; } if (req.http.X-Varnish-Esi-Method == "ajax" && req.http.X-Varnish-Esi-Access == "private") { set resp.http.Cache-Control = "no-cache"; } if (false || client.ip ~ debug_acl) { set resp.http.X-Varnish-Hits = obj.hits; set resp.http.X-Varnish-Esi-Method = req.http.X-Varnish-Esi-Method; set resp.http.X-Varnish-Esi-Access = req.http.X-Varnish-Esi-Access; set resp.http.X-Varnish-Currency = req.http.X-Varnish-Currency; set resp.http.X-Varnish-Store = req.http.X-Varnish-Store; } else { unset resp.http.X-Varnish; unset resp.http.Via; unset resp.http.X-Powered-By; unset resp.http.Server; unset resp.http.X-Turpentine-Cache; unset resp.http.X-Turpentine-Esi; unset resp.http.X-Turpentine-Flush-Events; unset resp.http.X-Turpentine-Block; unset resp.http.X-Varnish-Session; unset resp.http.X-Varnish-Host; unset resp.http.X-Varnish-URL; unset resp.http.X-Varnish-Set-Cookie; } }

[rameshramachandran84-zz]

my concerns is why 404 error happens for certain products.

[miguelbalparda]

@rameshramachandran84 the VCL looks like a normal generated VCL. I don't think this issue is caused by Turpentine because the trimming of URLs usually are made by Magento or the webserver and moreover, some URLs seem to be working.

[rameshramachandran84-zz]

Thanks for reviewing the vcl file . The trimmed url : the 404 url mentioned in the issue also showing correct product data after decoding the ESI . Can you please let me know which file in the turpentine module validate the ESI url and identifies is the url valid or not .

[miguelbalparda]

That would be this part of the code, where /esi/ resolves to EsiController.php and /getBlock/ resolves to Nexcessnet_Turpentine_EsiController::getBlockAction()

[miguelbalparda]

Closing for lack of feedback.