toshan2014
commented
8 years ago After consulting Reliam this issue was resolved. However they could not pinpoint the source. Below is the HipChat conversation from Friday 13th May 2016:
Hey guys, we having an intermittent issue on akc.org. Can you help? Lance Allen 11:00 AM sure, what's going on? Simon Paulenoff 11:01 AM on akc.org homepage when you hover over the featured stories section Lance Allen 11:02 AM oh Simon Paulenoff 11:02 AM Hide card intermittently we will see that the stories are linking out to specific prod servers such ashttp://akc-prod-web07.reliam.com/news/judges-for-2017-westminster-kennel-club-show Thumbnail image Judges For 2017 Westminster Kennel Club Show Announced www.akc.org
The judging panel for the 2017 Westminster Kennel Club dog show includes Mr. Thomas H. Bradley III, scheduled to decide Best in Show. Lance Allen 11:02 AM right, the hostname of the web server Toshan Reebye 11:02 AM not that these are relative URLs not absolute Simon Paulenoff 11:02 AM http://akc-prod-web07.reliam.com/news/judges-for-2017-westminster-kennel-club-show Toshan Reebye 11:02 AM note Lance Allen 11:07 AM is content ever edited directly on a web node, or do you always use akc.org/admin (or whatever the EE admin console URL is) Simon Paulenoff 11:07 AM always the cms Lance Allen 11:12 AM investigating something on server side... Toshan Reebye 11:13 AM also note that this does not occur in stage and is intermittent on production, which is making me think it's dependent on what server is being hit Simon Paulenoff 11:45 AM @lance any update onyour end? Lance Allen 11:47 AM we're still troubleshooting. it's difficult to narrow down, but we can definitely reproduce it it's not always one server either, if you view source and search "akc-prod-web", I usually see 2 different servers being referenced Simon Paulenoff 11:48 AM ya i have seen 07, 01, and 02 Toshan Reebye 11:49 AM same it seems to be those three for me too Lance Allen 11:49 AM im getting 5 and 6 Simon Paulenoff 11:50 AM so that means only 3 is working correctly Brian Dunbar 11:57 AM question - are the featured stories supposed to read left to right in order of publication? I was pretty sure on some refreshes the GSD story (May 9) showed up on the right, now it's on the left and the Canine health story (may 12) is on the right Toshan Reebye 12:03 PM i see the Canine story ( may 12 ) in the middle Judges ( may 10 ) on the left Heroic ( may 9 ) on the right in this order 10 May - 12 May - 9 May Brian Dunbar 12:06 PM i hit refresh and it's back in the correct order, by date. Not related, I'm sure, but it jumped out at me as 'odd'. Toshan Reebye 12:08 PM when it's in the right order (12 - 10 - 9 ) it seems to be hitting akc.org/... 10 - 12 - 9 I was getting Web 7 Toshan Reebye 12:18 PM I'm getting Web 3 too actually Brian Dunbar 12:18 PM this might be a false trail, but it's (10 - 12 - 9) and linking to prod03 12 - 9 - 10 - prod02 10-12-9 prod0-3 Toshan Reebye 12:25 PM Actually the order is unrelated - I see that in the file it's set to sort=random so it would be normal for the order to change Kyle Bertelsen 12:36 PM Hello, folks! I'm checking in to see what the story might be Lance Allen 12:42 PM still troubleshooting Kyle, not entirely sure what the culprit would be Toshan Reebye 12:52 PM Maybe this could help - everytime I'm logged in I don't face the issue. It seems to happen only when logged out Kyle Bertelsen 12:53 PM And from previous experimentation, I believe that when logged-in, we're not getting cached versions. Toshan Reebye 1:09 PM Unless something changed, it's doing it when I'm logged in too now. Kyle Bertelsen 1:10 PM I see the problem when loading cache03 (via my /etc/hosts) Lance Allen 1:11 PM what about if you change it to cache02 or 1? Kyle Bertelsen 1:12 PM When not logged-in, no problem on 01 or 02, problem on all 3 when logged-in Lance Allen 1:15 PM i want to drop cache, but I'm hesitant Kyle Bertelsen 1:16 PM what makes you hesitant? Lance Allen 1:16 PM it puts more stress on the web nodes. last time we flushed cache EE started throwing errors Kyle Bertelsen 1:16 PM Ah! Yeah, that's a cautionary tale, isn't it? Brian Dunbar 1:17 PM @JoshuaMcSorley I put 22758 in a different board and lowered the priority. Joshua McSorley 1:17 PM Copy that Lance Allen 1:18 PM wrong room guys Brian Dunbar 1:18 PM .. .sorry about that Joshua McSorley 1:18 PM Our apologizes to AKC Kyle Bertelsen 1:19 PM Don't sweat it Toshan Reebye 1:21 PM not sure if it's related but line 821 was changed in .htaccess - the following was added: "E=shopurl:shop.akc.org" Lance Allen 1:23 PM Hide card what's weird is if you go directly to a web node, all the links are wrong EXCEPT for featured stories. http://akc-prod-web03.reliam.com Thumbnail image American Kennel Club - The Dog’s Champion www.akc.org
AKC.org offers information on dog breeds, dog ownership, dog training, health, nutrition, exercise & grooming, registering your dog, AKC competition events and affiliated clubs to help you discover more things to enjoy with your dog. it just feels like going against the grain Kyle Bertelsen 1:23 PM That is weird. Another exception: the featured breed (Skye Terrier) Seems that isn't the case when using /etc/hosts to direct akc.org to web03 Checking the others Okay, weird: Going through, first problems found in web06 -- which is linking to web01 (featured stories only) and same on web05 rechecking web06 now Confirmed. Lance Allen 1:34 PM you're setting hosts file of the web node to akc.org and it's linking to akc-prod-web01? rather, you're setting akc.org to an IP of a web node ? Kyle Bertelsen 1:35 PM Correct I'll paste in, so you can see
PRODUCTION
10.30.42.17 akc.org www.akc.org # akc-prod-web05
10.30.42.18 akc.org www.akc.org # akc-prod-web06
10.30.42.19 akc.org www.akc.org # akc-prod-web07
10.30.42.8 akc.org www.akc.org # akc-prod-web01
10.30.42.9 akc.org www.akc.org # akc-prod-web02
10.30.42.10 akc.org www.akc.org # akc-prod-web03
Show less here, I'm set to use web06 And when I do that with web05 or 06, I see featured stories links to web01 Lance Allen 1:39 PM hm. when I set my hosts file to what you linked there, the links all appear fine Kyle Bertelsen 1:40 PM I see the problem links only in the featured stories section Lance Allen 1:43 PM i only see the issue if I hit the cache servers first. If i point akc.org to any of the web nodes, all the links show up as they should: www.akc.org if i set hosts to a cache server, I get the issue we should probably just drop cache Lance Allen 1:52 PM i also cleared browser cache and cannot reproduce anymore Kyle Bertelsen 1:53 PM I'll try again Kyle Bertelsen 1:58 PM Can't reproduce logged-out. Trying logged-in. Can't reproduce logged-in, either. Going to make one last pass at each cache server Toshan Reebye 2:04 PM seem fine to me too Simon Paulenoff 2:05 PM seems to be working on my end Lance Allen 2:07 PM im testing other devices where ive never visited and not seeing it there either. are we all in agreement that it's browser caching? Kyle Bertelsen 2:07 PM Okay, I've tested every way I can think to test, and I can no longer reproduce. Any thoughts, Lance, on what all happened or what the fix was? @lance ? I don't think it had anything to do with browser caching Lance Allen 2:12 PM not entirely sure. we made no changes on our end i played with my hosts file and cleared my browser cache Kyle Bertelsen 2:12 PM Huh. So we didn't learn anything, I guess? Lance Allen 2:14 PM i guess so. im not sure why the hostname of the server would even populate there. i don't ever connect to them directly that way. i would suspect weirdness with varnish Kyle Bertelsen 2:14 PM Yeah, me too. Okay, thanks, Lance! Have a good weekend! Many, many thanks!
kylejbertelsen
We need to fix issue where certain homepage links are linking to individual reliam server links on the homepage
Specifically in the featured stories section:
Seems to occur when only the string is entered into the cms and not the full URL.
For example issue only occurs when a link is in this format /{section}/{sub section}/{content}/ It works fine if linked in the cms in this format: http://www.akc.org/{section}/{sub section}/{content}/
I believe this is related to #242
Please add an estimate to this ticket