503 error on edit of draft content item

[paregorios]

Initial error occurred approximately between 1:40 and 1:51 pm US Eastern Time on Thursday, 7 November 2024.

what happened:

• User logged into ISAW website
• User navigated to https://isaw.nyu.edu/publications/isaw-monographs/ain-el-gedida
• User retracted the published content item to drafting (because they can't check out a working copy, see: #534)
• User selected "edit" and was successfully redirected to https://isaw.nyu.edu/publications/isaw-monographs/ain-el-gedida/edit?_authenticator=aef0cd360134bf5966eba2a923d4cbe7555b41cc
• User added https://doi.org/10.18574/nyu/9781479848034.001.0001 to a new line in the Access URIs (one per line) field.
• User clicked "save", was redirected to https://isaw.nyu.edu/publications/isaw-monographs/ain-el-gedida/@@edit and shown the following error message:
• The same process and errors occur if the user chooses "cancel" instead of "save".
• User has repeated these steps, using both Chrome and Firefox and the same thing happens.

[paregorios]

Followup:

• As a user with admin privileges, I logged in (using firefox), edited the page, added the desired value to the indicated field, and saved the content item without incident.

[paregorios]

This is the same sort of problem we were seeing with #536.

[paregorios]

After I made the followup change (see above) myself, the user went back to chrome, navigated to the page, tried editing it again, and still on save got redirected to https://isaw.nyu.edu/publications/isaw-monographs/ain-el-gedida/@@edit and sees:

Error 503 Backend fetch failed
Backend fetch failed
Guru Meditation:
XID: 11819936
Varnish cache server

[paregorios]

I told the user to perform the following steps:

• please go to chrome -> settings -> privacy and security -> delete browsing data and in hte “Delete browsing data” dialog box:
• set “time range” to “last 24 hours”
• DEselect the “browsing history” checkbox
• SELECT the “cookies and other side data”
• SELECT “cached images and files”
• Then click the “Delete data” button.
• Then choose Chrome -> Quit Google Chrome
• Then restart chrome, navigate to the isaw website, log back in, and try the edit again.

[paregorios]

User reports that following the above steps has allowed them to edit, save, and submit the publication item in question, their immediate problem is resolved.

But this error remains the same as it ever was, despite previous speculative interventions and adjustments.

[paregorios]

I have log extracts available from the relevant time period for the original error, but do not want to post them here.

[alecpm]

I think we might be able to better understand the issue with a more targetted elimination of browsing data. If the error appears and persists, could the try going into into Chrome DevTools -> Application -> Cookies and delete the cookies:

• __ac
• _saml2_session
• _saml2_attributes
• idp_id

If they exist. They will need to login again, but I'm curious if that resolves the issue.

[alecpm]

It might also be helpful to know if he has any other cookies (besides _ga* and _gid from Google) on the .nyu.edu or isaw.nyu.edu domains.

[paregorios]

@alecpm Report from a user this morning:

I received the following error message when trying to visit a page on the ISAW website: "400 Bad Request". After refreshing, the page loaded, but before that I got this screen capture:

I got the user to give me screen captures of cookies in the dev too. I'll post these in a thread on slack.

[alecpm]

That Apache bad request error is one I’ve seen before on Dumbarton Oaks (well, the nginx version of the error), which is using a similar SAML setup (with Azure AD rather than Shibboleth). It’s sort of the reverse of the error we were seeing with the Varnish 503 errors in the backend response. There’s a limit to the size of input request headers that the web server allows (to prevent buffer overflow attacks). We’ll probably need to increase that limit to support this (truly remarkable) number of cookies.

[alecpm]

I've doubled the size of the request header size and line limits, which should prevent this error going forward. Let me know if it comes up again.

[alecpm]

Both the apache and varnish config updates are now included in the current and future (Plone 5.2/Python 3) ansible config.