502 Bad Gateway File Save Error in Cape Town prod, dev sandboxes

[eefaye]

Issue When attempting to save changes to .ipynb files in the Cape Town dev or prod sandboxes, the browser says "Saving started" then produces the error "File Save Error for Invalid response: 502 Bad Gateway". The browser will then say "Saving failed". The file will not be saved. Any changes since the last save are not captured, and right click>Download only downloads the last saved version of the file.

This issue is localised to the Cape Town sandboxes, Oregon does not display this issue.

Frequency Most files - intermittent. Error occurs on a daily basis, perhaps for about half an hour at a time, sometimes it will resolve itself temporarily Some files - consistently refuse to save. See Frequently_used_code/Imagery_on_web_map.ipynb. This is sometimes solved by downloading and reuploading the file (that solution worked for Burnt_area_mapping.ipynb, but does not exclude the file from the intermittent error The error occurs at different times of day and has no discernible pattern for file name or size. It so far has only affected .ipynb notebooks.

Severity High, it is not possible to reliably save work, especially problematic at the end of the work day. Not all files will experience the error at the same time.

Steps to recreate (consistent error)

1. Log into a Cape Town sandbox.
2. Open Frequently_used_code/Imagery_on_web_map.ipynb.
3. Make a change (markdown or code, anything).
4. Press Ctrl+S or the 'Save' button.

Steps to recreate (intermittent error)

1. Log into a Cape Town sandbox.
2. Create or open any .ipynb notebook and make a change.
3. Save the notebook.
4. Repeat 2,3 until error occurs.

[alexgleith]

I have a request for the next time this happens, @eefaye and others.

Please open the developer console for your browser, here for Chrome:

And have a look at what is logged there and see if any might be useful.

[eefaye]

@cbur24 @eloise-b @NMoghaddam @fangfy

[fangfy]

User log when getting 502 error

16:32, tidal notebook successfully saved From ~16:55 to 17:07, getting 502 errors trying to save Sentinel-1 notebook 17:07, S1 notebook saved succesfully

[NMoghaddam]

Both yesterday afternoon and this morning getting 502 error and autosaving failure in production sandbox.

[alexgleith]

That one is great, @NMoghaddam. Ping @whatnick and @NikitaGandhi it looks to me like the "cancelled due to load timeout" is an indicator of latency being an issue.

Although, checking in the code at those functions could be interesting. Can you ping me when you have these messages alive in a browser window, @NMoghaddam? We can explore the javascript and see if anything is obvious.

[cbur24]

@alexgleith more 502 gateways errors trying to save notebooks.

[cbur24]

Anecdotally this issue is happening less on the big instances

[fangfy]

Not sure if this is useful but it seems a notebook without output is saved automatically. Once the notebook if evaluated, I get a 502, probably because it's much bigger with images.