Last deploy causing 503s on prod

[cdrini]

Summary

• What is wrong? Observing Intermittent 503s on prod and sustained high memory usage resulting in ol-web3 becoming unresponsive.

• What caused it? #2838 in deploy-2020-01-16 ; exact issue in that PR yet-to-be-determined.
• What fixed it? Reverting the deploy until the PR in question was found, and then reverting the PR.
• Followup actions:
  • Tentatively switched from Thursday deploys to Tuesday deploys; this allows for more people to be around to observe errors, and allows Charles (for whom PST Thurs is Fri) to also be around to monitor deploys. This would have allowed us to catch the error faster and to link it to the deploy sooner.
• Action Taken:
  • Rolled back to previous deploy (deploy-2020-01-09)
  • @mekarpeles tested compress.py changes on ol-web3
  • @cdrini tested get_metadata changes changes on ol-web3

Steps to close

1. [x] Assignment: Is someone assinged to this issue? (joint effort between the two)
2. [x] Labels: Is there an Affects: label applied?
3. [x] Diagnosis: Add a description and scope of the issue
4. [x] Updates: As events unfold, is noteable provenance documented in issue comments? (i.e. useful debug commands / steps / learnings / reference links)
5. [x] "What caused it?" - please answer in summary
6. [x] "What fixed it?" - please answer in summary
7. [x] "Followup actions:" actions added to summary

[cdrini]

We rolled back the deploy yesterday to resolve

[cdrini]

We were noticing high memory usage on ol-web[34] as well as high swap rates on ol-web4. This suggests a memory leak.

Committed memory on ol-web4 (web3 was similary)

Swap memory on ol-web3 (0 on web4) http://ol-web3.us.archive.org:8088/mrtg/committed.html

[cdrini]

Here's what went out on prod on deploy: https://github.com/internetarchive/openlibrary/compare/deploy-2020-01-09...deploy-2020-01-16?w=1

This files seem like the largest changes:

• https://github.com/internetarchive/openlibrary/compare/deploy-2020-01-09...deploy-2020-01-16?w=1#diff-76be93633d51ef7465bfa65e0eb27536
• https://github.com/internetarchive/openlibrary/compare/deploy-2020-01-09...deploy-2020-01-16?w=1#diff-3e6660b9e68c94c878f3d23b229ff2c8
• https://github.com/internetarchive/openlibrary/compare/deploy-2020-01-09...deploy-2020-01-16?w=1#diff-58d924a8f655f9667826a405b04aa1a1

[tfmorris]

I'm not sure what 2 files you're talking about since those three links all show the same contents, which looks like a list of commits.

[cdrini]

Oh, that seems like a Github bug. Click on "files changed" tab, ~then press enter in the URL bar~. Does that work? The hash should go to the right file in the diff

[tfmorris]

That doesn't work for me either. Do these files have names?

[mekarpeles]

We've reverted compress.py on 970b31b on ol-web3 and are waiting to see what happens with memory:

http://ol-web3.us.archive.org:8088/mrtg/committed.html

[mekarpeles]

compress.py revert did not do the trick, following plan to test get_metadata

Wondering if we may be memoizing more content?

We froze up on ol-web3 when testing our compress.py rollback and restarted here: https://gnt-webmgr.us.archive.org/cluster/cluster1/ol-web3.us.archive.org#overview

[cdrini]

This is the reverse diff for the get_metadata PR:

git diff 836044c3 836044c3~

[cdrini]

We're going to try the above command now :+1:

[cdrini]

Yep; with the revert on web3 for ~1hr, we didn't see the spike, so it looks like the memory issue was introduced in #2838

http://ol-web3.us.archive.org:8088/mrtg/committed.html

[tfmorris]

In addition to all the code reorganization, there's also a switch from urllib to Requests, which I consider fairly significant. I wouldn't expect it to cause an issue, but it's worth investigating.

[tfmorris]

Possible Requests based memory leak on Python 2.7 https://github.com/psf/requests/issues/4553

[tfmorris]

Is the "Steps to Close" a thing? If so, why is this closed?

With my teams, I'd also address:

• "What can we improve from a process standpoint to minimize the likelihood of this error or similar ones happen in the future?" -- It seems like load testing of the dev/staging server might be one thing to consider. -- Red/Green deploys so that we don't take out the whole site and can roll back easily might be another improvement

[cdrini]

This is waiting for the "What caused it?" section, which will be determined once #2899 completed.

[hornc]

What caused it, https://github.com/internetarchive/openlibrary/pull/2909#issuecomment-578580052

Existing code was double caching. My first refactor must have enable one of the paths, that while it looked like it should have been used, clearly wasn't. Removing the duplication fixed the memory usage.

Fixed in #2909