search

cpan-testers / cpantesters-api

An API in to data held by CPAN Testers: Test reports and CPAN uploads
Other
4 stars 4 forks source link

Find out why POST bodies are being left in /tmp #23

Closed preaction closed 6 years ago

preaction commented 6 years ago

Something is happening where a test reporter submits a report, a file is written to /tmp (matching /tmp/mojo.tmp.* with the test report inside, and the file is being left there. Eventually, /tmp fills up and then the box starts having problems (in this case, the test reports stopped being processed). These test reports are all about 17 MB each, which means something.

In the 5 minutes since I deleted all of /tmp/mojo.tmp.*, 3 new files have been written. Later, one of those was removed automatically, but 2 remain...

preaction commented 6 years ago

This looked to be a combination of multiple things:

  1. The MOJO_MAX_MESSAGE_SIZE was 16MB (the default), and if the message was bigger, the request would end immediately.
  2. Once I raised MOJO_MAX_MESSAGE_SIZE to 32MB, there remained a single report of 29MB that could not be processed. Mojo::JSON was trying to parse the pathological amount of escape sequences in the serialized Metabase::Fact object and was ballooning the process memory usage to >7GB (which triggered the OOM killer).

Both of these are now fixed, and I haven't seen any new mojo.tmp files.