search

overlay-market / overlay-risk

Risk metrics for various data streams
MIT License
13 stars 4 forks source link

[WIP] Recreating R14 error with minimal code and finding a fix #33

Open deepsp94 opened 2 years ago

deepsp94 commented 2 years ago

Problem: Running the following script on Linux throws the R14 error (excessive memory use).

image

The graph shows that memory use increases continuously. At the 0.5 GB mark, Heroku (which runs on linux OS) periodically throws the R14 error and at the 1 GB mark it shuts down and restarts the whole process (dyno). Script: https://github.com/overlay-market/overlay-risk/blob/heroku-r14/scripts/r14_error.py

Background:

  1. The script is using pystable, which is a python wrapper of the C library libstable. The excessive memory use is due to the use of pystable.
  2. Pystable is only usable on mac computers or linux OS with glibc versions 2.31/2.32 (heroku runs on linux with glibc 2.31)
  3. Running the same script on a mac computer (m1) doesn't result in monotonically increasing use of memory. I've observed this behaviour on Linux only.

Objective: The script should work on Linux without memory issues.

Resources to get started: How to install pystable: https://pypi.org/project/pystable/ Pystable repo: https://github.com/overlay-market/pystable Libstable resources: https://www.jstatsoft.org/article/view/v078i01 About R14 error in the context of ruby applications: https://devcenter.heroku.com/articles/ruby-memory-use