search

avendesora / pythonbible

A python library for validating, parsing, normalizing scripture references and retrieving scripture texts (for open source and public domain versions)
https://docs.python.bible
MIT License
60 stars 12 forks source link

pythonbible package exceed memory(Process running mem=660M(129.0%)) usage on Heroku server #33

Closed williams9438 closed 2 years ago

williams9438 commented 3 years ago

Process running mem=660M(129.0%) when using this package on heroku

williams9438 commented 3 years ago

How can i resolve this issue? Any recommendation?

avendesora commented 3 years ago

Can you provide a code example that causes this memory issue? Thanks.

avendesora commented 3 years ago

My guess is that it is due to overly aggressive caching within the pythonbible library. We need to do some real profiling to determine what the optimal max size of the cache ought to be. Currently, it is set to None, which means that the cache can grow indefinitely. For a quick fix, I'm going to change that to be the default max size (128 entries, I think), and will release that change shortly. For a long-term fix, I'll create a project to determine the optimal max size of the cache based on profiling exercises as well as possibly allowing for configurable cache settings.

avendesora commented 3 years ago

I have released a hotfix that might help with this issue. Please upgrade to the latest version of pythonbible and try it out and let me know whether or not it resolves your specific issue. You can upgrade to the latest version by running either:

pip install -U pythonbible

or

pip install pythonbible==0.4.1

Thanks.

williams9438 commented 3 years ago

Sorry for replying late. i looked into the library and i think the reason the memory exceeding was because the library loads all the bible passages into memory at once and the cache grows

i was able to solve this by setting my Heroku server WEB_CONCURRENCY Environment variable WEB_CONCURRENCY = 1

williams9438 commented 3 years ago

My guess is that it is due to overly aggressive caching within the pythonbible library. We need to do some real profiling to determine what the optimal max size of the cache ought to be. Currently, it is set to None, which means that the cache can grow indefinitely. For a quick fix, I'm going to change that to be the default max size (128 entries, I think), and will release that change shortly. For a long-term fix, I'll create a project to determine the optimal max size of the cache based on profiling exercises as well as possibly allowing for configurable cache settings.

Yes