Fix memory leak

[sergei-maertens]

There is a memory leak in the /api/ endpoint - it's causing the memory usage of the container to grow and eventually get OOM-killed (because we do set our memory limits).

The /api/ endpoint is used as health check, but of course this should not be memory leaking in the first place.

Possible mitigations:

• cache the API schema and serve the cache
• use another _health endpoint for the health check

[sergei-maertens]

Some debugging with mem_top seems to confirm that drf_spectacular is problematic:

refs:
5742    <class 'list'> ['# module pyparsing.py\n', '#\n', '# Copyright (c) 2003-2018  Paul T. McGuire\n', '#\n', '# Permiss
5404    <class 'dict'> {PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/rollup.py'):
3364    <class 'dict'> {140013453740288: <weakref at 0x7f576b8d4db0; to 'type' at 0x7f576c2c8900 (type)>, 140013453737696: 
2562    <class 'dict'> {<weakref at 0x7f576512bf90; to 'ActivityViewSet' at 0x7f576a7f40a0>: <drf_spectacular.utils.extend_
2514    <class 'dict'> {(('urn:oasis:names:tc:opendocument:xmlns:animation:1.0', 'audio-level'), None): <function cnv_doubl
2180    <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
2180    <class 'tuple'> (<module 'MarkupPy' from '/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/MarkupPy/__init__.p
2041    <class 'frozenset'> frozenset({PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/ro
1526    <class 'list'> ['"""Thread module emulating a subset of Java\'s threading model."""\n', '\n', 'import os as _os\n',
1311    <class 'dict'> {'__name__': 'lib', '__doc__': None, '__package__': None, '__loader__': None, '__spec__': None, '_or

bytes:
147552   {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
73816    {140013453740288: <weakref at 0x7f576b8d4db0; to 'type' at 0x7f576c2c8900 (type)>, 140013453737696: 
73816    {PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/rollup.py'):
65752    frozenset({PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/ro
47160    ['# module pyparsing.py\n', '#\n', '# Copyright (c) 2003-2018  Paul T. McGuire\n', '#\n', '# Permiss
36960    {'__module__': 'lxml.etree', '__doc__': 'Libxml2 error types', '__dict__': <attribute '__dict__' of 
36960    {'application/javascript': ['.js', '.mjs'], 'application/json': ['.json'], 'application/manifest+jso
36960    {(('urn:oasis:names:tc:opendocument:xmlns:animation:1.0', 'audio-level'), None): <function cnv_doubl
36960    {'CRYPTOGRAPHY_PACKAGE_VERSION': <cdata 'char *' 0x7f57665bf000>, 'Cryptography_HAS_EC2M': 1, 'Crypt
36960    {'__name__': 'lib', '__doc__': None, '__package__': None, '__loader__': None, '__spec__': None, '_or

types:
89589    <class 'dict'>
45074    <class 'list'>
35322    <class 'tuple'>
33857    <class 'function'>
18048    <class 'collections.OrderedDict'>
10647    <class 'weakref'>
9547     <class 'cell'>
5134     <class 'type'>
4784     <class 'pathlib.PosixPath'>
4183     <class 'django.core.validators.ProhibitNullCharactersValidator'>

some time later

refs:
5742    <class 'list'> ['# module pyparsing.py\n', '#\n', '# Copyright (c) 2003-2018  Paul T. McGuire\n', '#\n', '# Permiss
5404    <class 'dict'> {PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/rollup.py'):
4354    <class 'dict'> {<weakref at 0x7f576512bf90; to 'ActivityViewSet' at 0x7f576a7f40a0>: <drf_spectacular.utils.extend_
3364    <class 'dict'> {140013453740288: <weakref at 0x7f576b8d4db0; to 'type' at 0x7f576c2c8900 (type)>, 140013453737696: 
2514    <class 'dict'> {(('urn:oasis:names:tc:opendocument:xmlns:animation:1.0', 'audio-level'), None): <function cnv_doubl
2180    <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
2180    <class 'tuple'> (<module 'MarkupPy' from '/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/MarkupPy/__init__.p
2041    <class 'frozenset'> frozenset({PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/ro
1526    <class 'list'> ['"""Thread module emulating a subset of Java\'s threading model."""\n', '\n', 'import os as _os\n',
1311    <class 'dict'> {'__name__': 'lib', '__doc__': None, '__package__': None, '__loader__': None, '__spec__': None, '_or

bytes:
147552   {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
73816    {140013453740288: <weakref at 0x7f576b8d4db0; to 'type' at 0x7f576c2c8900 (type)>, 140013453737696: 
73816    {PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/rollup.py'):
73816    {<weakref at 0x7f576512bf90; to 'ActivityViewSet' at 0x7f576a7f40a0>: <drf_spectacular.utils.extend_
65752    frozenset({PosixPath('/home/bbt/.virtualenvs/zac/lib/python3.9/site-packages/elasticsearch/client/ro
47160    ['# module pyparsing.py\n', '#\n', '# Copyright (c) 2003-2018  Paul T. McGuire\n', '#\n', '# Permiss
36960    {'__module__': 'lxml.etree', '__doc__': 'Libxml2 error types', '__dict__': <attribute '__dict__' of 
36960    {'application/javascript': ['.js', '.mjs'], 'application/json': ['.json'], 'application/manifest+jso
36960    {(('urn:oasis:names:tc:opendocument:xmlns:animation:1.0', 'audio-level'), None): <function cnv_doubl
36960    {'CRYPTOGRAPHY_PACKAGE_VERSION': <cdata 'char *' 0x7f57665bf000>, 'Cryptography_HAS_EC2M': 1, 'Crypt

types:
133115   <class 'dict'>
64550    <class 'list'>
42050    <class 'tuple'>
35333    <class 'function'>
30305    <class 'collections.OrderedDict'>
12517    <class 'weakref'>
10749    <class 'cell'>
6997     <class 'django.core.validators.ProhibitNullCharactersValidator'>
6967     <class 'rest_framework.validators.ProhibitSurrogateCharactersValidator'>
6494     <class 'drf_spectacular.plumbing.ResolvedComponent'>

[sergei-maertens]

526731cbf276a60e962be1ae74698cf6445e49d4 is the first step in treating the symptoms, but of course that doesn't fix the memory leak itself yet

[sergei-maertens]

Upstream bug report: https://github.com/tfranzel/drf-spectacular/issues/597