gitlab-importer
commented
1 year ago In Heptapod by @mattip on Oct 13, 2022, 08:56
That version of PyPy is quite old (released Feb 2019). Does the problem reproduce on 7.3.9?
Open gitlab-importer opened 1 year ago
gitlab-importer
commented
1 year ago In Heptapod by @mattip on Oct 13, 2022, 08:56
That version of PyPy is quite old (released Feb 2019). Does the problem reproduce on 7.3.9?
gitlab-importer
commented
1 year ago In Heptapod by @egmkang on Oct 13, 2022, 09:05
@mattip Python 3.6.9 (7.3.1+dfsg-4, Apr 22 2020, 05:15:29) has this problem too
gitlab-importer
commented
1 year ago In Heptapod by @egmkang on Oct 13, 2022, 11:10
Total memory consumed:
GC used: 9.4MB (peak: 21.3MB)
in arenas: 6.7MB
rawmalloced: 1.7MB
nursery: 1.0MB
raw assembler used: 271.8kB
-----------------------------
Total: 9.6MB
Total memory allocated:
GC allocated: 18.7MB (peak: 23.0MB)
in arenas: 14.3MB
rawmalloced: 6.0MB
nursery: 1.0MB
raw assembler allocated: 1.0MB
-----------------------------
Total: 19.7MB
Total time spent in GC: 19.474Python 3.9.12 (05fbe3aa5b0845e6c37239768aa455451aa5faba, Mar 29 2022, 08:15:34)
[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.@mattip
gitlab-importer
commented
1 year ago In Heptapod by @egmkang on Oct 14, 2022, 09:41
getattr(dbm.lib, dbm.funcs['fetch'])(db._aobj, dat)just call this code, memory will growth
gitlab-importer
commented
1 year ago In Heptapod by @mattip on Oct 14, 2022, 10:10
>>>> Python 3.6.9 (7.3.1+dfsg-4, Apr 22 2020, 05:15:29)
[PyPy 7.3.1 with GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import dbm
>>>> getattr(dbm.lib, dbm.funcs['fetch'])(db._aobj, dat)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'dbm' has no attribute 'lib'
>>>> import dbm.lib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'dbm.lib'In Heptapod by @egmkang on Oct 14, 2022, 10:20
@mattip pypy3 use this:
import dbm
import random
import gc
db = dbm.open("test", "c")
db.close()
db = dbm.open("test", "r")
total_count = 0
while True:
value = db.get("13131313131313", "")
total_count += 1
if total_count % 300000 == 0:
gc.collect_step()
print(gc.get_stats(False))In Heptapod by @mattip on Dec 17, 2022, 22:25
This code uses ctypes to call a function. If I short-circuit the __call__ to only wrap the arguments, skip the call, and return None, allocated memory grows. So it seems something is leaking out when calling _conv_param. I think it has to do with using memmove to copy a buffer around here since when I comment that out (for the db.get call), the allocated memory no longer grows.
gitlab-importer
commented
1 year ago In Heptapod by @mattip on Dec 18, 2022, 06:48
Adding this as a blocker for v7.3.11 since it seems to be a general ctypes issue and not specific to dbm
gitlab-importer
commented
1 year ago In Heptapod by @mattip on Dec 18, 2022, 09:47
If I short-circuit the
__call__to only wrap the arguments, skip the call, and return None, allocated memory grows.
A few hours later, I cannot reproduce this. Now the memory only grows when I actually do the call. The way you have set up the test, it is failing to find the data in the database (the database is empty). If I write a single key to the database, I do not see a leak. So I think this is something around the error path. Can you confirm that is correct?
If so, this is really only a dbm problem, and not a general one with ctypes. I am removing the milestone.
In Heptapod by @egmkang on Oct 13, 2022, 08:49
this is a simple example:
memory_leak.py
Python 2.7.13 PyPy 7.0.0 Ubuntu 20.04
pypy3 has this problem too:
pmap 1:
run for a while:
the diff is: