dalcinl
commented
7 years ago @agronholm As you mentioned you are busy, here you have a reproducer with output from runs before and after the changes in this PR. Note how the count (and obviously memory footprint) of bytearray instances keeps going up while you iterate over as_completed() and Executor.map() iterators.
Reproducer script:
# file: reproducer.py
import gc
import time
from pympler import summary
from pympler import muppy
from concurrent.futures import ThreadPoolExecutor, as_completed
def random_data(*args):
time.sleep(1)
return bytearray(1024*1024*2)
def gen_executor0():
with ThreadPoolExecutor(1) as executor:
futures = [executor.submit(random_data) for x in range(5)]
for future in as_completed(futures):
futures.remove(future)
future = None
yield
def gen_executor1():
with ThreadPoolExecutor(1) as executor:
for y in executor.map(random_data, range(5)):
y = None
yield
def main(gen_executor):
for cycle in gen_executor():
gc.collect()
summary.print_(summary.summarize(muppy.get_objects()), limit=4)
if __name__ == '__main__':
print("as_completed()")
main(gen_executor0)
print("executor.map()")
main(gen_executor1)Run before this PR:
$ python reproducer.py
as_completed()
types | # objects | total size
=========== | =========== | ============
bytearray | 1 | 2.00 MB
dict | 767 | 1.14 MB
str | 8468 | 803.97 KB
code | 2595 | 324.38 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 2 | 4.00 MB
dict | 768 | 1.14 MB
str | 8468 | 803.97 KB
code | 2595 | 324.38 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 3 | 6.00 MB
dict | 767 | 1.14 MB
str | 8468 | 803.97 KB
code | 2595 | 324.38 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 4 | 8.00 MB
dict | 766 | 1.14 MB
str | 8468 | 803.97 KB
code | 2595 | 324.38 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 5 | 10.00 MB
dict | 765 | 1.14 MB
str | 8466 | 803.89 KB
code | 2595 | 324.38 KB
executor.map()
types | # objects | total size
=========== | =========== | ============
bytearray | 1 | 2.00 MB
dict | 766 | 1.14 MB
str | 8465 | 803.84 KB
code | 2596 | 324.50 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 2 | 4.00 MB
dict | 765 | 1.14 MB
str | 8465 | 803.84 KB
code | 2596 | 324.50 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 3 | 6.00 MB
dict | 764 | 1.14 MB
str | 8465 | 803.84 KB
code | 2596 | 324.50 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 4 | 8.00 MB
dict | 763 | 1.14 MB
str | 8465 | 803.84 KB
code | 2596 | 324.50 KB
types | # objects | total size
=========== | =========== | ============
bytearray | 5 | 10.00 MB
dict | 762 | 1.14 MB
str | 8463 | 803.76 KB
code | 2596 | 324.50 KBRun after this PR:
$ python reproducer.py
as_completed()
types | # objects | total size
======= | =========== | ============
dict | 765 | 1.14 MB
str | 8470 | 804.44 KB
code | 2596 | 324.50 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 764 | 1.14 MB
str | 8470 | 804.44 KB
code | 2596 | 324.50 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 761 | 1.14 MB
str | 8470 | 804.44 KB
code | 2596 | 324.50 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 758 | 1.13 MB
str | 8470 | 804.44 KB
code | 2596 | 324.50 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 755 | 1.13 MB
str | 8461 | 804.04 KB
code | 2596 | 324.50 KB
type | 167 | 148.64 KB
executor.map()
types | # objects | total size
======= | =========== | ============
dict | 764 | 1.14 MB
str | 8467 | 804.30 KB
code | 2597 | 324.62 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 761 | 1.14 MB
str | 8467 | 804.30 KB
code | 2597 | 324.62 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 758 | 1.13 MB
str | 8467 | 804.30 KB
code | 2597 | 324.62 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 755 | 1.13 MB
str | 8467 | 804.30 KB
code | 2597 | 324.62 KB
type | 167 | 148.64 KB
types | # objects | total size
======= | =========== | ============
dict | 752 | 1.13 MB
str | 8458 | 803.91 KB
code | 2597 | 324.62 KB
type | 167 | 148.64 KB
agronholm
Python issues: