search

idaholab / MontePy

MontePy is the most user friendly Python library (API) to read, edit, and write MCNP input files.
https://www.montepy.org/
MIT License
28 stars 6 forks source link

Memory leaks with copy.deepcopy #514

Open MicahGale opened 2 weeks ago

MicahGale commented 2 weeks ago

So I was making a lot of copies of a cell with copy.deepcopy and it went very badly

import montepy
problem = montepy.read_input("foo")
for i in range(1_000):
    new_cell = copy.deepcopy(problem.cells[1])
    ...

This script kept crashing. It got really slow around i==11 and was using over 3GB of RAM!

This is very telling for when I ran ctrl-C:

^CTraceback (most recent call last):
  File "/home/mgale/dev/montepy/mod_big_model.py", line 11, in <module>
    new_cell = copy.deepcopy(base_cell)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mgale/mambaforge/lib/python3.10/copy.py", line 264, in _reconstruct
    args = (deepcopy(arg, memo) for arg in args)
KeyboardInterrupt

We have a copy recursion problem.

MicahGale commented 2 weeks ago

I think we need __weakref__ for self._problem

MicahGale commented 2 weeks ago

Also to implement #469

MicahGale commented 2 weeks ago

Temporary workaround:

  1. Unlink problem
  2. Copy object
  3. relink object
problem =...
base_cell = problem.cells[1]
base_cell._problem = None
new_cell = copy.deepcopy(base_cell)
base_cell.link_to_problem(problem)
... # make more copies if needed, etc
new_cell.link_to_problem(problem)

This still isn't super memory efficient, but it works well enough.

MicahGale commented 6 days ago

I have a hunch that this is causing #463.

MicahGale commented 5 days ago

Weakref does in fact resolve this, but breaks pickle.

MicahGale commented 5 days ago

I have a hunch that this is causing #463.

I tried fixing #463 with #539, and it does not seem to address the problem at all. But this would help break the infinite recursion that makes debugging impossible.