search

diprism / fggs

Factor Graph Grammars in Python
MIT License
13 stars 3 forks source link

einsum nondeterminism exhausts memory #169

Closed ccshan closed 1 year ago

ccshan commented 1 year ago

For a while the numeric result of sum_product computations has been nondeterministic in very insignificant digits. Somehow Newton's method exacerbates this for parsing long strings: When length=50 for example, sometimes it takes a fraction of a minute to compute 2.6261177280410e-25 exactly, but sometimes torch_semiring_einsum.extend.adjust_size tries to repeat a tensor from size 103^4 to size 103^5 and fails to allocate the needed 90GB of memory. Even when the computation does finish, running times vary wildly.

ccshan commented 1 year ago

Ah, replacing some Sets by Dict[,None]s seems to have restored determinism. It would still be nice for einsum planning to take tensor size into account though.

(Thanks to https://pytrace.com/)