google-research / dex-lang

Research language for array processing in the Haskell/ML family
BSD 3-Clause "New" or "Revised" License
1.58k stars 107 forks source link

Fix box memory management #1175

Closed dougalm closed 1 year ago

dougalm commented 1 year ago

The bug: we use null pointers to represent uninitialized boxes. These can occur in the unused side of a sum type, for example the value Nothing : Maybe (List Float) will have a null pointer for the List Float side. We check whether the pointer is null before doing a copy. Not only would it be bad to dereference the null pointer, but allocating the memory for the result would also fail because we'd be determining the allocation size based on the garbage value in the "size" part of the List Float. Previously in the null case we just didn't do any copy, but actually we need to go further than that to maintain the invariant that "garbage size implies null pointer". We need to explicitly put a null pointer in the destination because the current value might be non-null. Note that this is not because it might contain uninitialized memory -- the initialization we do at allocation time handles that -- but because it might have had a legitimate non-null pointer stored there, such as if a Ref (Maybe (List Float)) previously contained the Just case.

In implementing the fix, it ended up being easier to just clean up the way we handle boxed memory generally. A few changes: