!!! `project` causes memory inconsistencies

Profpatsch commented 2 years ago

project (and potentially also inject, we haven’t verified`, will cause undefined behavior, which we noticed & verified in a production system.

The bug only appears with optimizations enabled (only tested -O2), not in ghci

I don’t have a minimal repro (yet), but I can describe what we did:

There was one function which loaded some data in IO, constructed a ist of rec, then before returning used project to reduce unnecessary fields in every list element:

loadData :: IO (Record '[ "foo" := Text ])
loadData = do
  listOfRes <- <fetch data>
  pure 
    $ map 
         (project @'["foo" := Text, "bar" := Int] @'["foo" := Text])
         listOfRes

And another function that pulled a subset of fields from the record, like this:

requestBody = do
  dat <- loadData

  let mapDat d = (Json.object [
         ("foo", get #foo d)
        ])

   pure $ Json.Array (map mapDat dat)

Let’s say the final data was

[ 
  { foo = "xx" },
  { foo = "yy" },
  { foo = "zz" },
]

Then what would be actually returned was (!!):

[ 
  { foo = "xx" },
  { foo = "xx" },
  { foo = "xx" },
]

now, if you rewrote the function to deeply evaluate the projected record:

loadData :: IO (Record '[ "foo" := Text ])
loadData = do
  listOfRes <- <fetch data>

  print listOfRes   -- <- deeply evaluate the record by printing it

  pure 
    $ map 
         (project @'["foo" := Text, "bar" := Int] @'["foo" := Text])
         listOfRes

It would again return

[ 
  { foo = "xx" },
  { foo = "yy" },
  { foo = "zz" },
]

When we rewrote the original function to construct a new record instead of projecting:

loadData :: IO (Record '[ "foo" := Text ])
loadData = do
  listOfRes <- <fetch data>
  pure 
    $ map 
         (\res -> rcons #foo (get #foo res) rnil)
         listOfRes

The problem went away.

Profpatsch commented 2 years ago

cc @sheaf who wrote project a while ago

sheaf commented 2 years ago

I don't recall the exact details, but I think I saw an issue like this start happening with GHC 9.0 or 9.2. The issue went away with -fno-full-laziness. What version of GHC are you using, and does the problem still happen with -fno-full-laziness?