composewell / streamly

High performance, concurrent functional programming abstractions
https://streamly.composewell.com
Other
866 stars 66 forks source link

`unpin` operation should check the size of array before unpinning #2471

Open adithyaov opened 1 year ago

adithyaov commented 1 year ago

GHC always pins memory > 4k (or 3k). You cannot unpin a large block.

adithyaov commented 9 months ago

We can proceed further with this once we have the reference of the doc where this knowledge is gleaned from.

adithyaov commented 3 weeks ago

The isPinned function returns False even if the Array created clearly is more than 4k.

import Streamly.Internal.Data.MutByteArray

main :: IO ()
main = do
    val <- new 40000000
    print $ isPinned val

The above program returns False.

The pinned bit is probably set with the primitive so regardless of the implementation of how the array is stored we get what we set.

This is problematic. isPinned might return False even if the internal array is pinned.

Given the implementation details, should we modify isPinned to check for the size of the array and return True if it is greater than 3k? But this will depend on the implementation details in GHC. GHC might change this threshold. This is probably not the best way to go forward.

We should leave the code as it is. We can re-copy when pinning. Using pinnedNew we can be 100% sure that the array is pinned.

adithyaov commented 3 weeks ago

The new API should pin memory if it exceeds LARGE_BLOCK_THREASHOLD.