search

ezyang / compact

Compact regions library for Haskell
BSD 3-Clause "New" or "Revised" License
82 stars 7 forks source link

Add serialization from/to ByteStrings? #3

Open TerrorJack opened 6 years ago

TerrorJack commented 6 years ago

To use compact to serialize from/to ByteStrings, currently we have to go through temporary files, which makes it a bit inconvenient. Surely in-memory serialization has a lot use cases, so it would be nice if it is supported :)

bgamari commented 6 years ago

Note that the interface for this would likely need to be CPS'd due to the interface of withSerializedCompact. The consumer would be expected for force the entire lazy bytestring.

ezyang commented 6 years ago

Is the terminal state of the compact normal form truly a ByteString, or is the ByteString simply being used as a convenient interchange format for some other API of interest?

I'm a bit nervous about adding a lazy ByteString API because, although it in principle should be possible to do, it is very tricky to do correctly. Unlike most ForeignPtrs, the lazy ByteString has to either be CPS'd (as @bgamari suggests) or setup to keep a Haskell data live (not trivial, but should be possible). But I guess we should bite the bullet and do it.

rightfold commented 5 years ago

In addition to bytestrings it would be useful if the library allows serializing to a region in memory given by a pointer. The primary use case for this I have in mind is shared memory with mmap.