Closed snoyberg closed 8 years ago
Making this (obviously terribly inefficient) replacement seems to fix the problem:
toBufIOWith :: Buffer -> BufSize -> (Buffer -> Int -> IO ()) -> Builder -> IO ()
toBufIOWith buf !size io builder =
let (PS fptr off siz) = L.toStrict $ toLazyByteString builder
in withForeignPtr fptr $ \ptr -> io (ptr `plusPtr` off) siz
The proper way to do this would be, when nlen
is greater than the buffer size, allocate a new buffer of the right size and pass it in.
However, I think we're still left open for an interleaved output race condition. Specifically, if the write
system call doesn't write all bytes, another thread could take a stab at sending some bytes before our buffer is emptied. The three solutions I can think of for this are:
MVar
-based blocking whenever writing to the FD, which will likely kill performanceatomicModifyIORef'
to let multiple threads all put data into a single ref, but I haven't figured out the right mechanisms to make sure that we don't get interleaved I/O. Perhaps some kind of tricks like we use in auto-update could be employed.I was curious about how solving this issue outside fast-logger with STM and MVar would work, here is some benchmarks: https://rawgit.com/lehins/practice/master/haskell/fast-logger/results.html Code is here: https://github.com/lehins/practice/blob/master/haskell/fast-logger/how-fast-logger.hs It seems that for entries larger than the buffer size and with many threads STM approach does outperform even native (buggy) log writing. Thought it might be useful here.
I would close this issue thanks to #103. Please reopen if necessary.
The following reproduces the problem (also available as a Gist):
The issue appears to be that each thread's individual buffer fills up, and then the chunks are sent interleaved to the
Handle
.