Use pipes to stream logs

[stolyaroleh]

..instead of collecting them in memory and then returning everything at once.

I did this by changing the MonadStore to:

type MonadStore a = ExceptT Error (Producer Logger (ReaderT Socket IO)) a

It's a Producer of log messages that can fail with an Error:

data Error =
    LogError Int LBS.ByteString
  | ParseError String
  | ConnError String

runStore can now accept a Consumer Logger IO (Either Error a) parameter to consume logs incrementally. I also added a runStore_ that simply discards all logs.

Similarly, I changed the runOpArgs and runOp: now they accept a Get a parser that is used to get the result. runOp_ and runOpArgs_ assume that op does not return anything interesting.

I also implemented the buildPaths op to confirm that this works.

[stolyaroleh]

I am not sure about the choice of streaming library. I know about pipes and conduit, and I picked the former because it seems to have fewer dependencies.

[sorki]

This is nice! Personally I like pipes. Will try to integrate these changes when prerequisites are in place.

[stolyaroleh]

Thanks! I have a fork of hnix-store where I have implemented all ops for a personal project, and this was my attempt to merge my efforts upstream. I just noticed that I have a fix for a bug I introduced with these changes, will update the PR soon. Let me know if you need help with integration.

[bqv]

Soon? :D

[sorki]

It's not quite clear if pipes or dlist from anoher PR or streaming lib.. which is why I've started exploring effect systems so you could possibly use any of that by just replacing interpreter.

[thomasjm]

For a simple solution that doesn't require adding any extra dependencies, why not just use an IO callback? I.e., plumb a function with signature Logger -> IO () through the MonadStore type, perhaps as part of the ReaderT context.

Then the user can specify this callback and do whatever they like with it, such as feed it into a streaming library. (I assume this is possible with pipes?)

[sorki]

Soon? :D

Soon^tm! :upside_down_face:

[sorki]

For a simple solution that doesn't require adding any extra dependencies, why not just use an IO callback? I.e., plumb a function with signature Logger -> IO () through the MonadStore type, perhaps as part of the ReaderT context.

Then the user can specify this callback and do whatever they like with it, such as feed it into a streaming library. (I assume this is possible with pipes?)

This was obvious to me but there were many more things that need handling first, but it is done now and you can implement your own runner that uses a custom appendLog. The typeclass is not quite ideal yet I think because there are too many methods to my liking.

If anyone knows a good solution how to make it easier to implement please step forward! (or even better submit a PR :smiley_cat: )

I will try to implement another runner that prints log to stdout soon^tm just to see how bad it is because the default one is only useful for testing.