LWT support

[thelema]

We should provide monadic interfaces to our data structures so LWT can interface with them better

[thelema]

Should we depend on LWT? How can we best interoperate with LWT?

[mdekstrand]

LWT is awesome, but I don't know that depending on it is necessarily a good or useful thing. Maybe provide a batteries.lwt package that pulls in additional LWT integration if LWT is available?

[mdekstrand]

More thoughts on integrating Lwt... would it be practical to functorize a bunch of things over a cooperative monad functor, so that Lwt can be optionally used but allow direct code asl well?

[gasche]

We have a minimal Monad interface in BatMonad.S. Providing monadic versions of most data structure traversal (in the spirit of Haskell's mapM / foldM etc.) should be easy.

Would this provide a satisfying Lwt integration ? What else would you want ?

[mdekstrand]

That would be sufficient so far as I am concerned.

3.12's first-class modules could be useful for introducing a lightweight interface, so you can do foldM Lwt ... without having to instantiate a functor. I think that it would be good for any such interface to exist in parallel with (or perhaps build on) the functor interface, so that functor instantiation can be used if desired and it makes it easier to keep 3.11 compatibility.

[c-cube]

I'm very curious about how many people currently use BatIO. If they are not too many, it would be very interesting to make BatIO generic over an effect monad (such as Lwt or Async, or a trivial implementation that is just the usual sequential behavior), which would also make it more useful compared to the stdlib. Any thoughts on that?

[UnixJunkie]

Maybe I don't understand this discussion clearly, but I hope batteries' users will not be forced to have anything to do with Lwt or Async unless they explicitly require it.

[c-cube]

@UnixJunkie I don't think anyone here wanted that :)

[thelema]

I'd be fine with this kind of adaptation of BatIO. I'm not even too worried about performance in this case.

On Thu, Jan 9, 2014 at 11:25 AM, Simon Cruanes notifications@github.comwrote:

@UnixJunkie https://github.com/UnixJunkie I don't think anyone here wanted that :)

— Reply to this email directly or view it on GitHubhttps://github.com/ocaml-batteries-team/batteries-included/issues/10#issuecomment-31949034 .

[c-cube]

So, BatInnerIO and BatIO would provide a module type, parametrized over an IO monad (Lwt, Async, or the trivial "monad" where type 'a t = 'a, I guess, for the usual blocking behavior -- unless we want to implement BatFuture but that's probably not a good idea). Operations on input and output would return the corresponding 'a IO.t type, and all code that is related to BatIO would use the monadic interface.

It sounds like it's going to slow down a bit for sequential, blocking code (the small overhead of using a trivial monad), but on the other hand it would make batteries' IO infrastructure (printers, readers, etc.) usable in a cooperative threading context.

[c-cube]

I've started trying to make Bat{Inner,}IO abstract on a monadic type 'a m (at least in .mli files). One of the issues that has become apparent is that some equivalent of Lwt_stream is needed for some functions (e.g. BatIO.BigEndian.doubles_of, that currently reads a float BatEnum.t.) In a cooperative world, this cannot work with Enum, but only with an equivalent of BatSeq that would look like the following:

module MonadSeq : sig
  type 'a m  (* the monad *)

  type 'a t = unit -> 'a node m
  and 'a node = Nil | Cons of 'a * 'a t

  val nil : 'a -> 'a t
  val cons : 'a -> 'a t -> 'a t

  val fold : ('b -> 'a -> 'b m) -> 'b -> 'a t -> 'b m 
  ...
end

So the possibilities to deal with this are:

1. ignore those stream functions (only provide them for the sequential case)
2. require that the 'IO_Monadthat is abstracted upon provides such a stream type (seeLwt_stream`)
3. replace BatSeq with a monadic abstraction of it (again with a specialization for the sequential case). This may interact with #501. Since the purpose of BatSeq is to iterate on too big enumerations to fit in memory, actually abstracting over IO doesn't sound so stupid to me.

[gasche]

Well (1) is obviously a reasonable choice for the short-term. Long-term, (2) doesn't make sense (such monadic streams should be parametric over the monads, not provided separately with each monad), so it means rather something like (3).

I'm not sure if we could make Enum monadic. Maybe that doesn't make sense, but my intuition would rather be that it is very hard, because it corresponds to understanding the invariants -- I've long had the intuition that we could formalize clone it terms of observable side-effects.

[c-cube]

@gasche uou're right about Enum. I said BatSeq but for IO it's actually more natural to have consumable enumerations (since in general the IO state is also consumable, e.g. sockets or user inputs). The problem, as you pointed out is that Enum assumes clonability. Maybe we should focus on Enum, with clonability implemented as a shared, hidden queue + 1 pointer per clone (elements past all pointers are discarded, when an instance reads a new value it pushes it in the queue and updates its pointer). That's quite complicated, but may also be a useful mecanism (e.g. IO.copy in combination with clonable enums would provide the same functionality as Unix's tee program).

Big drawback: Enum is already very complicated, and making a monadic version of it/making it monadic would impact both complexity of the code and its performance.