dinakajoy commented 2 years ago

Browser support for Irmin-server

Problem

There is need to support communication between the browser and irmin-server. Currently irmin-server uses flows to communicate between the server and the client. Flows are byte-streams like a TCP connection and these are established using Conduit. In the browser, there is currently no mechanism we know of for having a bi-directional, stream-oriented communication channel

The closest thing is a websocket which solves bidirectional communication, but is inherently message-oriented and not stream-oriented. Websockets sound like a good solution but it could be implemented in a few different ways.

Solutions

A conduit-like flow using websockets

One solution is to impose a stream-like structure layered on top of the message-oriented websocket connection. This could look like the following for example:

let websocket_handler server client =
    let flow = Obj.magic () in (* BAD! *)
    let rec fill_ic channel client =
      let* frame = Websocket_lwt_unix.Connected_client.recv client in
      Lwt_io.write channel frame.content >>= fun () ->
      fill_ic channel client
    in
    let rec send_oc channel client =
      Lwt_io.read channel >>= fun content ->
      Websocket_lwt_unix.Connected_client.send client (Websocket.Frame.create ~content ()) >>= fun () ->
      send_oc channel client
    in
    let input_ic, input_oc = Lwt_io.pipe () in
    let output_ic, output_oc = Lwt_io.pipe () in
    Lwt.async (fun () -> fill_ic input_oc client);
    Lwt.async (fun () -> send_oc output_ic client);
    callback server flow input_ic output_oc

Although it is unclear if this will be sufficient but the basic idea is to have two pipes for reading and writing. When we receive a websocket message we add it to the reading channel and when we send a message we add it to the writing channel.

Reimplement protocol but with message-oriented

Going this route means that most features has to be reimplemented to support a message-oriented communication

The IO module - The IO module currently uses Conduit_lwt_unix, which communicates using flow.
The protocol - The protocol is currently structured around conn_intf which calls the IO module which uses flow.

Questions

What would the API look like if we support a message-oriented and stream-oriented communication.

patricoferris commented 2 years ago

There's now a (quite untested) working implementation of the "A conduit-like flow using websockets" approach that @dinakajoy and I have been hacking on.

The biggest difference between the approach in this issue and that one, is that we have to be a bit more clever about how we read from the pipes. We have to parse the constructed packets into strings to send as a websocket message. The code diff is here: https://github.com/mirage/irmin-server/compare/master...patricoferris:ws?expand=1.

Note this code requires both:

An upstream version of graphql (thanks @zshipko !): https://github.com/andreas/ocaml-graphql-server/pull/206
And the accompanying, compatible irmin libraries: https://github.com/mirage/irmin/pull/1843

zshipko commented 2 years ago

Awesome!!! I will try running this and see how it goes