stdlib

[andreypopp]

We want to ship a set of libraries with rrun, I'm now thinking more about libraries which implement portable APIs to operating system facilities rather than data structures (though we should explore this topic too).

How to expose stdlib to user code

I think we should use [%import "..."] syntax which is used already for user code. Currently such syntax only allows relative paths and https URLs, that means we can use plain (non-path) strings for stdlib:

module Path = [%import "path"]
module Cmd = [%import "cmd"]

CamelCase or snake_case

Definitely CamelCase.

Concurrency primitive

All I/O should be concurrent, like in Node.

Decide what we should use:

• lwt
• async
• repromise

I'd like to explore repromise but we might need to sync up with aantron on that - is it ready for experimentation at least?

Proposed modules to include in stdlib

• Promise - concurrency primitive exposed (repromise or whatever we've chosen).

  • Promise.return

  • Promise.join

  • Promise.sleep

  • Promise.List.map

  • Promise.List.iter

• Path - working with filesystem paths, I like how fpath is designed, we might just take it and use.

• Cmd - compose command invocations and execute them, I like how bos library has Bos.Cmd and Bos.OS.Cmd, few notes regarding them though:

  • While bos has two separate modules for constructing commands and running commands, I think we should go with a single Cmd module.

  • Bos.Cmd allows to construct and empty command invocation which is then fails if you try executing it. I think we should disallow that and I already have such Cmd implementation in esy-lib/Cmd, we could take it.

  • I like how bos allows you to compose command invocations with combinators (modified bos API in the example below):

  let catString = (s) =>
    Cmd.(inString(s) |> run v("cat") |> outString);

• Fs - access to file system:

  • Fs.readFile
  • Fs.writeFile
  • Fs.listDir
  • Fs.createDir
  • ...

• Json

• Http - HTTP protocol

  • Http.Fetch - HTTP client

  • Http.Serve - HTTP server

• CommandLine - parsing command line arguments/options. Take a look at cmdliner maybe? Though it's not as trivial to use, might take something more simple.

[kennetpostigo]

@andreypopp Gotcha, sounds like a good start, i'll sync with Anton on repromise and see where it's at on the native side, and see if we can leverage luv. I'll also go ahead and take a look and get acquainted with some of the libs that you mentioned. I'll ping back here once I've done that and am ready to start implementing some of this. Hopefully tomorrow and sunday! 😁

[aantron]

Repromise native is ready. See https://github.com/aantron/repromise_lwt for a POC for a complete native async library based on Repromise. It basically uses Repromise as the front-end API, and shows how to wrap Lwt as the back end.

(I started with this as a POC because Lwt already exists, but the long-term plan is to use https://github.com/aantron/luv as the low-level back end, actually for both Repromise and Lwt). Either way, Lwt and Repromise native get unified.

There may be some bugs in Repromise native, as normal for software, because nobody is using it yet. But it is feature-complete and well-exercised by its own test suite :)

[kennetpostigo]

@aantron that looks really awesome! So, I think what you're saying is to use repromise_lwt for now as the promise library in rrun and then when luv is ready switch to a repromise backend that uses luv?

Since rrun isn't used by anyone we have the freedom to break things without upsetting folks, so I think it's possible to iterate and beta test repromise with luv later on or now (if you think it's possible) as we start building out this stdlib and build out tests for rrun.

[aantron]

That could be a good path, but I'm not sure what's best for rrun, as I'm not familiar enough with it yet.

BTW terminology-wise, Repromise is the promise library. repromise_lwt uses the back end of Lwt as the I/O library (the part that provides Node-like "ticks" and the select/epoll/kevent call, and "drives" the promise library).

Also BTW, the main reason to use Repromise is if you want a JS-promise-like API (but cleaned up to avoid rejections, unsoundness, and other problems). It's also possible to consider something totally different for async, but I honestly don't have any better suggestion :)

[andreypopp]

Thanks @aantron

What do you think regarding API design "lwt vs repromise" if we are not concerned (yet maybe) about JS target?

[aantron]

The Lwt API could use a lot of fixes, as it has developed organically over a very long time.

The Repromise API is a bit of a fresh start and is pretty clean. It has few users, so we can evolve it much faster. It might be simplistic in some ways, like in its async exception handling, but this can also be a virtue as it is very easy to understand what happens :)

If you plan to wrap either one in a Promise module anyway, it might not matter that much which one you use underneath. Lwt does have a couple semantic quirks (like callback-calling order and some weird exception handling cases) that are difficult to fix, while Repromise fixes them and sticks close to the JS semantics.

These semantic differences will be apparent even if you wrap Lwt in a Promise module. They will show up as very rare, but weird, corner cases, pretty much as they already do in Lwt.

To be more precise, this refers to the possibility of unbounded stack growth, a higher likelihood of I/O starvation (both from tending to call callbacks immediately), and exceptions sometimes escaping from bind instead of becoming promise rejections (Repromise doesn't have rejections at all, by comparison).

[kennetpostigo]

@andreypopp I think we should use repromise at the moment using the lwt backend, I think there isn't much of an issue since the projects will become unified anyway.

There is the argument that many libraries already use lwt for async but we could provide a compat helper if needed. If we do a good enough job with the stdlib modules I don't think many people will have this issue.

Probably don't even need to wrap repromise in a Promise module, we can just vendor it and have it imported like:

module Promise = [%import "repromise"];

[andreypopp]

@kennetpostigo yes, repromise looks good, let's use it.