WIP: HTTP client builtins

[pchiusano]

DRAFT, in progress, a mess right now.

Working on API for HTTP, HTTPs clients.

unique type Http.Method 
  = GET | HEAD | POST | PUT | DELETE | CONNECT | OPTIONS | TRACE | PATCH

unique type Http.Request g
  = { method : Method
    , headers : [(Text,Text)] 
    , url : Text
    , body : Body g
    , timeout : Duration
    }

-- allow streaming requests / responses - the streams have access to `g`
-- so you can do things like streaming upload of a file
unique type Body g = { mimeType : MimeType, bytes : '{g, Stream Bytes} () }

unique type MimeType = MimeType Text
unique type Http.StatusCode = StatusCode Nat

unique type Http.Status
  = { code : Http.StatusCode, message : Text }

unique type Http.Version 
  = { major : Nat, minor : Nat }

unique type Http.Response g
  = { status : Http.Status
    , version : Http.Version
    , headers : [(Text,Text)]
    , body : Body g
    }

-- this should just do the right thing when url is http vs https
Http.send : Http.Request g ->{g, Http.Client} Http.Response (Body {Http.Client})

-- the Http.Client handler is in charge of config for things like TLS, etc
Http.client.run.handler : Request {Http.Client,g} a ->{IO,g} a
Http.client.run : '{g, Http.Client} a ->{g,IO} a

Questions:

• Not sure how to implement in new runtime. Maybe put these in IOSource?
• What about HTTP2? Could we just have this do the right thing? (Server can switch the client to HTTP2 for a connection)
• Can we have this "just work" if the url is https? That would be ideal.
• Redirects?

/cc @stew

[aryairani]

Awesome!

Some thoughts off the top of my head:

• Probably won't matter any time soon, if ever, but note that custom HTTP verbs are a thing some people want. 😐
• the media type has more structure, if we want to explicitly support that

/cc @rossabaker — I know it will look different in Unison than in Scala, but any thoughts? :)

[pchiusano]

I think @stew has more thoughts on this too, we just discussed offline. But quick summary is will likely just have a builtin TLS implementation (binding to Haskell tls library) and a pure unison http library that uses the tls library plus tcp sockets.

[stew]

Here are some of my thoughts:

• I think having a client that "just works" with "http://" vs a "https://" is something that is expected for an http client these days. I think that it is also expected that you should be able to configure the TLS engine when you are creating a TLS handler, for example, I should have the equivalent of curl's -k, and I should be able to give it a custom list of CA certs.

• I think that we'll need to implement http 1.1 regardless of what we do with http2 and we should prioritize that over worrying about http2 right out of the gate.

• I think implementing a http client in unison is not crazy, but implementing something like TLS would be. I think that if we build a client that can do just GET and POST, then I think all the rest of http 1.1 like DELETE and redirects would be a very easy follo-on

[pchiusano]

Okay, great, so sounds like we're leaning toward: HTTP(S) client library is pure Unison, rather than builtin, but it uses TCP sockets (already in IO) and a to be written builtin TLS implementation (see #1693) for the HTTPS part. We just focus on HTTP 1.1 for now since HTTP2 is like 50x more work and isn't needed right away. I'd guess an HTTP2 client could still be pure Unison though and maybe someone will be interested in taking that on as a library effort.

I think it is good to make things pure Unison (rather than builtins) when they reasonably can be... agree that reimplementing TLS feels crazy.

@stew will confirm that http 1.1 client could just be reasonably pure Unison given a TLS implementation and builtins for TCP sockets and threading which already exist. Main question is: is https literally just http over a tls-based TCP connection? If that's all it is, then "should be nbd."

[aryairani]

If this "cat" is to be believed, then:

HTTPS is just the HTTP protocol but with data encryption using SSL/TLS.

but I'm hesitant to trust anything without a torso.

[rossabaker]

So, http4s has constants for every method in the IANA registry as well as support for custom methods. It's good to be permissive as a client, because someone will eventually need to call some arrogantly verbed API. The specs slowly add new ones, so modeling it as a sum type now means breaking changes later.

But this is Unison, so those changes in due time will be graceful, and what you have now will work almost everywhere.

[pchiusano]

Cool, thanks @rossabaker. In that case I might just do:

unique type Http.Method = Method Text

-- and then just have some named constants to prevent spelling errors
Http.Method.GET = Method "GET"
Http.Method.POST = Method "POST"

[rossabaker]

If you want to be a standards pedant, there are strings that aren't valid HTTP methods, and you might consider Text -> Optional Method smart constructor.

method = token
token = 1*tchar
tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
    "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA

In Haskell, I've been experimenting with pattern synonyms for things that are intuitively sum types today, but need an escape hatch for futureproofing or other weird shit. I don't know whether Unison has anything of the sort, but it's a satisfying idiom that I'd apply to http4s' Method if we had it in Scala: https://github.com/brendanhay/amazonka/pull/597

These are exotic cases, and your simple model above should work well in practice.

[ceedubs]

There is always room for improvement, but I think that we can say mission accomplished on this one! https://share.unison-lang.org/@stew/code/latest/namespaces/public/projects/httpclient/latest