Component: HttpClient - Githubissues

Dark avails a base Http client, for making calls to endpoints across the internet. This issue organies varios improvements to, and testing of, the HttpClient and its surrounding helper functions.

These tasks are largely unblocked, and accessible to anyone interested.

This needs notable organization, clearly.

[ ] more httpclient logs
- Paul: "I wish that the error message was properly logged in honeycomb when something goes wrong with a http call."
[ ] reassess claim: "Dark can't connect to ipv6 addresses" (see #2715)
- note: this might 'just work' now, since the migration from OCaml to F#.
[ ] backfill a lot of tests
[ ] Httpclient should have middleware (see old #3426)
[ ] Socks proxy hides good errors when the site doesn't exist (see #4005)
[ ] review and extract from: "add new version of existing httpclient without the eggregious problems" (#3839)
[ ] review http request/response headers for how they may impact the .NET client
- https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Request_fields
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
[ ] think about various scenarios of how people use httpclients
- do we support them all? We don't have to make everything easy - just possible.
- (exclude server -> client communication, for now)
[ ] review various response status codes
- would the .NET client do anything odd depending on the status?
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Status
[ ] review SocketsHttpHandler properties
- https://learn.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler?view=net-7.0
[ ] review HttpMessageHandler properties
- https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler?view=net-7.0
[ ] continue to back-fill test cases from HttpClient::[x]_v5 tests
[ ] test using the client against various http protocols
[ ] emulate various scenarios of using the httpclient, in testing
- [ ] fetch a webpage and the pages that come with it
- [ ] CRUD some webpages (html, .js, .css)
- [ ] CRUD some JSON data
- [ ] CRUD some text data
- [ ] CRUD some XML data
- [ ] CRUD some binary data
[ ] sometimes you only care to test the request-handling or response-handling. (how) can we write tests that focus on one or the other?
- [ ] define a fn (in F#) that:
- takes in a tuple of ::request's parameters
- predicts what the request to the .NET client will be
- [ ] same, but for the response (input an HTTP response - what will the Dval output be?)
[ ] consider if fuzztesting would make any sense
- it wouldn't be so difficult to generate (verb, headers, body)
- either based on "scenarios" (e.g. CRUD some JSON data)
- or totally random (with known headers, with random body, etc.)
[ ] should we release other functions along with this?
- [ ] compression
- [ ] redirect
- [ ] headers (in tuple form)
[ ] remove irrelevant bits from the existing .test files
- there are cases when the HTTP response returns a header, needlessly, and the handler code checks for it, outside of the specific test case
[ ] migrate some test cases from individual .test files to the main httpbaseclient.tests file
[ ] test: huge request bodies
[ ] test: huge response bodies
[ ] test: url with credentials
[ ] ensure we have enough telemetry in place
- [ ] request/response body length
[ ] use it for something real to see if it's solid (i.e. emulate static assets endpoint)
[ ] consider publishing Tuples before publishing this fn
how does the client handle different requests?
- by verb
- empty
- all the expected ones (get, post, etc.)
- connect?
- custom
  - reasonable
  - bogus
  - huge
  - emoji
- by headers
- by body
- by combination of these things (aligning content-type header with body, etc.)
how does the client handle different response?
- by status
- by headers
- empty key
- empty value
- empty both
- filled both
- ThisCase
- thisCase
- this-case
- ``-weird-$(!8547=SYMBOLS
- by body
- empty
- medium
- huge
- chunked
- by combination of these things
Some HttpClient ergonomics inspiration (long-term):
- https://github.com/cognitedata/oryx#getting-started
- https://github.com/fsprojects/FsHttp/blob/master/src/FsHttp/Domain.fs
do we need to support (or prepare to support)
- caching
- cookies
- ETAG
ensure compression headers are passed through properly (i.e. from .net response to Dark response)
[ ] (for when WASM is relevant again,) what do we do with httpclient across runtimes? We had to adjust the implementation slightly to work across runtimes, and it'd probably be a mistake to not record which one we used, when recording traces and such
[ ] support HTTP streaming (and streaming those payloads to whatever is using the http client)

quick thoughts on making httpclient calls 'safe' and enabling a user to either

be knowledgeable about what URLs might be called, and/or
explicitly approve some set of URLs that may be called, by passing some 'reference' down the chain of fn calls

I suppose this is most similar to having a DB-accessing function take a DbRef, and passing that DbRef down 'from the top'.

first, note that URLs look roughly like:

so, define some types like

type SchemeRule = 
    | AnyScheme
    | SpecificScheme of string

type AuthorityRule = 
    | AnyAuthority
    | SpecificDomain of string
    | AnySubdomainOf of string

type PathRule = 
    | AnyPath
    | StartsWith of string
    | EqualsTo of string

type QueryRule = 
    | AnyQuery
    | ContainsKey of string
    | KeyValue of string * string

type FragmentRule = 
    | AnyFragment
    | FragmentEquals of string

type UrlRule = {
    Scheme: SchemeRule
    Authority: AuthorityRule
    Path: PathRule
    Query: QueryRule
    Fragment: FragmentRule
}

let exampleUsage = {
    Scheme = SpecificScheme "https"
    Authority = AnySubdomainOf "google.com"
    Path = AnyPath
    Query = AnyQuery
    Fragment = AnyFragment
}

and from there, it feels like there are a few ways to use those types to provide info/security:

update the httpclient to take in a UrlRule value, along with the url to call, and validate before calling
- i.e. let httpClient (rules: UrlRule) (url: string): HttpClientResult = ...
- immediately, this feels a bit weak -
use static analysis or something to provide metadata on fns, indicating what Urls they access (in a similar structure) and avail that at least for information to users at the top
? something else like this, I'm not quite sure. ideas?

darklang / dark

Component: HttpClient #4707