HTTP module

[luisdavim]

Hi, I know there's a fetch module already that provides some support for http requests but I'm looking into an embedded DSL for http api testing. Something like https://docs.racket-lang.org/riposte/index.html?=read-line&=read-line

I've looked into the following:

• https://github.com/andrewpillar/req
• https://github.com/andygello555/sttp
• https://github.com/seriousben/net-script
• https://github.com/damiva/TengoHTTP

I think something like this would be a nice addition to risor, if you agree, I could look into creating a new module for risor that would allow this use case.

[myzie]

Yeah, I think something along these lines is a great idea. Makes sense to use Risor to test APIs 👍

The one aspect I'm less sure of is having an embedded DSL specifically, as opposed to having a library that uses existing Risor syntax. For example, we could do a Risor wrapper of something like github.com/gavv/httexpect.

Which could be something like this in Risor:

expect.put("/fruits/orange").with_json({weight: 100}).expect().status(expect.status_no_content)

In any case, I really like adding this type of capability quite a lot. Could you elaborate on what you think this might look like?

[luisdavim]

Yeah, maybe we can evolve fetch a bit to fit the use case better, I think it's mainly lacking more control/access over the response headers and cookies. When I mentioned the DSL it was thinking that for my project I'd disable most of the functionality and leave only what's needed for this type of testing but the syntax should be in line with what risor already has. Httpexpect looks interesting I'll take a closer look and think about some sample scripts I'd like to be able to create using risor and post here for feedback.

[luisdavim]

I was thinking we can create 2 new types, like a request and a response, similar to what was done for the exec module. The request would have methods to help build, mutate and send the requests and the response would include the response headers, body and status code and some methods to allow accessing them.

The request would be similar to what fetch and NewRequestFromParams do.

A script could look something like this:

req := http.request(https://foo.bar, {
  method: "POST",
  headers: {
    Host: "bar.foo", // this is a special case in go, needs to be added to the request and for TLS it needs to be added to the transport
  }
})

req.addCookie("session", token)

resp := req.send()

if res.status != 200 {
  print(error)
}

print(resp.headers)

if this makes sense to you, I can get started with a module implementation, open a WIP PR and we can flesh out the details from there.

[myzie]

Right, something along these lines sounds good.

fetch returns an object.HttpResponse and maybe that can be added to as needed for response details. For example it does have a header attribute already.

And I agree that having a way to create a wrapped Go HTTP request as a way to gain some more flexibility sounds fine.

You might also want to check if any of these features would be easy additions to fetch as well.

[myzie]

Just in case you hadn't seen the header attribute on the HTTP response object yet:

risor -c "fetch('https://api.ipify.org?format=json').header"

{
  "Connection": [
    "keep-alive"
  ],
  "Content-Length": [
    "23"
  ],
  "Content-Type": [
    "application/json"
  ],
  "Date": [
    "Fri, 29 Dec 2023 01:09:46 GMT"
  ],
  "Server": [
    "nginx/1.25.1"
  ],
  "Vary": [
    "Origin"
  ]
}

[luisdavim]

Yes, I did, I think the main difference would be the ability to build and mutate the request and decoupling that from sending it. We can also add some convenience methods to create cookies. We can reuse most of the code but I'd make it a separate module so it's not a breaking change.

It would be nice if we could move the http utils and the response type to the fetch module, like it's done with exec and pgx. At least the http utils, can't be used from outside because they are in an internal package. I could enhance fetch to add what I miss and then use it on my code to achieve the API testing DSL that I need.

What if I'd refactor fetch into a module named http, adding an alias named fetch to keep backwards compatibility and move the utils and response there? Then we can also have convenience methods like http.get, http.post, etc... this would mostly be reusing the existing code...

[luisdavim]

I started https://github.com/risor-io/risor/pull/146 to explore how that would look like, the first commit is the refactor only with no code added or removed, just moved around.