Find an http client that allows to get performance metrics from each request

[hamax97]

• This comparison from Sylwia vargas suggests to use Faraday:
  • Compared: Faraday, http.rb, rest-client, httparty, Net::HTTP
  • Faraday:
  • has great community support.
  • best memory allocation metrics when downloading.
  • offers middlewares (maybe this will be useful to get custom metrics)
  • I can't find yet the middleware (internal or external) for extracting these metrics.
  • http.rb (now http) seems like a good option:
  • offers good performance
  • uses the NodeJS http parser
  • BUT COMMUNITY DOES NOT SEEM TO BE ACTIVE.
  • DO NOT USE Net::HTTP. It's cumbersome. It has the worst memory allocation metrics.

Benchmark used for memory allocation when downloading: https://gist.github.com/janko/238bbcc78b369ce3438365e5507bc671

[hamax97]

Excon seems like a good option:

• it provides instrumentation that allows you to time requests: link here.
• It uses ActiveSupport::Notifications for this. Maybe I can borrow this idea.
• BUT IT SEEMS TO HAVE A VERY INNACTIVE COMMUNITY.

[hamax97]

After asking ChatGPT:

• For Faraday there's the gem: faraday-request-timer
• For httparty there's the gem: httparty-timer
• Typhoeus has it built in the response object, but the community is innactive. Last commit was on August 2022.

[hamax97]

• How about using these OpenTelemetry convetions?
• How about using the OTel SDK for sending metrics to a backend? It could be a csv file, or it could be InfluxDB, or wathever.
• There's an SKD for Ruby: https://opentelemetry.io/docs/instrumentation/ruby/getting-started/
  • An instrumentation for multiple libraries, including HTTP libraries: Faraday, http, http-client, excon, net_http.
  • https://github.com/open-telemetry/opentelemetry-ruby-contrib/tree/main/instrumentation

[hamax97]

How about using the HTTP client httpx? It seems to have lots of nice features:

• Faraday can use it as a backend.
• handles current requests by default
• will try to reuse sockets and TLS sessions as much as possible and interleave them as much as the underlying protocol allows
• you can pass the TCP socket yourself
• OpenTelemetry doesn't have built-in support for this one
• Seems to be very active.
• It uses GitLab and has a mirror in GitHub: https://gitlab.com/os85/httpx
• Handles streaming requests, multipart requests, and many more.
• All features come as a plug-in, so it's light-weight ...
• MAYBE I CAN IMPLEMENT A CUSTOM PLUGIN FOR MEASURING PERFORMANCE
  • Also, checkout the HTTPX::Session.wrap ... it may be what you need

Wiki: https://honeyryderchuck.gitlab.io/httpx/wiki/home.html

[hamax97]

Perhaps it'll be better to use a low-level library, perhaps a C library with bindings in Ruby.

[hamax97]

How about async-http to use in conjunction with async. Remember the C library with bindings to Ruby too.

[hamax97]

First experience with async. In general it was a poor user experience:

• I tried hitting the web server restful-booker.
• It works fine when using only POST or GET.
• It shows a warning when trying to use PUT or DELETE.
• The warning is the following:

  warn: Async::Pool::Controller: Async::Pool::Controller Gardener [oid=0x1a79e4] [ec=0x1a79f8] [pid=14008] [2023-07-14 11:13:00 -0500]
             | Closing resource while still in use!                                                  
             | {"resource":"#<Async::HTTP::Protocol::HTTP1::Client:0x00007f91c761b790>","usage":1}

• It seems like it's using the HTTP version 1, not version 1.1, which doesn't have support for PUT or POST.
• I tried setting the protocol version manually, but it didn't work. I tried this:

  client = Async::HTTP::Internet.new(protocol: Async::HTTP::Protocol::HTTP11)