Open armanbilge opened 1 year ago
@armanbilge Interesting! :heart_eyes: Thanks for providing all the details :relaxed: I'm currently overwhelmed by the sheer amount of work I have to do in my job, but I'll take care of this one as soon as I can.
I read the documents and implemented some stuff required for supporting curl multi-socket which is the base of this. I've got a couple of questions:
In this section:
Then, start a fiber looping on read-readiness and another looping on write-readiness. Whenever the socket is ready, the curl_multi_socket_action should be invoked.
I can't understand how this looping needs to be implemented. Libcurl
requires to be notified when there are some changes on the FD state, but this API seems to be for detecting the changes themselves by checking the FD? Am I correct or I'm missing something?
Whenever the socket is ready
Or to put it in other words, how to detect if the socket is ready?
FileDescriptorPoller
? Are there constructors somewhere? Currently I just ignored this by getting it in the constructor arguments, but I wanted to know the big picture and how it's going to be in the final builder?I can't understand how this looping needs to be implemented.
pollReadRec
and pollWriteRec
already implement that loop :) those methods will call the provided function every time the socket becomes ready on reading or writing.
So basically you will want to start a fiber like this:
pollHandle.pollReadRec(()) { _ =>
IO {
curl_multi_socket_action(multi_handle, sockfd, CURL_CSELECT_IN, null)
Left(()) // loop forever
}
}
and a similar one for the write loop. This is a fiber that will invoke curl_multi_socket_action
with the appropriate arguments every time the socket indicates readiness for read/write.
2. How to get a
FileDescriptorPoller
?
This is the most unstable part of the API, that Daniel and I are still debating over. Here's how to do it currently, but it might change.
Got it, Thank you!
Thanks for your work, don't hesitate if you have other questions!! 🚀
The tests are passing for the new runtime, however it needs polishing and some cleaning. And also one side effect of using this new runtime is that it doesn't support windows. @armanbilge What do you think about this? I think we can keep both the old runtime and this new implementation using FD poller, in order not to drop the win support.
The tests are passing for the new runtime
Wow, amazing!! Thanks so much for trying this out.
I think we can keep both the old runtime and this new implementation using FD poller, in order not to drop the win support.
Yes, this is exactly what I'm thinking as well, if it's not too much maintenance burden we should keep both implementations.
Edit:
both the old runtime
Btw, the old runtime can be refactored as a PollingSystem
.
Btw, the old runtime can be refactored as a PollingSystem.
Yes, I'm gonna do it after cleaning this one. Currently I've just put a nowarn annotation on it so that I can focus on the task at hand.
Currently to use http4s-curl we use
CurlApp
which replaces theIORuntime
with aCurlRuntime
. This means that it is not possible to integrate http4s-curl with FS2-based applications (Ember, Skunk, Lepus) and other native libraries (e.g. SNUnit).In Cats Effect 3.6 the default runtime offers file descriptor I/O polling that works with all of these libraries. The idea would be to use
IOApp
as usual and then to construct aResource[IO, Client[IO]]
with aCurlClientBuilder
(similar to Ember).Here's a rough sketch for how to implement a
Client[IO]
using file descriptor polling:Create a
MapRef[IO, FileDescriptor, (Fiber, Fiber)]
to keep track of a dedicated fiber for polling each relevant file descriptor for each type of interest (read interest vs write interest).Configure the
CURLMOPT_SOCKETFUNCTION
to get notified about when cURL needs to start/stop monitoring sockets for read/writes. Use these callbacks to start/cancel fibers in theMapRef
. https://curl.se/libcurl/c/CURLMOPT_SOCKETFUNCTION.htmlTo add a new socket you should register the file descriptor.
Then, start a fiber looping on
read
-readiness and another looping onwrite
-readiness. Whenever the socket is ready, thecurl_multi_socket_action
should be invoked. https://curl.se/libcurl/c/curl_multi_socket_action.htmlA callback for
CURLMOPT_TIMERFUNCTION
should also be registered. It should start a fiber thatsleep
s the requested time and then callscurl_multi_socket_action
. https://curl.se/libcurl/c/CURLMOPT_TIMERFUNCTION.html