HDFGroup / vol-rest

HDF5 REST VOL Connector
Other
5 stars 8 forks source link

Refactor cURL use into helper functions with local handles #92

Closed mattjala closed 3 months ago

mattjala commented 9 months ago

Most curl requests are now done through new helper functions in rest_vol.c, RV_curl_post, RV_curl_put RV_curl_delete, and RV_curl_get. These functions each create a local curl handle, perform the request, then clean up the handle afterwards.

In order to allow for cases where specific error codes are OK or even expected (checking for the existence of files, flush requests that expect to get HTTP 204 NO CONTENT) the curl helpers return the HTTP code that the request returned, or -1 if the request failed.

The curl helper functions take in the server's base URL through the server_info_t struct, so they only need the endpoint as a separate argument. For this reason, most instances of request_url have been changed to request_endpoint, and no longer include the base URL. This request_endpoint is what is passed to the curl helpers.

I also introduced a new global variable H5_rest_curl_initialized_g to track if curl is initialized. This avoids not-initializing or double-initializing it in the new helper to create a curl handle, RV_curl_setup_handle.

Places that still use the global curl handle:

This is a draft PR since it's based on the branch that stores server connection information on individual file objects (#86).

mattjala commented 8 months ago

After a discussion with Jordan, this should probably be changed to use a pool of handles, one per server. Curl handles are made to be reused, so creating and destroying one for each request is probably wasteful.

mattjala commented 8 months ago

The curl helpers now set fields on the global curl handle, instead of creating and destroying a handle for each operation.

See here: "libcurl keeps a pool of old connections alive. [...] Reusing a connection instead of creating a new one offers significant benefits in speed and required resources. [...] When you are using the easy API, or, more specifically, curl_easy_perform(), libcurl will keep the pool associated with the specific easy handle. Then reusing the same easy handle will ensure it can reuse its connection."