passing client_options to S3 object store

[cabrrtt]

Hi @roeap I came here after reading this ticket on the arrow-datafusion-python repo. I have a question on the S3 ObjectStore and whether it is possible to pass ClientOptions struct to Rust through the python bindings?

[roeap]

Hey @cabrrtt, some of the storage otions end up as client options, but most are currently not passed down.

That said, it would be quite easy to add that. Would you expect to pass this as part of the current storage options, or in a separate map? ALso would you care about configuring retry options, which are internally handled separately?

[cabrrtt]

Good questions! It's probably a question of what is the most pythonic approach. In my case I wish to pass a map of arbitrary headers down to ClientOptions, and for that a dict is the most natural route I believe.

Regarding retry, I think this is also necessary. Does it make sense to follow the same signature as the Rust api, as a nested dict?

retry_config = {
  "max_retries": int,
  "retry_timeout": int ms,
  "backoff": {
    "init_backoff": int ms,
    "max_backoff": int ms,
    "base": float
  }
}

[cabrrtt]

I think logically this would mean if it were to follow the Rust api, for headers:

default_headers: {
   "x-amz-meta-...": string,
   ...
}

though ergonomically this feels better named just headers(?)