pfnet / pfio

IO library to access various filesystems with unified API
https://pfio.readthedocs.io/
MIT License
52 stars 20 forks source link

[Experimental] Custom scheme #300

Closed kuenishi closed 1 year ago

kuenishi commented 1 year ago

When a configuration file $XDG_CONFIG_HOME/pfio.ini is placed, it provides an injection point to customize FS resolution from scheme ( s3 part of s3://some/path/to/file ). It lets us configure endpoint, secret, arbitrary configurations that's available at the kwargs of each FSes. If you have some s3-compat storage system which is not S3, the configuration file would be like:

[foobar]
scheme = s3
endpoint = https://your-s3-endpoint.example.com

And apps will code like this:

  fs = pfio.v2.from_url('foobar://path/to/a')
  with fs.open('filename') as fp:
      fp.read()

Possible discussions:

TODOs

kuenishi commented 1 year ago

Let us load secrets via environmental variables e.g.

[foobar]
scheme = s3
endpoint = https://foobar.example.com/
aws access_key_id = $CUSTOM_ACCESS_KEY
aws_secret_access_key = $CUSTOM_ACCESS_SECRET

Other discussions:

kuenishi commented 1 year ago

Assigned to 2.5.1 instead as it's yet an experimental feature.