Closed y1r closed 11 months ago
Thank you for your detailed review! I renamed normpath
to _canonical_name
, and it now returns url-like string such as scheme://endpoint/bucket/key_name
. Also, zipfile
is too broad name as reserved name, thus I renamed it to pfio-zipfs
to avoid namespace pollution.
This PR adds HTTPCachedFS, a wrapper to hook read operations and cache the content with HTTPConnector. HTTPCachedFS wrapper hooks read operations, then try HTTPCache to retrieve its content. The underlying FS will not be called if the data is in the cache. Other operations like write and list will be transferred to the underlying FS.
In order to introduce such a feature, I added
_canonical_name(path)
to FS interface and implementations._canonical_name(path)
returnspath
with filesystem information to distinguish Local("abc")._canonical_name("somefile") and S3("abc")._canonical_name("somefile"). S3 and HDFS will encode its endpoint to the path to distinguish multiple endpoints (like AWS's S3 and some S3 compatible storage systems).Users can enable HTTPCache transparently by following methods:
http_cache="http://cache-service/"
topfio.v2.from_url
method.from_url
automatically wrapspfio.v2.FS
to hook all read operations. This will work with local, s3, and zipfs.somefile
inhoge.zip
will behttp://{given_url}/hoge.zip/zipfile/somefile
.HTTPCachedFS
directly