search

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Apache License 2.0
1.78k stars 285 forks source link

Use path with bucket name if it's an s3 path and a custom filesystem. #598

Closed acmore closed 4 years ago

acmore commented 4 years ago

When using s3 compatible object storage, we would like to pass in a S3FileSystem overriding the endpoint. The path should include the bucket name when it's s3 object storage.

CLAassistant commented 4 years ago

CLA assistant check
All committers have signed the CLA.

acmore commented 4 years ago

Can you please add a line describing the fix to the release notes (docs/release-notes.rst)?

Sure. Added.

codecov[bot] commented 4 years ago

Codecov Report

Merging #598 into master will decrease coverage by 0.50%. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #598      +/-   ##
==========================================
- Coverage   85.68%   85.18%   -0.51%     
==========================================
  Files          87       87              
  Lines        4994     4994              
  Branches      795      795              
==========================================
- Hits         4279     4254      -25     
- Misses        578      592      +14     
- Partials      137      148      +11
Impacted Files Coverage Δ
petastorm/etl/dataset_metadata.py 87.41% <100.00%> (ø)
petastorm/compat.py 60.97% <0.00%> (-39.03%) :arrow_down:
petastorm/tf_utils.py 85.10% <0.00%> (-3.55%) :arrow_down:
petastorm/arrow_reader_worker.py 90.06% <0.00%> (-1.99%) :arrow_down:
petastorm/py_dict_reader_worker.py 94.48% <0.00%> (-0.79%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ffaf6b6...1d58153. Read the comment docs.