search

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Apache License 2.0
1.78k stars 285 forks source link

Deprecate compat library since we no longer support pre pyarrow 0.17 #618

Closed selitvin closed 3 years ago

selitvin commented 3 years ago

This PR removes compat.py and cleans up the code. The PR should be a noop with pyarrow >= 0.15

codecov[bot] commented 3 years ago

Codecov Report

Merging #618 (675fed3) into master (a38a283) will increase coverage by 0.22%. The diff coverage is 88.88%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #618      +/-   ##
==========================================
+ Coverage   85.13%   85.35%   +0.22%     
==========================================
  Files          87       86       -1     
  Lines        4998     4951      -47     
  Branches      797      785      -12     
==========================================
- Hits         4255     4226      -29     
+ Misses        594      585       -9     
+ Partials      149      140       -9
Impacted Files Coverage Δ
petastorm/etl/rowgroup_indexing.py 64.51% <0.00%> (-0.57%) :arrow_down:
petastorm/arrow_reader_worker.py 90.72% <80.00%> (+0.66%) :arrow_up:
petastorm/etl/dataset_metadata.py 87.33% <100.00%> (-0.09%) :arrow_down:
petastorm/py_dict_reader_worker.py 95.23% <100.00%> (+0.74%) :arrow_up:
...tastorm/pyarrow_helpers/tests/test_batch_buffer.py 100.00% <100.00%> (ø)
petastorm/unischema.py 93.24% <100.00%> (-0.04%) :arrow_down:
petastorm/utils.py 91.37% <100.00%> (-0.15%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a38a283...675fed3. Read the comment docs.