search

dask-contrib / dask-deltatable

A Delta Lake reader for Dask
BSD 3-Clause "New" or "Revised" License
46 stars 15 forks source link

Pass a `Mapping` instead of `ParquetFileWriteOptions` #51

Closed j-bennet closed 1 year ago

j-bennet commented 1 year ago

ParquetFileWriteOptions can't be pickled, so we can't pass it to distributed workers. But we can pass a Mapping to create the options from. There's no problem passing a pyarrow.Schema though.

Closes https://github.com/dask-contrib/dask-deltatable/issues/50

cc @fjetter

codecov-commenter commented 1 year ago

Codecov Report

Merging #51 (f24ee92) into main (b6a675e) will increase coverage by 0.15%. The diff coverage is 50.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main      #51      +/-   ##
==========================================
+ Coverage   74.45%   74.61%   +0.15%     
==========================================
  Files           6        6              
  Lines         321      323       +2     
==========================================
+ Hits          239      241       +2     
  Misses         82       82
Impacted Files Coverage Δ
dask_deltatable/write.py 66.29% <50.00%> (+0.77%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more