dask-contrib / dask-deltatable

A Delta Lake reader for Dask
BSD 3-Clause "New" or "Revised" License
46 stars 15 forks source link

Fix the PickleError with RawDeltaTable #57

Closed j-bennet closed 1 year ago

j-bennet commented 1 year ago

Fixes an error when reading a Dask dataframe with dask-deltalake and distributed.Client.

<dask.highlevelgraph.HighLevelGraph object at 0x13fac3fd0>
 0. read-delta-table-f039f9f4fa4ebf9b2dd57eb38dcfa70e
>.
Traceback (most recent call last):
  File "/Users/jbennet/mambaforge/envs/dask-deltatable/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
TypeError: cannot pickle 'builtins.RawDeltaTable' object

This PR looks like a lot of changes, but that's only because I moved the data fixtures to conftest.py so they can be shared between modules. The real changes are all in core.py.

cc @fjetter

Closes https://github.com/dask-contrib/dask-deltatable/issues/56.

codecov-commenter commented 1 year ago

Codecov Report

Merging #57 (a8f1e56) into main (dbeb8cc) will increase coverage by 0.15%. The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main      #57      +/-   ##
==========================================
+ Coverage   74.61%   74.76%   +0.15%     
==========================================
  Files           6        6              
  Lines         323      325       +2     
==========================================
+ Hits          241      243       +2     
  Misses         82       82              
Impacted Files Coverage Δ
dask_deltatable/core.py 91.76% <100.00%> (+0.19%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

j-bennet commented 1 year ago

Thanks @fjetter @jrbourbeau , I'll rework this. Hopefully that will help with performance, too.

j-bennet commented 1 year ago

@fjetter It's green.