search

uccross / skyhookdm-pythonclient

Python client for SkyhookDM for IRIS-HEP project.
0 stars 0 forks source link

Create a row partitioner #16

Open xweichu opened 4 years ago

xweichu commented 4 years ago

Create a row partitioner which partitions the Arrow based on rows. Make each subpartition close to 10MB.

xweichu commented 4 years ago

Here is an example of the partitioner: https://github.com/uccross/skyhookdm-pythonclient/blob/master/client/skyhookdmclient/skyhook.py#L77 , the partitioner function should be extracted.