Open xweichu opened 4 years ago
Create a row partitioner which partitions the Arrow based on rows. Make each subpartition close to 10MB.
Here is an example of the partitioner: https://github.com/uccross/skyhookdm-pythonclient/blob/master/client/skyhookdmclient/skyhook.py#L77 , the partitioner function should be extracted.
Create a row partitioner which partitions the Arrow based on rows. Make each subpartition close to 10MB.