Experimental change to improve IO performance when multiple json files are mapped to each dask-dataframe partition.
Context: I was originally exploring a similar optimization to improve remote-storage performance, and found a significant perf bump for local storage as well.
Experimental change to improve IO performance when multiple json files are mapped to each dask-dataframe partition.
Context: I was originally exploring a similar optimization to improve remote-storage performance, and found a significant perf bump for local storage as well.