search

vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.25k stars 590 forks source link

Extract pre filter test #2194

Open Ben-Epstein opened 2 years ago

Ben-Epstein commented 2 years ago

@maartenbreddels

i would expect this to fail on extract because there is one pre_filtered task (0) and one not pre_filtered task (1) but this seems to work fine for some reason.

maartenbreddels commented 2 years ago

ok, i pushed a failing test, the situation where it failed is.. a filtered dataframe, where 1 of the tasks does pre-filtering (like sum), and the other does not (nop uses mapreduce).