search

ieaves / tenzing

MIT License
0 stars 0 forks source link

partitioner implementation, many fixes #48

Closed ieaves closed 5 years ago

ieaves commented 5 years ago

@sbrugman I didn't get to the tests but I'm 90% confident all of these are correct at this point and issues that crop up are issues with the tests not with the implementation.

ieaves commented 5 years ago

There is still a partitioner issue with irreconcilable types.

If I give you pd.Series([True, np.inf, False])

Pandas can only represent that as an object, when we partition out np.inf, it is still an object, not boolean. Pandas has a detect_dtype type function (I don't recall the name). We could use it after every partitioning to force the data into a more representative base type (it's kind of slow though).