static-frame / arraykit

Python C Extensions for StaticFrame
Other
8 stars 2 forks source link

Implement `isna_array_object()` #16

Closed flexatone closed 1 year ago

chaburkland commented 3 years ago

After examining the reference Python code for this, I'm not convinced there will be enough potential speed gains to warrant all the complexity of the added C code.

The code is just a list of if-elif-else statements that each delegate to a slightly different numpy call. I can't forsee how to out-perform numpy on those calls, and I don't think C if-elif-else statements are materially faster than Python if-elif-else statements.

I could be wrong, but my hunch is this is not a good candidate for converting to C.

flexatone commented 3 years ago

The only opportunity here, as I saw it, had to do with handling object arrays. The approach below requires two iterations; it should be possible in one if we write our own object array processor. So what we might benefit from is a focused "isna_arrry_object()" utility function that produces a Boolean array in one iteration.

    if include_none:
        return np.not_equal(array, array) | np.equal(array, None)