Open MariusMerkleQC opened 1 month ago
Yeah, this seems like a big footgun to me, I've been bitten by it before, and I made sure all my inputs to join_asof
are defensively sorted, but it comes at a largeish CPU and memory cost. I'd prefer if it could just fail/throw (perhaps with a verify_sorted
flag if people really don't want to pay the cost to check for sortedness).
Yes, we should check this.
I was looking at this yesterday. We check sortedness in a asof_join w/o by groups, but explicitly don't check it when there are groups. I'm assuming this is because forcing the entire on column to be sorted is too restrictive... since the actual required condition is that the on column is sorted within each by group.
I could try implementing an up front check for that condition, unless someone else is already on it, or you think it's too complicated for a new contributor.
I was looking at this yesterday. We check sortedness in a asof_join w/o by groups, but explicitly don't check it when there are groups. I'm assuming this is because forcing the entire on column to be sorted is too restrictive... since the actual required condition is that the on column is sorted within each by group.
I could try implementing an up front check for that condition, unless someone else is already on it, or you think it's too complicated for a new contributor.
Is it possible to check for sortedness within a group or is that too expensive?
I think we might be able to check for this condition in the process of doing the join itself, rather than trying to verify it up front.
My thought was to check this in the AsOfJoinStrategy .next() function, make that function and its caller return PolarsResults, and propagate the error back up to where we can call polars_bail. Let me know if that sounds like going down the wrong path.
Checks
Reproducible example
Log output
No response
Issue description
Even though
df_right
is not sorted, the code runs without interruption and will have an incorrect joined result.Expected behavior
It should raise the following error instead. It does so if you remove
by="key"
above.Installed versions