Open michae2 opened 2 years ago
Another problem case of this is in https://github.com/cockroachlabs/support/issues/2197
No rows are qualified for the predicate, so the lookup join limit hint ends up being too small.
Idea: If there is a filter on a lookup join, don't adjust the rowcount down so much in computeLookupJoinCost
because that filter would be cheaper to apply as a constrained scan using another index.
I've come across a case in the wild where the limit hint is 1000x off from the actual number of rows needed to scan. The problem appears worst when we apply the generic 1/3rd selectivity to filters that we don't have more accurate information about. If filter is actually more selective than 1/3, then we'll come up with a wildly low limit hint and end up with a poor plan.
Maybe the generic filter selectivity should be used for row counts, but not for limit hints. In other words, when we don't have a good idea about the selectivity of the filter, why not assume that all rows will be produced in order to satisfy the limit?
Another case of this is https://github.com/cockroachlabs/support/issues/2447 Except in that case we weren't using default filter selectivity. If we could mark each selectivity estimate with a confidence level, then "no confidence" estimates could use a selectivity of 1 for limit hint costing and for "low confidence", maybe use something higher than the original selectivity estimate, but not 1.
I have a theory that we are simply overly optimistic when costing with a limit hint. For example, in a scan, we take the limit hint as the row count. This assumes that there is an even distribution of rows across the index that will satisfy the limit in a parent expression. This seems like a best-case scenario in most cases, and wildly optimistic.
Should we instead consider the row count to be the average of the limit hint and the worst case of scanning the entire table? Alternatively, we could use the average of the limit that produces the limit hint, the limit hint, and the number of rows in the table.
This is similar to the idea of having a confidence interval of row counts or costs, but would not require any additional machinery.
Disregard my last comment. Any change to make the limit hint -> row count estimation more or less conservative will benefit certain edge cases but be a detriment to others. I don't see a good solution here besides trying to refine costing to more accurately reflect real-world execution time (not sure if there is even room for improvement here) or speeding up the execution time of the bad plans so that they are less bad when chosen.
We've seen a handful of cases where the limit hint is wildly wrong because the selectivity of a filter cannot be accurately determined and defaults to 1/3, a wild over-estimation. I've outlined some improvements we can make to our selectivity estimates in #119966.
As of 22.1, the optimizer's limit hint calculation for selects assumes that the selected rows are evenly distributed amongst the input rows: https://github.com/cockroachdb/cockroach/blob/d20fd55bab4c3ff08336831d42c457a36a1acdec/pkg/sql/opt/xform/physical_props.go#L164-L168
This is the best we can do with the information we have, but it can be wildly wrong if the selected rows are not evenly distributed. If the selected rows come at the end of the input then we could read many more rows than the limit hint suggests. We've seen a few cases in the wild of scans reading 15x - 20x more rows than the limit hint suggests.
Here's a demonstration:
Unfortunately we don't really have enough information to cost this correctly. If the selected rows are at the beginning of the input then the plan is indeed good. Here's the same demonstration but with selected rows at the beginning:
And, in, fact, if the selected rows are distributed entirely at the beginning, the limit hint can be wrong in the other direction (too pessimistic). This one is a little harder to demonstrate, we need a bigger table and a smaller proportion of selected rows:
So it's not clear that a one-size-fits-all approach of pessimizing limit hints would be the right thing to do.
Maybe multicolumn histograms could help some of these cases.
Jira issue: CRDB-17671