Open sorentwo opened 5 months ago
Thanks for your report, I hate to say it but this is roughly the performance difference I’d expect.
The partially matching nature of maps (#{ a := b }
matching all maps with that pair) makes it impossible to bind keys which turns your query into a linear scan, completely tanking performance.
I will look into adding a way to match maps exactly, but that will not make it in before OTP 27.1 at the earliest.
Thanks for the quick reply. The performance difference makes sense when you consider that it's a linear scan, which also explains the difference between select
and select_reverse
.
I've worked around the issue on my side for now, but I suspect this could trip up other applications.
I think this somewhat comes from the ETS interface completely intertwining access by patterns and access by key. In my opinion the proper, long-term solution would be to separate the two. They have very different performance characteristics where a difference between key lookup vs linear scan can be very subtle and unexpected - they should be separate APIs.
Aren't they separated already? lookup
and friends access by key, whereas select
and friends access by pattern.
@jhogberg right, but with no lookup_replace
and such we lose a lot of functionality when wanting to lookup on a key -- being unable to do atomic and isolated actions on elements based on a key lookup.
In OpenTelemetry we, for now, not sure what long term will look like, do binary_to_term
on the map to keep it from using variable matching.
In OpenTelemetry we, for now, not sure what long term will look like, do binary_to_term on the map to keep it from using variable matching.
For the metrics mentioned above I switched to phash2
to hash the map, then stored the map outside of the key. Thankfully it was a simple change.
@ferd had a similar idea. Though his was to still have the map in the key and use an ordered set. Since wouldn't you need to use a bag
if you are using phash2
with the map outside of the key? That wouldn't work for us particularly since we use select_replace
which doesn't support bag
.
But I guess if you are already using bag
there is no question that phash2 with guard makes the most sense :)
Oh I missed this:
I will look into adding a way to match maps exactly, but that will not make it in before OTP 27.1 at the earliest.
Yaya! Solves my request for lookup_replace
:)
Since wouldn't you need to use a bag if you are using phash2 with the map outside of the key? That wouldn't work for us particularly since we use select_replace which doesn't support bag.
The key is itself a tuple of a series name, a hash of the labels, and a timestamp. The table is an ordered_set
, but we have to use lookup
followed by insert
because it's merging values and not replacing them.
Describe the bug There is a severe performance regression for
ets:select/3
when keys contain a map.To Reproduce This benchmark (in Elixir, sorry) reproduces the issue and demonstrates the difference between OTP 26.2.4 and 26.2.5.
Here's the relevant portion of the benchmark for OTP 26.2.5:
Removing the map portion of the key alleviates the issue, and adds a nice speed boost for all operations:
Expected behavior There shouldn't be such an extreme difference in performance.
Affected versions OTP 26.2.5
Additional context Originally reported in this oban issue.