basho / riak_pipe

Riak Pipelines
Apache License 2.0
162 stars 60 forks source link

Bugfix/listkeys filters #105

Closed erikleitch closed 8 years ago

erikleitch commented 8 years ago

Context

stream_list_keys has always returned inconsistent results when used with TS data. I've tracked this down to the coverage filter functions that are applied during the fold over keys on a vnode. These filters check whether a key should be included, by checking if its hash falls within a specified hash range for a vnode. These filters are applied to the storage keys in leveldb. For TS, however, these filters need to be applied not to the storage key (which is the local key), but instead to the partition key, which can be derived from the local key by an appropriate transformation.

This PR addresses this inconsistency by changing the filter functions to use a key conversion function on the storage key prior to hashing. It is one of three PRs: riak_kv (https://github.com/basho/riak_kv/pull/1404), riak_core (https://github.com/basho/riak_core/pull/834) and riak_pipe (https://github.com/basho/riak_pipe/pull/105). For reference, I include my diagram of the listkeys code path here:

RiakTS Listkeys Path

Changes in this repo

hmmr commented 8 years ago

+1 9c1f887

hmmr commented 8 years ago

@borshop merge