search

tarantool / crud

Easy assess to data stored in vshard cluster
BSD 2-Clause "Simplified" License
40 stars 15 forks source link

Custom cursors support #346

Open bitgorbovsky opened 1 year ago

bitgorbovsky commented 1 year ago

There is a problem with cursor based pagination on requests by secondary indexes. Consider a simple key-value schema:

field type info
key string primary key
value string
tag string secondary index, non unique

The call crud.select('kv', {{'=', 'tag', 'tagValue'}}, {first = 10, after = ...}) makes a fullscan of secondary index from starting point (tagValue) to tuple equaling cursor. Under heavy load it leads to CPU consumption and cluster performance degradation.

To solve this problem, we have to create a special unique index paging with two fields {tag, key} and implement pagination manually, by executing such calls like

local rs = crud.select('kv', {{'==', 'paging', {'tagValue', box.NULL}}}, {first = 10})

...

local last_key = rs.ros[#rs.rows][0]
rs = crud.select('kv', {{'>', 'paging', {'tagValue', last_key}}}, {first = 10})
...

It would be nice to introduce some feature like passing name of paging index and using key from this index as cursors

DifferentialOrange commented 1 year ago

We also may consider supporting new pagiation here: https://github.com/tarantool/tarantool/issues/7639