storj / team-metainfo

GNU Affero General Public License v3.0
0 stars 0 forks source link

Bring back optimization for listing prefixes #115

Closed mniewrzal closed 2 years ago

mniewrzal commented 2 years ago

Currently, we have some issues with listing large prefixes. In the past (before metabase) we had optimization that was helping with that but we didn't move it to metabase. This optimization is for non-recursive prefix listing.

Optimization was working in a way that if while listing key contained delimiter (/) then the current listing page was dropped and the cursor for the next page was set to prefix<byte of / + 1>.

Old implementation can be found here (see skipPrefix): https://review.dev.storj.io/c/storj/storj/+/4607/3/storage/cockroachkv/ordered_iterator.go

Acceptance Criteria:

storjBuildBot commented 2 years ago

Change https://review.dev.storj.io/c/storj/storj/+/8043 mentions this issue.

mniewrzal commented 2 years ago

All additional optimization can be discussed here https://github.com/storj/team-metainfo/issues/116