search

RediSearch / RediSearch

A query and indexing engine for Redis, providing secondary indexing, full-text search, vector similarity search and aggregations.
https://redis.io/docs/stack/search/
Other
5.33k stars 513 forks source link

[BUG] Redis Search silently fails to Sort when the index schema is too large #4494

Open claridgicus opened 4 months ago

claridgicus commented 4 months ago

Describe the bug I am running a redissearch instance to curate product information for many merchants Merchants divvy up their products into user-friendly "categories" which typically contain their own unique sort order. I have loaded in the merchants' products using prefix keys to partition them from each other I create a "Category" index, which contains useful information for a customer to interact with, price, tags, title etc and also includes a reference to a nested JSON object "sortOrder"

My index looks a bit like this

'$.title': {
    type: SchemaFieldTypes.TEXT,
    SORTABLE: true,
    AS: 'title',
},

'$.tags': {
    type: SchemaFieldTypes.TAG,
    AS: 'tags',
},

"$['sortOrder']['example1']": {
        type: SchemaFieldTypes.NUMERIC,
        SORTABLE: true, 
        AS: 'example1' 
}  << I might have 2-600 of these for a given store

When an index exceeds 250 sort definitions (which if you ask redis to create an index it will successfully create one without error or warning) querying for the index with an appropriate SortBy will result in out-of-order objects.

To Reproduce Steps to reproduce the behaviour:

  1. Load in a couple of thousand product objects
  2. Define an arbitrary numeric sort for them in a child property
  3. Create an index that has 250 unique sorts
  4. Query for products, with "SortBy" set to be one of your unique sorts

Expected behaviour This should still work, albeit slowly - or the index should fail to create in the first instance

I can provide access to a redis instance over DM to someone to debug an actual dataset if required.

raz-mon commented 4 months ago

Hi @claridgicus, thanks for reaching out! Which RediSearch version are you using? Can you please share:

Did you change any config params? Sure, an rdb/aof file for us to reproduce with would be great.

claridgicus commented 4 months ago

Hey @raz-mon

I'm using RedisLabs and I've tried two different nodes No config changes

Save and BgSave are unavailable in Redislabs, but I can provide you creds if you can provide me a way to send them to you (to my non production node)

My 7.2 Test Node Redis 7.2.3 Search and query v 2.8.11

My Main Production Node Redis 6.2.6 Search and query v 2.6.15

I have attempted a version with the maxtextfields option - but that doesn't help me either (documentation on that feature could also be slightly improved)

here's my query - nothing too hairy "idx:cherrie-baby:cp:" "(@vendor:Jellycat) (@implode:[1 2])" "SORTBY" "jellycats" "ASC" "LIMIT" "0" "20"

the outputted sort order of the query on a broken index is

[
  undefined, undefined, undefined,
  undefined, undefined, 89,
  undefined, undefined, 15,
  undefined, undefined, undefined,
  undefined, 153,       undefined,
  108,       undefined, undefined,
  undefined, undefined
]

if I give the undefined entries a number say '999' they will still appear out of order in the results

the output on a non broken index is

[
   1,  4,  5,  6,  7,  8,  9,
  10, 11, 12, 13, 14, 15, 16,
  17, 18, 19, 20, 21, 22
]

here's my big boy "index info"

I've tried removing "-", removing any entry in the sort that contains a number in it, creating a base index without the sort fields and altering to add all my sorts

How I currently have it working, whenever I exceed 250 sorts it breaks, so I'm degrading the feature set of my app to provide the sort functionality to the key groups my client requires - without going back to what I was doing, (one index per sort order - which results in a memory usage many times what I can justify for a client)

{
  indexName: 'idx:cherrie-baby:cp:',
  indexOptions: [],
  indexDefinition: [Object: null prototype] {
    key_type: 'JSON',
    prefixes: [ 'cherrie-baby:p:' ],
    default_score: '1'
  },
  attributes: [
    [Object: null prototype] {
      identifier: '$.title',
      attribute: 'title',
      type: 'TEXT',
      WEIGHT: '1',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.tags',
      attribute: 'tags',
      type: 'TAG',
      SEPARATOR: ''
    },
    [Object: null prototype] {
      identifier: '$.id',
      attribute: 'id',
      type: 'TEXT',
      WEIGHT: '1'
    },
    [Object: null prototype] {
      identifier: '$.created_date',
      attribute: 'created_date',
      type: 'TEXT',
      WEIGHT: '1',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.parents-frank-green',
      attribute: 'parents-frank-green',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycats',
      attribute: 'jellycats',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-dog',
      attribute: 'jellycat-dog',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti-dresses',
      attribute: 'minti-dresses',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.subo',
      attribute: 'subo',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.acorn-kids',
      attribute: 'acorn-kids',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-sale',
      attribute: 'girl-sale',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.aster-oak',
      attribute: 'aster-oak',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.wilson-frenchy-boys',
      attribute: 'wilson-frenchy-boys',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby',
      attribute: 'huxbaby',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.big-kids',
      attribute: 'big-kids',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.speedy-monkey',
      attribute: 'speedy-monkey',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.olli-ella-strolleys-luggys',
      attribute: 'olli-ella-strolleys-luggys',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.rock-your-baby-dresses',
      attribute: 'rock-your-baby-dresses',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.connetix-clear',
      attribute: 'connetix-clear',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.goldie-ace',
      attribute: 'goldie-ace',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.activity-craft',
      attribute: 'activity-craft',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.connetix-pastel',
      attribute: 'connetix-pastel',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.teethers-bibs-dummies',
      attribute: 'teethers-bibs-dummies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.summertime',
      attribute: 'summertime',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrival-main',
      attribute: 'new-arrival-main',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-sale-tops',
      attribute: 'girl-sale-tops',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-shoes',
      attribute: 'baby-shoes',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.original-bibs-dummies',
      attribute: 'original-bibs-dummies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-girl-bottoms',
      attribute: 'huxbaby-girl-bottoms',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti-girls',
      attribute: 'minti-girls',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.easter',
      attribute: 'easter',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.the-somewhere-co',
      attribute: 'the-somewhere-co',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-boy-decor',
      attribute: 'baby-boy-decor',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrivals-clothing',
      attribute: 'new-arrivals-clothing',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.connetix-rainbow',
      attribute: 'connetix-rainbow',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.love-to-dream',
      attribute: 'love-to-dream',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.my-dream-light',
      attribute: 'my-dream-light',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.toy-sale-1',
      attribute: 'toy-sale-1',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.ethicool',
      attribute: 'ethicool',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.penny-scallan',
      attribute: 'penny-scallan',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.boxing-day',
      attribute: 'boxing-day',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.kinderfeets',
      attribute: 'kinderfeets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.heydoodle',
      attribute: 'heydoodle',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-mythical',
      attribute: 'jellycat-mythical',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.timber-tinkers',
      attribute: 'timber-tinkers',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.de-lux-bibs-dummies',
      attribute: 'de-lux-bibs-dummies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.milky-girls',
      attribute: 'milky-girls',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.magnetic-play',
      attribute: 'magnetic-play',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.toddler-gifts',
      attribute: 'toddler-gifts',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.womens-shoes',
      attribute: 'womens-shoes',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.salt-water-originals',
      attribute: 'salt-water-originals',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.construction',
      attribute: 'construction',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrival-boy',
      attribute: 'new-arrival-boy',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.write-to-me',
      attribute: 'write-to-me',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.rock-your-baby-girls',
      attribute: 'rock-your-baby-girls',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.alex-ant',
      attribute: 'alex-ant',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti',
      attribute: 'minti',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-amuseable-sports',
      attribute: 'jellycat-amuseable-sports',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-boy-sale-2022',
      attribute: 'baby-boy-sale-2022',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.boxing-day-girl',
      attribute: 'boxing-day-girl',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.wilson-frenchy',
      attribute: 'wilson-frenchy',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.ziggy-lou',
      attribute: 'ziggy-lou',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-swimwear',
      attribute: 'girl-swimwear',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.boy-hair-accessories-hats',
      attribute: 'boy-hair-accessories-hats',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrival-boy-clothing',
      attribute: 'new-arrival-boy-clothing',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.snuggle-hunny-blankets',
      attribute: 'snuggle-hunny-blankets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-girl-sale-2022',
      attribute: 'baby-girl-sale-2022',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti-jumpers-and-jackets',
      attribute: 'minti-jumpers-and-jackets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.bebe',
      attribute: 'bebe',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-swimwear-sale',
      attribute: 'girl-swimwear-sale',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.crywolf',
      attribute: 'crywolf',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-sea-life',
      attribute: 'jellycat-sea-life',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.all-4-ella',
      attribute: 'all-4-ella',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.for-parents',
      attribute: 'for-parents',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.rock-your-baby-swim',
      attribute: 'rock-your-baby-swim',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.nature',
      attribute: 'nature',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.eos-summer-baby',
      attribute: 'eos-summer-baby',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-socks-tights',
      attribute: 'baby-socks-tights',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.mystery-boxes',
      attribute: 'mystery-boxes',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.crywolf-accessories',
      attribute: 'crywolf-accessories',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.milky',
      attribute: 'milky',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.holdie-folk',
      attribute: 'holdie-folk',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.bella-lace',
      attribute: 'bella-lace',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.winter',
      attribute: 'winter',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.frank-green',
      attribute: 'frank-green',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.puzzles',
      attribute: 'puzzles',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.bf40',
      attribute: 'bf40',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.medium-jellycat-bunnies',
      attribute: 'medium-jellycat-bunnies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.wilson-frenchy-girlg',
      attribute: 'wilson-frenchy-girlg',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-glasses-sunglasses',
      attribute: 'baby-glasses-sunglasses',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.ob-designs',
      attribute: 'ob-designs',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-swimwear',
      attribute: 'huxbaby-swimwear',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.dinkum-doll',
      attribute: 'dinkum-doll',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-girl',
      attribute: 'huxbaby-girl',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-girl-clothing',
      attribute: 'baby-girl-clothing',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.towels',
      attribute: 'towels',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.puffers',
      attribute: 'puffers',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.babiators',
      attribute: 'babiators',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.christmas-toys',
      attribute: 'christmas-toys',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.olli-ella-baskets',
      attribute: 'olli-ella-baskets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-girl-tops',
      attribute: 'huxbaby-girl-tops',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    ... 225 more items
  ],
  numDocs: '6949',
  maxDocId: '6949',
  numTerms: '12767',
  numRecords: '179692',
  invertedSzMb: '0.9440984725952148',
  vectorIndexSzMb: '0',
  totalInvertedIndexBlocks: '28787',
  offsetVectorsSzMb: '0.1320819854736328',
  docTableSizeMb: '0.7863950729370117',
  sortableValuesSizeMb: '10.280499458312988',
  keyTableSizeMb: '0.29492950439453125',
  recordsPerDocAvg: '0',
  bytesPerRecordAvg: '25.858684539794922',
  offsetsPerTermAvg: '5.509199142456055',
  offsetBitsPerRecordAvg: '0.7707521915435791',
  hashIndexingFailures: '8',
  indexing: '0',
  percentIndexed: '3131.8291015625',
  gcStats: {
    bytesCollected: undefined,
    totalMsRun: undefined,
    totalCycles: undefined,
    averageCycleTimeMs: undefined,
    lastRunTimeMs: undefined,
    gcNumericTreesMissed: undefined,
    gcBlocksDenied: undefined
  },
  cursorStats: {
    globalIdle: undefined,
    globalTotal: undefined,
    indexCapacity: undefined,
    idnexTotal: undefined
  },
  stopWords: 8
}
claridgicus commented 4 months ago

I can probably spin up a local node and provide a dump if providing the creds is too hard.

raz-mon commented 4 months ago

Hi @claridgicus, Thanks for the info.

Let me make sure I understand you correctly - what you refer to as "sorts" are fields of your index that you may use to sort the results of a query, correct? And when you have more then 250 such fields, the sorting doesn't work (i.e., a "broken index"). Is this the case for all fields you try sorting by? Does the sorting succeed for the same field (that doesn't succeed for more than 250 sorts) and for the same data when you have less than 250 sorts? This is interesting.

Can you provide a dump.rdb file or a script that reproduces this issue? Whatever is more comfortable for you. If you go for the rdb, please state the used versions as well as the queries you run showing the unsuccessful sort.

Thanks

claridgicus commented 4 months ago

@raz-mon I will try to get you a dump tomorrow.

A bit of clarification - I've debugged my way through my application to find the issue occurs when the index hits 256 sortable fields (exact number) regardless of type.

When the index exceeds the 256 - all the sorts stop working.

It does work when I alter the index below the 256 limit, so by just arbitrarily removing some sorts from my index it will resume working normally - so a "broken" index can be altered into a working index.

claridgicus commented 3 months ago

Sorry I have had a mega couple of weeks, I will see about that dump this week

github-actions[bot] commented 1 month ago

This issue is stale because it has been open for 60 days with no activity.