Raises --> 'list index out of range' when keys loaded > ~605K

[msarm]

This issue is very interesting but consistent when we start loading keys > ~605K.

Run the below code and you have to wait for an hour or more based on your system configuration to replicate this issue.

import datetime
from typing import Optional

from redis_om import Field, HashModel, Migrator, get_redis_connection

# This Redis instance is tuned for durability.
REDIS_DATA_URL = "redis://localhost:6379"

class Person(HashModel):
    first_name: str = Field(index=True)
    last_name: str = Field(index=True)
    emp_no: int =  Field(index=True, sortable=True)

    class Meta:
        global_key_prefix = "person"

# set redis connection
Person.Meta.database = get_redis_connection(url=REDIS_DATA_URL,
                                                  decode_responses=True)
# apply migrations
Migrator().run()

# initilize employee
emp_no = 0

try:

    # find the last employee added
    result = Person.find().sort_by('-emp_no').first()

    # if we find a result, we get the last added employee number
    if (result is not None):
        emp_no = result.emp_no

except Exception as ex:
    # when record not exist, then pass
    pass

# set a fag for while loop 
able_to_find_record = True

# add employe untill redis returns the last added employee 
while (able_to_find_record):

    # add new employee
    emp_no += 1
    person = Person(first_name="John" + str(emp_no), last_name="Doe", emp_no=emp_no)
    person.save()

    # get the last added employee
    result = Person.find().sort_by('-emp_no').first()

    # when redis fails to find the last added employee, we exit the while loop
    if (result is None or result.emp_no!= emp_no):
        able_to_find_record = False
        print("Unable to find record" + str(emp_no))

Error is raised from the below method, it expects the result here but the Redis search library returns an empty array. I believe the issue is with the Redis library or Redis server when keys are over > ~605K

    @classmethod
    def from_redis(cls, res: Any):
        # TODO: Parsing logic copied from redisearch-py. Evaluate.
        import six
        from six.moves import xrange
        from six.moves import zip as izip

        def to_string(s):
            if isinstance(s, six.string_types):
                return s
            elif isinstance(s, six.binary_type):
                return s.decode("utf-8", "ignore")
            else:
                return s  # Not a string we care about

        docs = []
        step = 2  # Because the result has content
        offset = 1  # The first item is the count of total matches.

        for i in xrange(1, len(res), step):
            fields_offset = offset

            fields = dict(
                dict(
                    izip(
                        map(to_string, res[i + fields_offset][::2]), <<-- Issue is raised here

Here is the error raised

list index out of range
  File "/Users/user/projects/dev/fxLib/.venv/lib/python3.9/site-packages/redis_om/model/model.py", line 1204, in from_redis
    map(to_string, res[i + fields_offset][::2]),
  File "/Users/user/projects/dev/fxLib/.venv/lib/python3.9/site-packages/redis_om/model/model.py", line 727, in execute
    results = self.model.from_redis(raw_result)
  File "/Users/user/projects/dev/fxLib/.venv/lib/python3.9/site-packages/redis_om/model/model.py", line 752, in first
    results = query.execute(exhaust_results=False)
  File "/Users/user/projects/dev/fxLib/tests/Test_redis2.py", line 34, in <module> (Current frame)
    result = Person.find().sort_by('-emp_no').first()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,

I'm using below stack: docket image: "redis/redis-stack:latest" (may be 6.2.2-v1) DIGEST:sha256:27666e8e1b632cc02bfb926bf9cbbda650aed2b818444c58613379167e12369e

redis --> 4.2.2 redis-om --> 0.0.26

Let me know if you need more information.

[msarm]

Update: tested with the following docker images and the issue still persists in all images.

• 6.2.2-V2
• edge-edge
• edge
• 7.0.0-RC2-edge

[msarm]

UPDATE: Was able to go further and found out its Redis server issue and NOT with redis-om or redis library.

When I fired the last added employee query directly using RedisInsight, I get no results but the match count shows 650K but not results.

[msarm]

@simonprickett - It's clear to me now that it's not a redis-om issue, do I need to close here and open a new issue in the Redis server repository?

[msarm]

Update: Here is FT.INFO & FT.PROFILE

[simonprickett]

@msarm yes I think this needs to be a new issue against RediSearch at https://github.com/redisearch/redisearch - thanks.

[msarm]

Finally, figured out what is going on.

Here are my learnings:

• client timeout (close the connection with the client if the client is idle for X seconds)
  • configure this limit via Redis.conf
  • simply using CONFIG SET timeout < value>
• query timeout (max amount of time in milliseconds that a search query is allowed to run)
  • redis-server --loadmodule ./redisearch.so TIMEOUT < value >
  • FT.CONFIG SET timeout < value >
    • ON_TIMEOUT policy (response policy for queries that exceed the TIMEOUT setting - default is RETURN)
    • RETURN: this policy will return the top results accumulated by the query until it timed out.
    • FAIL: will return an error when the query exceeds the timeout value.

Here is what happened here:

• wrongly usage of query timeout with client timeout (confused since both share the same parameter name TIMEOUT)
• default response policy is to return results without raising an error, in my case it timed out with the default 500 ms and the server was not able to find any results until it timed out and returned an empty result without raising any error.