Add better support for LUA scripts

[JCHacking]

Is your feature request related to a problem? Please describe. I am trying to mock the redi backend for asgi-ratelimit and I am encountering the following error:

Traceback (most recent call last):
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\commands\core.py", line 5172, in __call__
    return await client.evalsha(self.sha, len(keys), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 610, in execute_command
    return await conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\retry.py", line 59, in call_with_retry
    return await do()
           ^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 584, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 631, in parse_response
    response = await connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\fakeredis\aioredis.py", line 171, in read_response
    raise response
redis.exceptions.NoScriptError: No matching script. Please use EVAL.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\starlette\middleware\errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\ratelimit\core.py", line 94, in __call__
    retry_after = await self.backend.retry_after(path, user, rule)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\ratelimit\backends\redis.py", line 51, in retry_after
    await self.lua_script(keys=list(ruleset.keys()), args=[json.dumps(ruleset)])
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\commands\core.py", line 5178, in __call__
    return await client.evalsha(self.sha, len(keys), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 610, in execute_command
    return await conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\retry.py", line 59, in call_with_retry
    return await do()
           ^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 584, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\redis\asyncio\client.py", line 631, in parse_response
    response = await connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jcmencia\PycharmProjects\NGCSQALIBS\py_api\venv\Lib\site-packages\fakeredis\aioredis.py", line 171, in read_response
    raise response
redis.exceptions.ResponseError: Error running script (call to f_283a893bdb53a688600eff30001e6fb4e4ebd4f5): @user_script:?: [string "<python>"]:2: attempt to index a nil value (global 'cjson')
stack traceback:
    [string "<python>"]:2: in main chunk

Describe the solution you'd like Add support for cjson? I don't know if this is only the problem. I don't know exactly if this would be for lupa or for fakeredis, if you can give me some guidance I would appreciate it.

Related issues:

• https://github.com/jamesls/fakeredis/issues/25
• https://github.com/jamesls/fakeredis/issues/183

Additional context I have tried this code:

from fakeredis.aioredis import FakeRedis
from fastapi import FastAPI
from ratelimit import RateLimitMiddleware, Rule
from ratelimit.backends.redis import RedisBackend
from ratelimit.types import Scope

app = FastAPI(debug=True)

async def authenticate(scope: Scope) -> tuple[str, str]:
    return str(scope["client"]), "default"

app.add_middleware(
    RateLimitMiddleware,
    authenticate=authenticate,
    backend=RedisBackend(
        FakeRedis.from_url("redis://redis")
    ),
    config={
        "": [Rule(minute=1)]
    }
)

@app.get("/")
async def test():
    return "A"

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[cunla]

Did you install fakeredis with lua support? https://fakeredis.readthedocs.io/en/latest/#installation

[JCHacking]

Did you install fakeredis with lua support? https://fakeredis.readthedocs.io/en/latest/#installation

Yes

[cunla]

I checked, to load cjson, it should be compiled for the machine where fakeredis is running.

I might add an environment variable LUA_MODULES stating which modules should be loaded.

[JCHacking]

So from what you say, to avoid this error I have to compile it on the machine that is going to run it with the environment variable LUA_MODULES=cjson right?

Or could this be a bug in the lua script? It looks like this is the lua script running underneath. I understand that the failure is that it does not find the cjson module.

local ruleset = cjson.decode(ARGV[1])

-- Set limits
for i, key in pairs(KEYS) do
    redis.call('SET', key, ruleset[key][1], 'EX', ruleset[key][2], 'NX')
end

-- Check limits
for i = 1, #KEYS do
    local value = redis.call('GET', KEYS[i])
    if value and tonumber(value) < 1 then
        return ruleset[KEYS[i]][2]
    end
end

-- Decrease limits
for i, key in pairs(KEYS) do
    redis.call('DECR', key)
end
return 0

[cunla]

Well, I would need to implement this, but yeah, eventually, that would be the way it works.

[JCHacking]

I use poetry to manage dependencies, so when I get a whl from pypi I will download that one. I understand that in that case I will not use anything from LUA_MODULES because I will download it already compiled.

[cunla]

does cjson have a pypi package?

[JCHacking]

It seems that there are packages like ujson that give this functionality (I have not tested them).

[cunla]

We need something where this would run:

import lupa

lupa.LuaRuntime().execute("require('cjson')")

I don't think ujson meets this requirement

[JCHacking]

Trying with this code I get this error:

lupa.lua54.LuaError: [string "<python>"]:1: module 'cjson' not found:
    no field package.preload['cjson']
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\lua\cjson.lua'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\lua\cjson\init.lua'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\cjson.lua'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\cjson\init.lua'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\..\share\lua\5.4\cjson.lua'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\..\share\lua\5.4\cjson\init.lua'
    no file '.\cjson.lua'
    no file '.\cjson\init.lua'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\cjson.dll'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\..\lib\lua\5.4\cjson.dll'
    no file 'C:\Users\jcmencia\AppData\Local\Programs\Python\Python312\loadall.dll'
    no file '.\cjson.dll'
stack traceback:
    [string "<python>"]:1: in main chunk
    [C]: in function 'require'

A little closer to making it work we are!

[JCHacking]

This is probably something that should be implemented in the lupa package.

[cunla]

lupa supports importing modules, it simply does not have cjson ootb.

I managed to install cjson:

• First, install luarocks (I am using mac so brew install luarocks, but there is an installation for windows).
• Then, install cjson using luarocks install lua-cjson
• Then, copy cjson.so (or in your case I assume cjson.dll) to the project homedir.
• Then, the code I wrote should work.

[JCHacking]

I have been researching and there is a "cleaner" way to do this:

import lupa

lupa.LuaRuntime().require("cjson")

[cunla]

Does it work without installing cjson first?

[JCHacking]

Does it work without installing cjson first?

No, as you said it seems that lupa does not bring the cjson module with it.

This makes more "complex", in my case, mock redis for asgi-ratelimit since I want for my CI to use python base only, and it doesn't seem right to upload cjson.dll or cjson.so to the repository either.

[cunla]

yeah, I am not sure why asgi-ratelimit chose to implement a ratelimit using a script that uses cjson - it seems a bit cumbersome. But regardless, fakeredis now supports having LUA modules

[dvora-h]

@cunla Hi, I have the same issue, and I don't understand what the solution is - where should I add this code lupa.LuaRuntime().require("cjson")? and where to copy the cjson.so file?

[dvora-h]

This is the error I'm getting:

lupa.LuaRuntime().require("cjson")
lupa/lua54.pyx:502: in lupa.lua54.LuaRuntime.require
    ???
lupa/lua54.pyx:1835: in lupa.lua54.call_lua
    ???
lupa/lua54.pyx:1861: in lupa.lua54.execute_lua_call
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   lupa.lua54.LuaError: module 'cjson' not found:
E       no field package.preload['cjson']
E       no file '/usr/local/share/lua/5.4/cjson.lua'
E       no file '/usr/local/share/lua/5.4/cjson/init.lua'
E       no file '/usr/local/lib/lua/5.4/cjson.lua'
E       no file '/usr/local/lib/lua/5.4/cjson/init.lua'
E       no file './cjson.lua'
E       no file './cjson/init.lua'
E       no file '/usr/local/lib/lua/5.4/cjson.so'
E       no file '/usr/local/lib/lua/5.4/loadall.so'
E       no file './cjson.so'
E   stack traceback:
E       [C]: in function 'require'

[cunla]

Shalom Dvora,

Here is an explanation how to use it in fakeredis.

The gist:

import fakeredis
r = fakeredis.FakeStrictRedis(lua_modules={"my_module.so"})

In terms of where to put cjson.so - You have in the stacktrace where LUA is looking for it. Note that lupa's LUA runtime is not your system's installed LUA runtime, e.g., if you are using luarocks and you have cjson.so in $LUA_LIBDIR - lupa won't search for it.

I recommend you add it to the project's working directory (./cjson.so) if you are sharing the code with others - that way, you can commit cjson.so in the root dir of the repo and when others pull they won't need to do extra configuration.

[dvora-h]

Thanks @cunla for the quick response! this is my code: FakeRedis(decode_responses=True, lua_modules={"cjson.so"}) it's in my_test.py file, I'm not sure I understand what you said about luarock, I'm using it, but copied the the cjson.so to the same directory where my_test.py file is located, so this is the stucture: tests: my_test.py cjson.so I'm still getting this error:

2024-05-30 15:43:20,330 ERROR [77342]  Failed to load LUA module "cjson.so", make sure it is installed: module 'cjson.so' not found:
    no field package.preload['cjson.so']
    no file './cjson/so.lua'
    no file '/usr/local/share/lua/5.1/cjson/so.lua'
    no file '/usr/local/share/lua/5.1/cjson/so/init.lua'
    no file '/usr/local/lib/lua/5.1/cjson/so.lua'
    no file '/usr/local/lib/lua/5.1/cjson/so/init.lua'
    no file './cjson/so.so'
    no file '/usr/local/lib/lua/5.1/cjson/so.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'
    no file './cjson.so'
    no file '/usr/local/lib/lua/5.1/cjson.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'

Am I doing something wrong?

[cunla]

your code seems right. How do you run the tests? what is the working directory relative to where is cjson.so?

say the structure is:

./
./cjson.so
./tests
./tests/my_test.py

then if you are executing the test from ./ you should be fine. If you are running the test from ./tests/ then move cjson.so to that directory.

[dvora-h]

I run from tests directory, with pytest pytest my_test.py::test_fake_redis and my_test.py and cjson.so are both in the tests directory, I think this is how it should be, but I still have the error. Do you have any idea why?

[cunla]

... I am pretty sure the working directory you are running from is incorrect.

Try this:

Add to your test: os.listdir() to see whether cjson.so is there.

• If it is not, then check which dir you are in in the python context and copy the file there.
• If it is, then from a python console, try running (this is the source):

  
  import lupa
  with lupa.allow_lua_module_loading()
   from lupa import lua54

lua = lua51.LuaRuntime() lua.require('cjson')

If this gives the same error (and the file is there), it may be corrupted, or compiled for the wrong LuaRuntime version.

[dvora-h]

@cunla The working directory is correct, the cjson.so file is in the list. I don't know if I need to try this code with lua54 or lua51, it imports 54 but use 51, what is right?

with lupa.allow_lua_module_loading()
     from lupa import lua54

lua = lua51.LuaRuntime()

And thanks again.

[cunla]

Up to you what you are trying to do. Redis is using 5.1.

You need to use the same version inside the with and outside.