Executing a Lua function from within Elixir does not return the mutated state

[markmeeus]

I'm building an elixir app where admins can schedule some functions somewhere in the future. In order to call them later, I store these functions in a genserver state.

It seems that these functions can be called against a newer state (not sure), but they don't return the updated state after being called.

In the example below I was expecting the last match to fail because the state should have changed (var should be "Update global") Also when inspecting the state, that global var is still "Later global var".

Am I calling the encoding and the function incorrectly? Or would this be a bug in luerl?

code = """
  var = "Some global var"

  function update() 
    var = "Updated global"    
  end

  var = "Later global var"
  schedule(update)  
"""

{:ok, schedule_agent} = Agent.start_link fn -> nil end

schedule_handler = fn [fun], luerl_state ->
  :ok = Agent.update(schedule_agent, fn _ -> fun end)
  {[true], luerl_state}
end

luerl_state = :luerl_sandbox.init()
luerl_state = :luerl.set_table(["schedule"], schedule_handler, luerl_state)

# running the code calls the schedule function
{:ok, [], luerl_state} = :luerl_new.do(code, luerl_state)

# we can now fetch the schedule function from our agent
scheduled_func = Agent.get(schedule_agent, fn func -> func end)
# encode, and call
{{:erl_func, encoded_func}, luerl_state} =:luerl_new.encode(scheduled_func, luerl_state)
{_, ^luerl_state} = encoded_func.([], luerl_state) 

[markmeeus]

I'm quite sure encoding the funtion is not what I need to do here. It's not a erl function. It looks like call_chunk is the function I need to call, but when I do that:

{:lua_error, {:undefined_function, #Function<6.103755144/1 in :luerl.decode/3>},

I inspected the luerl_state and indeed, that function is not in there.

[markmeeus]

I think I may have pinpointed the problem In luerl.erl line 387 there is this function def:

decode(#funref{}=Fun, State, _) ->
    F = fun(Args) ->
        {Args1, State1} = encode_list(Args, State),
        {Ret, State2} = luerl_emul:functioncall(Fun, Args1, State1),
        decode_list(Ret, State2)
    end,
    F;  

If I read this correctly, an internal function is decoded to an external function by wrapping it in a function that encodes the args and decodes the result. But it only returns the result, discarding the new state.

so instead of decode_list(Ret, State2) it should return {decode_list(Ret, State2), State2}

However, that would be a breaking change.... I will try to take the unencoded route and implement this wrapper myself for now

[markmeeus]

I think I found a way to have it working in a backwards compatible way

In the luerl.erl file:

decode(#funref{}=Fun, State, _) ->
    F = fun([#luerl{}=NewSt|Args]) ->
            {Args1, State1} = encode_list(Args, NewSt),
            {Ret, State2} = luerl_emul:functioncall(Fun, Args1, State1),
            {decode_list(Ret, State2), State2};
        (Args) ->
            {Args1, State1} = encode_list(Args, State),
            {Ret, State2} = luerl_emul:functioncall(Fun, Args1, State1),
            decode_list(Ret, State2)
    end,
    F;                      %Just a bare fun

This way, the function can be called from within erlang with a more recent state than when the function was passed from lua to erl. Also, when passing a new luerl state, a new luerl state will be returned as the second tuple value.

Would this change be acceptable in a PR? If so, I'll be happy to create one.

[markmeeus]

After some thought and investigation I think there are a few major concerns with this approach.

First, constructing the function like this, the old state will have to kept around in Erlang memory because it is used by the returned function.

If the Lua script would call schedule many times, it would mean a separate copy of luerl state is kept in memory for as long as the functions are alive in Erlang land. This could start to add up quite quickly.

This got me thinking about GC, also in Lua land, because if Lua doesn't know Erlang is referencing this function, and Lua isn't, it would probably cause some problems. And indeed, calling gc on luerl causes an error when I call the function with the gc'ed state... (The script above is fine, but passing an anonymous function to schedule isn't.

I'll have to do some more research to see if there is a way around this.

(I have tried implementing the schedule function entirely in Lua, keeping the function in a global table, and passing the key to erlang, but for some reason, I seem to be getting errors after GC in this scenario as well, strange ...)

[markmeeus]

Closing this ticket because it all comes down to not using the encoded path.