Pub.Lua or Brl.Reflection thread safety?

[GWRon]

Since yesterday I am trying to get my AI scripts (based on Lua) to work in extra threads.

So each AI has a "TLuaEngine"-object which does all the Lua stuff. Now I create a thread for each AI in which Update() (defined in BlitzMax) is constantly called (for now ;-)). This Update() calls a method of the lua engine CallLuaFunction(name, params).

Removing the CallLuaFunction() ... works as expected. Adding in the CallLuaFunction() ... segfault Wrapping the CallLuaFunction into a LockMutex() ... UnlockMutex() ... works

The segfault has this backtrace:

#0  0x00000000005eb818 in avl_del (entry=entry@entry=0x7fffeb6529f0, 
    root=root@entry=0x8ad300 <retain_root>)
    at /BlitzMaxNG/mod/brl.mod/blitz.mod/tree/tree.c:685
#1  0x00000000005cfaa4 in bbGCRelease (p=<optimized out>)
    at /BlitzMaxNG/mod/brl.mod/blitz.mod/blitz_gc.c:187
#2  0x00000000005b5623 in bmx_map_ptrmap_insert (key=0x8a4f60 <bbArrayClass>, 
    value=0x7fffeb64a000, root=<optimized out>)
    at /BlitzMaxNG/mod/brl.mod/map.mod/map.c:169
#3  0x00000000005ae5aa in _brl_map_ptrmap_TPtrMap_Insert_pbTObject (
    o=0x7ffff7e38f70, bbt_key=0x8a4f60 <bbArrayClass> "\240K\212", 
    bbt_value=0x7fffeb64a000)
    at /BlitzMaxNG/mod/brl.mod/map.mod/.bmx/ptrmap.bmx.debug.linux.x64.c:108
#4  0x00000000004a86cb in _brl_reflection_TTypeId_Init_SipbTTTypeId (
    o=0x7fffeb64a000, bbt_name=bbt_name@entry=0x7fffeb655c40, 
    bbt_size=bbt_size@entry=8, 
    bbt_class=bbt_class@entry=0x8a4f60 <bbArrayClass> "\240K\212", 
    bbt_supor=bbt_supor@entry=0x8ad178 <bbNullObject>)
    at /BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x64.c:2990
---Type <return> to continue, or q <return> to quit---
#5  0x00000000004a384f in _brl_reflection_TTypeId_ArrayType_i (o=0x7ffff7e538f0, bbt_dims=1)
    at /BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x64.c:428
#6  0x00000000004c9d6b in brl_reflection_TypeIdForTag (bbt_ty=0x7fffeb655ca0)
    at /BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x64.c:16918
#7  0x00000000004cde31 in _brl_reflection_TTypeId__Resolve (o=0x7ffff12b70d0)
    at /BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x64.c:3728
#8  0x00000000004a9dc3 in brl_reflection_TTypeId__Update_i ()
    at /BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x64.c:3307
#9  0x00000000004ccabd in brl_reflection_TTypeId_ForObject_TTTypeId_TObject (bbt_obj=0x7ffff12afc40)
    at /BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x64.c:2723
#10 0x000000000041366b in __m_base_util_luaengine_TLuaEngine_CallLuaFunction_SaTObject (o=0x7ffff7e5cf00, 
    bbt_name=bbt_name@entry=0x863ad0 <_s9>, bbt_args=bbt_args@entry=0x7ffff7e6a9c0)
    at /Projekte/Dig.git/.bmx/base.util.luaengine.bmx.debug.linux.x64.c:3925
#11 0x0000000000407307 in __m_lua_threaded_TAI_Update (o=0x7ffff7e5bde0)
    at /Projekte/Dig.git/samples/lua/.bmx/lua_threaded.bmx.console.debug.linux.x64.c:391
#12 0x0000000000407037 in _m_lua_threaded_TAI_UpdateThread_TObject_TObject (bbt_data=0x7ffff7e5bde0)
    at /Projekte/Dig.git/samples/lua/.bmx/lua_threaded.bmx.console.debug.linux.x64.c:348
#13 0x000000000059eaeb in brl_threads_TThread__EntryStub_TObject_TObject (bbt_data=0x7ffff7e6ad50)
    at /BlitzMaxNG/mod/brl.mod/threads.mod/.bmx/threads.bmx.debug.linux.x64.c:271
#14 0x00000000005cedc8 in threadProc (p=0x7ffff7e2fbe0)
    at /BlitzMaxNG/mod/brl.mod/blitz.mod/blitz_thread.c:343
#15 0x00000000005f2081 in GC_inner_start_routine (sb=<Fehler beim Lesen der Variable: value has been optimized out>, 

The corresponding code is: https://github.com/GWRon/Dig/blob/master/base.util.luaengine.bmx

Sample code:

SuperStrict
Framework BRL.StandardIO
Import Brl.GlMax2D
Import "../../base.util.luaengine.bmx"

Type TAIFunctions {_exposeToLua}
    Field id:Int = 0

    Method AddLog:Int(s:String)
        logStack.Add("LUA AI"+id+": " + s)
    End Method

    Method CallUpdate:Int(arg:String)
        Print "TMyObject.test() invoked: " +arg
    End Method

    Method CallFromLua:Int(arg:String)
        Print "CallFromLua() invoked: " +arg
    End Method
End Type

Type TAI
    Field id:Int
    Field luaEngine:TLuaEngine
    Field aiFunctions:TAIFunctions
    Field _updateThread:TThread
    Field _updateThreadExit:Int = False

    Method Init(id:Int, script:String)
        Self.id = id

        aiFunctions = New TAIFunctions
        aiFunctions.id = id

        luaEngine = TLuaEngine.Create( LoadText(script) )

        luaEngine.RegisterBlitzmaxObject("MY", aiFunctions)
    End Method

    Method Start()
        Print "AI"+id+": Start()"

        _updateThread = CreateThread(UpdateThread, Self)
    End Method

    Method Stop()
        Print "AI"+id+": Stop()"

        _updateThreadExit = True
        WaitThread(_updateThread)
        'or DetachThread(_updateThread)

        'Reset
        _updateThreadExit = False
        _updateThread = Null
    End Method

    Method IsThreadActive:Int()
        Return _updateThread <> Null
    End Method

    Function UpdateThread:Object(data:Object)
        Local ai:TAI = TAI(data)

        Repeat
            ai.Update()
            Delay(250) 'Wait a bit
        Until ai._updateThreadExit
    End Function

'mutex shared for all
global callLuaFunctionMutex:TMutex = CreateMutex()  
    Method Update:Int()
        logStack.Add("AI" + id+": Update(). Millisecs="+MilliSecs())

        Local args:Object[] = New Object[1]
        args[0] = Object(String(MilliSecs())) 'minute

'============================
'uncomment to remove segfault       
'============================
'       LockMutex(callLuaFunctionMutex)
        luaEngine.CallLuaFunction("Update", args)
'       UnLockMutex(callLuaFunctionMutex)
    End Method
End Type

Type TLogStack
    Field entries:String[]
    Field entriesMax:Int = 20
    Field _addMutex:TMutex = CreateMutex()

    Method Add:String(s:String)
        LockMutex(_addMutex)

        entries :+ [s]
        If entries.length > entriesMax Then entries = entries[ (entries.length - entriesMax) .. ]

        UnlockMutex(_addMutex)
    End Method
End Type

Global logStack:TLogStack = New TLogStack
Global ai:TAI[4]
For Local i:Int = 0 Until 4
    ai[i] = New TAI
    ai[i].Init(i+1, "ai.lua")
Next

Graphics 800, 600
Repeat
    Cls
    If KeyHit(KEY_SPACE) And Not ai[0].IsThreadActive()
        For Local i:Int = 0 Until 4
            ai[i].Start()
        Next
    EndIf

    For Local i:Int = 0 Until logStack.entries.length
        DrawText(logStack.entries[i], 10, 10 + i*12)
    Next

    Flip
Until KeyHit(KEY_ESCAPE) Or AppTerminate()

ai.lua:

function Update(t)
  MY.AddLog("test ")
end

Is this a reflection issue or does the Lua binding trash some stuff?

[GWRon]

I tried to wrap the TTypeID.ForObject call (so avoids potentially inserting into the map "simultaneously") ...

?threaded
LockMutex(reflectionMutex)
?
                Local typeId:TTypeId = TTypeId.ForObject(args[i])
?threaded
UnLockMutex(reflectionMutex)
?

But this still lead to an segfault (in a different part):

#0  bbExThrow (p=0x7fffe32c65d0) at /BlitzMaxNG/mod/brl.mod/blitz.mod/blitz_ex.c:138
#1  0x00000000005c5dcc in brl_blitz_NullObjectError ()
    at /BlitzMaxNG/mod/brl.mod/blitz.mod/.bmx/blitz.bmx.debug.linux.x64.c:555
#2  0x00000000005c6e40 in bbNullObjectTest (o=0x8ad178 <bbNullObject>)
    at /BlitzMaxNG/mod/brl.mod/blitz.mod/blitz_object.c:231
#3  0x00000000004a5ef2 in _brl_reflection_TTypeId_FindMethod_S (o=0x7ffff0223d00, bbt_name=0x7fffe32d3860)

Think the brl.mod/reflection.mod is not threadsafe yet.

[HurryStarfish]

Are you using the reflection module from master or the brl_reflect branch? Either way, a lot of stuff in both of them currently isn't thread safe. I'm been planning to fix this eventually, but we should get a decision (from @woollybah?) on the topic of libffi: are we okay to use it in BlitzMax-NG from now on (it might restrict the platforms NG can run on)? If so, I'd like him to commit libffi to the brl_reflect branch (as mentioned before in bmx-ng/bcc/issues/413), since I had trouble getting it to build correctly. I'll then continue working on the reflection module.

[GWRon]

I am using the "master" branch.

So it seems for now (while developing the threaded AI) I need to add some more mutexes.