Segfault with reflection and arrays

[GWRon]

I am using this code to clone objects:

    'clones the given object
    'function is calling itself recursively for each property
    'returns the cloned object
    Function CloneObject:object(obj:object)
        'clone code is based on the work of "Azathoth"
        'http://www.blitzbasic.com/codearcs/codearcs.php?code=2132

        'skip cloning nothing
        If obj = Null Then Return Null

        'to access properties we need a TTypeID of the object
        Local objTypeID:TTypeId=TTypeId.ForObject(obj)

        '=== STRINGS ===
        If objTypeID.ExtendsType(StringTypeId) then return String(obj)

        '=== ARRAYS ===
        If objTypeID.ExtendsType(ArrayTypeId)
            'if an array does not contain elements, the reflection
            'cannot recognize which type the array contains (Null[])

            'accessing this "null[]"-arrays would lead to a thrown
            'error - so we need "try" to catch the exception.
            'Thanks to Brucey's persistence.mod (doing it similar)

            'objects name might be TMyType[] - remove the []-part
            Local objTypeName:string = objTypeID.name()[..objTypeID.name().length - 2]
            Local size:Int
            Try
                size = objTypeID.ArrayLength(obj)
            Catch e$
                objTypeName = "Object"
                size = 0
            End Try

            'if the object does not contain things in that array, the
            'copy wont need it too
            If size = 0 then return Null

            'create new array
            local clone:object = objTypeID.NewArray(objTypeID.ArrayLength(obj))
            'something failed, return a null object
            If not clone then return Null

            'clone each element of the array
            For Local i:int=0 Until objTypeID.ArrayLength(obj)
                'run recursive clone for arrays, objects and strings
                If objTypeID.ElementType().ExtendsType(ArrayTypeId) or objTypeID.ElementType().ExtendsType(StringTypeId) or objTypeID.ElementType().ExtendsType(ObjectTypeId)
                    objTypeID.SetArrayElement(clone, i, CloneObject(objTypeID.GetArrayElement(obj, i)))
                Else
                    objTypeID.SetArrayElement(clone, i, objTypeID.GetArrayElement(obj, i))
                EndIf
            Next

            return clone
        EndIf

        Local clone:object

        'use the objects specific clone method instead of our
        'generic approach
        'call a method "CloneObject:Int(original:obj)"
        Local mth:TMethod = objTypeID.FindMethod("CloneObject")
        If mth
            clone = objTypeID.NewObject()
            mth.Invoke(clone, [obj])
        Else

            '=== LISTS ===
            If objTypeID.ExtendsType(ListTypeID)
                local list:TList = CreateList()
                For local entry:object = EachIn TList(obj)
                    list.AddLast( CloneObject(entry) )
                Next
                return list
            EndIf

            '=== TMAPS ===
            If objTypeID.ExtendsType(MapTypeID)
                local map:TMap = CreateMap()
                For local key:string = EachIn TMap(obj).Keys()
                    map.Insert(key, CloneObject(TMap(obj).ValueForKey(key)) )
                Next
                return map
            EndIf   

            '=== OBJECTS ===
            'create a new instance of the objects type
            'Local clone:object = New obj
            clone = objTypeID.NewObject()

            'loop over all fields of the object
            For Local fld:TField=EachIn objTypeID.EnumFields()
                Local fldId:TTypeId=fld.TypeId()

                'only clone non-null-fields and if not explicitely forbidden
                If fld.Get(obj) And fld.MetaData("NoClone") = Null
                    'if explizitely stated, clone referenceable objects by
                    'reusing their reference, else deep clone it
                    If fld.MetaData("CloneUseReference")
                        fld.Set(clone, fld.Get(obj))
                    Else
                        fld.Set(clone, CloneObject(fld.Get(obj)))
                    EndIf
                EndIf
            Next
        EndIf

        'inform the clone that it got cloned
        'call a method "onGotCloned:Int(original:obj)"
        mth = objTypeID.FindMethod("onGotCloned")
        If mth then mth.Invoke(clone, [obj])

        Return clone
    End Function    

Using it, segfaults for me:

Program received signal SIGSEGV, Segmentation fault.
0x0859c949 in brl_reflection_TTypeId_ForObject (bbt_obj=0xc4322d0)
    at /home/ronny/Arbeit/Programmieren/BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x86.c:2497
2497            return (((struct brl_reflection_TTypeId_obj*)bbt_)->clas)->md_ArrayType(bbt_,1);

Backtrace:

#0  0x0859c949 in brl_reflection_TTypeId_ForObject (bbt_obj=0xc4322d0)
    at /home/ronny/Arbeit/Programmieren/BlitzMaxNG/mod/brl.mod/reflection.mod/.bmx/reflection.bmx.debug.linux.x86.c:2497
#1  0x08350475 in _base_util_helper_THelper_CloneObject (bbt_obj=0xc4322d0)
    at /home/ronny/Arbeit/Programmieren/BlitzMaxNG/TVTower-master/source/Dig/.bmx/base.util.helper.bmx.debug.linux.x86.c:459
#2  0x08351c3d in _base_util_helper_THelper_CloneObject (bbt_obj=0xe28f7f8)
    at /home/ronny/Arbeit/Programmieren/BlitzMaxNG/TVTower-master/source/Dig/.bmx/base.util.helper.bmx.debug.linux.x86.c:896

The last line in the c-file references fld.Set(clone, CloneObject(fld.Get(obj))) (when iterating over fields). The second last line references the portion

    'to access properties we need a TTypeID of the object
    Local objTypeID:TTypeId=TTypeId.ForObject(obj)

I printed out the field name right before "fld.Set()" and it was the field "cast" in my case, which is defined as Field cast:TProgrammePersonJob[] (an array).

So somehow this crashes when "obj" is an array of this type. I tried to reproduce it in a simple example but it did not crash for this. Maybe I am interpreting the BackTrace in a wrong way?

[woollybah]

Compiling with option "-gdb" can sometimes be useful too.

[GWRon]

Maybe you should add this option to MaxIDE then (Debug, Debug GDB). Recompiled the whole project with "-gdb" (recompile enforced as it else reused the normal debugbuild which existed already)

./bmk makeapp -t console -d -a -gdb "/path/to/my/project.bmx" ... happily recompiled the project ...

when running the project via "gdb" the crash details are the same (same for backtrace).

Did I miss a step?

EDIT: ok, recompiled the modules too:

Program received signal SIGSEGV, Segmentation fault.
0x0859cad9 in brl_reflection_TTypeId_ForObject (bbt_obj=0xce05330)
    at /home/ronny/Arbeit/Programmieren/BlitzMaxNG/mod/brl.mod/reflection.mod/reflection.bmx:1813
1813            EndIf

Right before that endif the source is this:

    Function ForObject:TTypeId( obj:Object )
        _Update
        Local class:Byte Ptr=bbRefGetObjectClass( obj )
        If class=ArrayTypeId._class
            If Not bbRefArrayLength( obj ) Return ArrayTypeId
            Return TypeIdForTag( bbRefArrayTypeTag( obj ) ).ArrayType()
        Else
            Return TTypeId( _classMap.ValueForKey( New TClass.SetClass( class ) ) )
        EndIf

So something there is not working as intended ?

[GWRon]

Ok, So I appended some "debug print" right before that Return TTypeId(...)

            if not _classMap then Print "ForObject: _classMap Missing"
            '

            if not class then Print "ForObject: class is null"
            '

            if not New TClass.SetClass( class ) then print "ForObject: SetClass failed"
            '

            if not _classMap.ValueForKey( New TClass.SetClass( class ) ) then Print "ForObject: _classMap.ValueForKey NULL"

            Return TTypeId( _classMap.ValueForKey( New TClass.SetClass( class ) ) )

It fails at the first ' which I assume means, it failed the line before (like with that "endif"). Which means _classMap is null in that call. But this _classMap is a global one ...so I assume this cannot be the case. Why is it failing at odd lines (like EndIf or ') ...

Ok ... so I expanded the first if to be:

            If _classMap
                Print "ForObject: _classMap exists"
            Else
                print "ForObject: _classMap Missing"
            EndIf

Recompiled modules, recompiled (of course then) my project ... and it prints some "_classMap exists" when cloning objects ... but as soon as it has to clone the type with "cast:TProgrammePersonJob[]" it fails at the line: Print "ForObject: _classMap exists" (which I think means it fails on If _classMap) ... but how could such a thing fail?

[GWRon]

I tried more and more things - I even created a new file only importing things used in the specific situation - and it does not crash there (cloning the basic objects by hand) but it fails as soon as it is done via the more complicated routines in the db-xml-loader.

I assume that there is something borked in reflection.mod and it needs a rare situation to make it break.

Feel free to download the most current version from: https://github.com/GWRon/TVTower and compile on your own. As compilation is a bit slow, I prepared a smaller example (still using much of the game objects) - this sample also segfaults. Place it next to "TVTower.bmx":

SuperStrict
Framework Brl.StandardIO
Import "source/game.database.bmx"

Global l:TDatabaseLoader = New TDatabaseLoader
l.Load("res/database/Default/database.xml")

The problematic line is in source/game.database.bmx:

    Method LoadV3ProgrammeLicenceFromNode:TProgrammeLicence(node:TxmlNode, xml:TXmlHelper, parentLicence:TProgrammeLicence = Null)
...
            'try to clone the parent's data - if that fails, create
            'a new instance
            if parentLicence then programmeData = TProgrammeData(THelper.CloneObject(parentLicence.data))

"data" is valid (checked that before).

But it only fails if the "TProgrammeData" contains something in the field "cast:TProgrammePersonJob[]" (leave it empty be commenting out the "AddCast")

Edit: it is not important if the "cast" is of type "TProgrammePersonJob" - it segfaults in that type as soon as I append a "complex" type:

Type TProgrammeData extends TGameObject {_exposeToLua}
    Field objectArray:TMap[] = [new TMap,new TMap]

will fail at "objectArray" then...

[GWRon]

Ok, I shrinked my example to this one:

SuperStrict
Framework Brl.StandardIO

Import "source/game.programme.programmedata.bmx"
Import "source/game.programme.programmeperson.bmx"

'create a base person so the cast of the data block can get filled
Local person:TProgrammePerson = New TProgrammePerson

'create an entry
Local programmeData:TProgrammeData = New TProgrammeData
'add sample cast - only fails with something set in the cast (else it is not "cloned")
programmeData.AddCast(New TProgrammePersonJob.Init(person, TVTProgrammePersonJob.ACTOR))

'this fails (when "cast"-field is the thing to clone)
THelper.CloneObject(programmeData)

Print "cloned..."

Any hints to narrow it down even further?

[GWRon]

Ok ... next try, this time I removed every external source, so the following code is a "single file" example:

EDIT: made the code even shorter:

SuperStrict
Framework Brl.StandardIO
Import Brl.Reflection

Type TDataBlock
    Field cast:TSubData[]
End Type

Type TSubData
    Field bla:Int
End Type

Local data:TDataBlock = New TDataBlock
'add sample entry - only works with this
data.cast :+ [New TSubData]

Print "Fetching objTypeIDs: "
Local objTypeID:TTypeId = TTypeId.ForObject(data)
For Local fld:TField = EachIn objTypeID.EnumFields()
    Print "  Field: "+fld.name
    If fld.Get(data)
        Print "  Field: not empty!"
        'this fails for our array
        Local subObjTypeID:TTypeId = TTypeId.ForObject(fld.Get(data))
    EndIf
Next
Print "Fetched."

It fails as soon as it has to get the "ForObject" of the array.

[woollybah]

The latest bcc commit should hopefully resolve this issue.

[GWRon]

BTW: If you want to close an issue in another repo: "Closes/Fixes bmx-ng/brl.mod#14" in your bcc-ng-commit would close this issue.

Sample of course compiles now...

ATM recompiling my game (takes a while) ... finished, failing in another situation now ... so prepare for new issues :laughing: