UE4Editor crashes when trying to destroy / spawn actor with Assertion failed: MeshObject

[paku2k]

Hello, I am experiencing frequent crashes with CARLA 0.9.14 built from source. I am starting the UE4Editor with the command {UE4_PATH} {CARLA_PROJECT_PATH} -game, -windowed -ResX=800, -ResY=800

I have a script which automatically navigates a vehicle around Town07 (also tried with Town04) and then gets destroyed and respawns at the beginning. At the 2nd or 3rd destroy (seems a little bit random), the UE4Editor crashes with the following fault: `[2024.07.04-09.25.17:055][972]LogCarla: UActorDispatcher::Destroying actor: 'vehicle.bmw.grandtourer' 4276ab00 [2024.07.04-09.25.17:055][972]LogCarla: BP_BmwGranTourer_C_1 [2024.07.04-09.25.17:056][972]LogCarla: Spawning actor 'vehicle.bmw.grandtourer' [2024.07.04-09.26.04:877][997]LogOutputDevice: Warning:

Script Stack (0 frames):

[2024.07.04-09.26.04:881][997]LogCore: Error: appError called: Assertion failed: MeshObject [File:/data/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/SkeletalMesh.cpp] [Line: 4720]

[2024.07.04-09.26.04:896][997]LogCore: === Critical error: === Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000003

[2024.07.04-09.26.04:896][997]LogCore: Assertion failed: MeshObject [File:/data/UnrealEngine_4.26/Engine/Source/Runtime/Engine/Private/SkeletalMesh.cpp] [Line: 4720] `

I have attached some log files with several occurrences of the crashing problem. CarlaUE4_1.log CarlaUE4_2.log CarlaUE4_3.log Diagnostics.txt

OS: Ubuntu 20.04 Graphics Card: 2 x Nvidia GTX1080 TI with driver 535.183.01 Carla: 0.9.14 UE4Editor: 4.26

I can't really figure out the reason for the crashing or what I could try to fix it. Is someone able to help me?

[Blyron]

It's a code issue we will need into this.

[Blyron]

Could you give us more info on how to repro this?

[Blyron]

Try to apply this piece of code in Unreal\CarlaUE4\Plugins\Carla\Source\Carla\Game\TaggedComponent.cpp Line 129 `` FPrimitiveSceneProxy UTaggedComponent::CreateSceneProxy(USkeletalMeshComponent SkeletalMeshComponent) { if (bShouldWaitFrame) { return nullptr; } ERHIFeatureLevel::Type SceneFeatureLevel = GetWorld()->FeatureLevel; FSkeletalMeshRenderData* SkelMeshRenderData = SkeletalMeshComponent->GetSkeletalMeshRenderData();

// Only create a scene proxy for rendering if properly initialized
if (SkelMeshRenderData &&
    SkelMeshRenderData->LODRenderData.IsValidIndex(SkeletalMeshComponent->PredictedLODLevel) &&
    !SkeletalMeshComponent->bHideSkin &&
    IsValid(SkeletalMeshComponent->MeshObject))
{
    // Only create a scene proxy if the bone count being used is supported, or if we don't have a skeleton (this is the case with destructibles)
    int32 MinLODIndex = SkeletalMeshComponent->ComputeMinLOD();
    int32 MaxBonesPerChunk = SkelMeshRenderData->GetMaxBonesPerSection(MinLODIndex);
    int32 MaxSupportedNumBones = SkeletalMeshComponent->MeshObject->IsCPUSkinned() ? MAX_int32 : GetFeatureLevelMaxNumberOfBones(SceneFeatureLevel);
    if (MaxBonesPerChunk <= MaxSupportedNumBones)
    {
        return new FTaggedSkeletalMeshSceneProxy(SkeletalMeshComponent, SkelMeshRenderData, TaggedMID);
    }
}

return nullptr; }

[paku2k]

Unfortunately I have trouble reproducing it myself right now. It seemed to happen sometimes when destroying an actor and reloading a map. But I am not sure about the exact circumstances.

I have run my script using UE4Editor without the -game tag and there it seems to run more stable - no crashes occured yet. Also when switching back to using the -game mode, it runs fine now. It seems to be a little bit random.

Nevertheless, I have added your code to the TaggedComponent.cpp - Only the line 140 changed as marked in the screenshot below, correct?

I will see if the problem reappears again and come back. Thanks for the help so far!

[Blyron]

How are you running CARLA? Using prebuilt package or using a make launch? It seems to be a memory corrumption issue related with proxys and memory not beeing freed properly, UE stuff.

Yeah only 140 is the change, the issue is that object being null later on

[GoodarzMehr]

I've been also experiencing this issue, though when using a packaged version of CARLA (0.9.14 and 0.9.15). In my case I run a lot (sometimes thousands) of scenarios consecutively where in each scenario I spawn several vehicles with cameras and lidars attached to each (you can think of it as the multi-agent version of CARLA's integration with RLlib example), along with other background vehicles and pedestrians. I destroy everything at the end of each scenario and reset, but sometimes (usually after about 50ish iterations) my code segfaults when trying to destroy everything.

[paku2k]

How are you running CARLA? Using prebuilt package or using a make launch? It seems to be a memory corrumption issue related with proxys and memory not beeing freed properly, UE stuff.

Yeah only 140 is the change, the issue is that object being null later on

I am using a make launch, so the Unreal editor with the carla project. @GoodarzMehr Okay so it is a pretty similar scenario for you, I also create and destroy actors and sensors frequently.

I wanted to make a new run tonight, but this morning I saw carla crashed after like 80 iterations with the following error, which is different from the one above:

` [2024.07.04-16.59.37:485][992]LogCore: Error: appError called: Assertion failed: AllTireConfigs[TireConfigID] == this [File:/data/UnrealEngine_4.26/Engine/Plugins/Runtime/PhysXVehicles/Source/PhysXVehicles/Private/TireConfig.cpp] [Line: 91]

[2024.07.04-16.59.37:558][992]LogCore: === Critical error: === Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000003

[2024.07.04-16.59.37:558][992]LogCore: Assertion failed: AllTireConfigs[TireConfigID] == this [File:/data/UnrealEngine_4.26/Engine/Plugins/Runtime/PhysXVehicles/Source/PhysXVehicles/Private/TireConfig.cpp] [Line: 91]

`

The log can be found here CarlaUE4.log

[Blyron]

It seems like the GC is not working properly. Very random issues.

[PatrickPromitzer]

I noticed something with the Carla Python package, but I am not sure if Carla was the problem.

If you start the Python script once and play multiple scenarios, at some point the RAM is full or the script doesn't work anymore. Restarting the Carla server will not help.

It works it you create a bash script and use a combination of IDs and and a python list

#!/bin/bash
script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd "${script_dir}" || exit
echo $(pwd)
CARLA_SCRIPT="carla_script.py"
PYTON_VENV="venv/bin/python"
for i in {0..2}
do
  echo "${PYTON_VENV} ${CARLA_SCRIPT} ${i}"
  ${PYTON_VENV} ${CARLA_SCRIPT} ${i}
done

import sys

if __name__ == '__main__':
    args = sys.argv
    scenario_number = int(args[1])
    scenarios_to_test = [ 
        "data_1",
        "data_2",
        "data_3"
    ]

[GoodarzMehr]

Try to apply this piece of code in Unreal\CarlaUE4\Plugins\Carla\Source\Carla\Game\TaggedComponent.cpp Line 129 `` FPrimitiveSceneProxy UTaggedComponent::CreateSceneProxy(USkeletalMeshComponent SkeletalMeshComponent) { if (bShouldWaitFrame) { return nullptr; } ERHIFeatureLevel::Type SceneFeatureLevel = GetWorld()->FeatureLevel; FSkeletalMeshRenderData* SkelMeshRenderData = SkeletalMeshComponent->GetSkeletalMeshRenderData();

// Only create a scene proxy for rendering if properly initialized
if (SkelMeshRenderData &&
  SkelMeshRenderData->LODRenderData.IsValidIndex(SkeletalMeshComponent->PredictedLODLevel) &&
  !SkeletalMeshComponent->bHideSkin &&
  IsValid(SkeletalMeshComponent->MeshObject))
{
  // Only create a scene proxy if the bone count being used is supported, or if we don't have a skeleton (this is the case with destructibles)
  int32 MinLODIndex = SkeletalMeshComponent->ComputeMinLOD();
  int32 MaxBonesPerChunk = SkelMeshRenderData->GetMaxBonesPerSection(MinLODIndex);
  int32 MaxSupportedNumBones = SkeletalMeshComponent->MeshObject->IsCPUSkinned() ? MAX_int32 : GetFeatureLevelMaxNumberOfBones(SceneFeatureLevel);
  if (MaxBonesPerChunk <= MaxSupportedNumBones)
  {
      return new FTaggedSkeletalMeshSceneProxy(SkeletalMeshComponent, SkelMeshRenderData, TaggedMID);
  }
}

return nullptr; }

Hi. When making this change I get the following error:

In file included from /home/goodi/Programs/CARLA/carla/Unreal/CarlaUE4/Plugins/Carla/Intermediate/Build/Linux/B4D820EA/UE4Editor/Development/Carla/Module.Carla.1_of_4.cpp:27:
/home/goodi/Programs/CARLA/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Game/TaggedComponent.cpp:142:3: error: no matching function for call to 'IsValid'
                IsValid(SkeletalMeshComponent->MeshObject))
                ^~~~~~~
/home/goodi/Programs/UnrealEngine_4.26/Engine/Source/Runtime/CoreUObject/Public/UObject/Object.h:1543:18: note: candidate function not viable: no known conversion from 'class FSkeletalMeshObject *' to 'const UObject *' for 1st argument
FORCEINLINE bool IsValid(const UObject *Test)

What am I missing?

[Blyron]

You are not missing nothing. I wrote wrongly the code. But I have checked that what I have purpossed is not a solution. So I guess we are within a problem in UE