Crashing at seemingly random points on Linux

[J-The-Fox]

As the title implies, Fly Dangerous will crash at random points throughout the game. It will crash when starting a timed trial, flying through a checkpoint, customizing the ship or just flying around in a timed trial. It however does not crash while in Free Roam mode.

It could be that Fly Dangerous or Unity aren't playing nicely with my graphics card (Intel Arc A770 LE) or there is something else that might be going on. However I am not sure on the exact issue that is causing the crash. I have only noticed it on Linux as Windows has no issues running it on the same computer. So at least from my testing so far, it's only a Linux side issue

[jukibom]

Interesting, thanks for this! I think this is the first report using an Arc card, especially on Linux... Is there anything of note in the log file from a crash? On Linux that should be at ~/.config/unity3d/StarGoat/FlyDangerous/Player.log.

[J-The-Fox]

There is a crash report that states the following message
From ~/.config/unity3d/StarGoat/FlyDangerous/Player.log

=================================================================
    Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
    Native stacktrace:
=================================================================
    0x7f7802114fee - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/flydangerous_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
    0x7f78020bdcc9 - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/flydangerous_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
    0x7f7802043310 - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/flydangerous_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
    0x7f780c562ab0 - /usr/lib/libc.so.6 : 
    0x7f780c68f946 - /usr/lib/libc.so.6 : 
    0x7f780d6d104b - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/UnityPlayer.so : 
    0x7f780de1188e - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/UnityPlayer.so : 
    0x7f780de12eb0 - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/UnityPlayer.so : 
    0x7f780de0e1df - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/UnityPlayer.so : 
    0x7f780de0df05 - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/UnityPlayer.so : 
    0x7f780d4e1d69 - /home/<my_user>/.local/share/Steam/steamapps/common/Fly Dangerous/UnityPlayer.so : 
    0x7f780c5b044b - /usr/lib/libc.so.6 : 
    0x7f780c633e40 - /usr/lib/libc.so.6 : 

=================================================================
    Telemetry Dumper:
=================================================================
Thread 0x7f7726ffd6c0 may have been prematurely finalized* Assertion at mono-threads.c:702, condition `info' not met, function:mono_thread_info_current, 
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (3243) : Trying to close low level socket support, but we still have sockets open!
Installing breakpad exception handler for appid(flydangerous)/version(1.0)/tid(9260)
ERROR: ld.so: object '/home/<my_user>/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
assert_20230531203254_3.dmp[9338]: Uploading dump (out-of-process)
/tmp/dumps/assert_20230531203254_3.dmp
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (3243) : Trying to close low level socket support, but we still have sockets open!
assert_20230531203254_3.dmp[9338]: Finished uploading minidump (out-of-process): success = yes
assert_20230531203254_3.dmp[9338]: response: Discarded=1
assert_20230531203254_3.dmp[9338]: file ''/tmp/dumps/assert_20230531203254_3.dmp'', upload yes: ''Discarded=1''
pid 9338 != 9337, skipping destruction (fork without exec?)

Possibly a Mono issue?

[jukibom]

Huh, that's very strange... I'm not sure what on earth steam is doing under the hood there if you're not in multiplayer. It does spin up an offline server for single player to keep the codebase simple so it may be related to that but it shouldn't create a listen server nor any kind of steam player lobby as it's not needed.

I'm wondering if that stack trace is a consequence of an early termination (I can see a few people reporting that specific error on application exit, specifically - possibly without calling steam api shutdown?) and may be a red herring.

It's strange that this happens when doing things like hitting checkpoints or just flying around in time trial, I'm struggling to think what the relevance there could even be that greatly differs from free roam. The only thing that comes to mind is replay recording which is a pretty hairy operation as it streams binary data to a file each tick. Is there anything in the log related to ReplayRecorder.cs by any chance?

[J-The-Fox]

I am also not quite sure what sockets its using and listening to as I haven't used the multiplayer side in Fly Dangerous. It's very possible it's something to do with steam itself. At least on the listening end.

It could be a bit of a red herring as there are a few lines in the crash report that point to a thread that may have terminated early such as

Thread 0x7f7726ffd6c0 may have been prematurely finalized* Assertion at mono-threads.c:702, condition `info' not met, function:mono_thread_info_current, 

I did look through both ~/.config/unity3d/StarGoat/FlyDangerous/Player.log and ~/.config/unity3d/StarGoat/FlyDangerous/Player-prev.log for any sign of ReplayRecorder but could not find anything in relation to that sadly.

I did however try to do some simple analysis on the dump file located at /tmp/dumps/assert_20230531203254_3.dmp and got a few interesting things to note. A few of the lines hold interesting data

COMMENT:  Assert( Assertion Failed: reaping pid: 39869 -- gameoverlayui):/data/src/steamexe/main.cpp:255

FAILURE_BUCKET_ID:  APPLICATION_FAULT_ffffffff_steam!unknown_error_in_process

Seems like steam may be having an issue? Still trying to wrap my head around this one
Since I was missing a few of the symbols needed I couldn't get a full analysis but it might help try to pinpoint the cause of the error

[jukibom]

Oh that's interesting, I wonder if disabling the steam overlay would help? Not sure why on earth free roam mode would be unaffected though 😅

[J-The-Fox]

It seemed to go on a bit longer and got through more timed trials but it still eventually crashed. Looking at the log files, it is still the same error. Not sure what the dump file has to say though. It is quite strange that it is only during timed trails and not during free roam though

[jukibom]

Yeah, I especially can't think of anything GPU-related (as this appears to be Arc specific, at least on the surface) unless it's something to do with the shader pipeline for checkpoints / billboards / boost portals.

Here's a way to narrow it down, potentially...

Try loading some maps (below) with the custom level panel which are the inverse of each (time trial with no objects, free roam with time trial objects) and see which one fails.

To do this, copy and paste into this element here in the single player menu

---

This is level JSON for an empty (no checkpoints or anything apart from level geo) time trial:

{
  "name": "Time Trial Test",
  "author": "Juki",
  "gameType": "Sprint",
  "environment": "Blue Planet",
  "location": "Space Station",
  "musicTrack": "Hooligans",
  "authorTimeTarget": 60.0,
  "startPosition": {},
  "startRotation": {}
}

This is level JSON for a free roam map with time trial objects loaded in:

{
  "name": "Free Roam Test",
  "author": "Juki",
  "gameType": "Free Roam",
  "environment": "Blue Planet",
  "location": "Space Station",
  "musicTrack": "Beautiful Catastrophe",
  "authorTimeTarget": 25.885,
  "startPosition": {
    "x": 1677.77441,
    "y": 1299.00122,
    "z": -5141.518
  },
  "startRotation": {},
  "checkpoints": [
    {
      "position": {
        "x": 1677.77441,
        "y": 1299.00122,
        "z": -5141.518
      },
      "rotation": {}
    },
    {
      "position": {
        "x": -252.0,
        "y": 982.0,
        "z": 412.0
      },
      "rotation": {},
      "type": 1
    },
    {
      "position": {
        "x": 1027.0,
        "y": 1616.0,
        "z": 4896.0
      },
      "rotation": {
        "y": 50.5142975
      },
      "type": 1
    },
    {
      "position": {
        "x": 2442.0,
        "y": 1671.0,
        "z": 5834.0
      },
      "rotation": {
        "y": 90.0
      },
      "type": 1
    },
    {
      "position": {
        "x": 3946.0,
        "y": 2225.0,
        "z": 4669.0
      },
      "rotation": {
        "y": 134.999939
      },
      "type": 1
    },
    {
      "position": {
        "x": 4181.0,
        "y": 1120.0,
        "z": 1306.0
      },
      "rotation": {},
      "type": 1
    },
    {
      "position": {
        "x": 4181.0,
        "y": 672.0012,
        "z": -1845.0
      },
      "rotation": {},
      "type": 1
    },
    {
      "position": {
        "x": 3274.0,
        "y": 1299.00122,
        "z": -5141.518
      },
      "rotation": {},
      "type": 2
    }
  ],
  "billboards": [
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 6663.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 5051.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 3452.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 1846.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 5051.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 6663.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 3452.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 1846.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    }
  ]
}

Thank you so much for your help with this!

[J-The-Fox]

It could very well be something Arc specific. It's still a pretty new card and although a lot of the main bugs have been worked out I would not at all be surprised if some still linger for a bit until drivers get updated and such.

After doing some testing with those two custom maps. I was in fact able to get it to crash (with the same error in ~/.config/unity3d/StarGoat/FlyDangerous/Player.log during the free roam map with the time trail objects loaded. During the time trail map with no objects gave me no crash after ~30 minutes of testing with reloading the map, restarting the game, etc. So it seems like it points to one of the objects being at fault? Seems like it narrowed it down though!

Always happy to help!

[jukibom]

That extremely helpful, thank you! Yeah that absolutely points to something in the shader graph pipeline for the billboards or checkpoints (boosters weren't in that map). Unfortunately those are kinda black boxes from my perspective 🙃

If I can throw together a minimal test case would you be willing to test that so I could submit a bug to Unity / Intel?

Also, we could narrow it down to checkpoints or billboards:

---

Checkpoints:

{
  "name": "Checkpoint Test",
  "author": "Juki",
  "gameType": "Free Roam",
  "environment": "Blue Planet",
  "location": "Space Station",
  "musicTrack": "Beautiful Catastrophe",
  "authorTimeTarget": 25.885,
  "startPosition": {
    "x": 1677.77441,
    "y": 1299.00122,
    "z": -5141.518
  },
  "startRotation": {},
  "checkpoints": [
    {
      "position": {
        "x": 1677.77441,
        "y": 1299.00122,
        "z": -5141.518
      },
      "rotation": {}
    },
    {
      "position": {
        "x": -252.0,
        "y": 982.0,
        "z": 412.0
      },
      "rotation": {},
      "type": 1
    },
    {
      "position": {
        "x": 1027.0,
        "y": 1616.0,
        "z": 4896.0
      },
      "rotation": {
        "y": 50.5142975
      },
      "type": 1
    },
    {
      "position": {
        "x": 2442.0,
        "y": 1671.0,
        "z": 5834.0
      },
      "rotation": {
        "y": 90.0
      },
      "type": 1
    },
    {
      "position": {
        "x": 3946.0,
        "y": 2225.0,
        "z": 4669.0
      },
      "rotation": {
        "y": 134.999939
      },
      "type": 1
    },
    {
      "position": {
        "x": 4181.0,
        "y": 1120.0,
        "z": 1306.0
      },
      "rotation": {},
      "type": 1
    },
    {
      "position": {
        "x": 4181.0,
        "y": 672.0012,
        "z": -1845.0
      },
      "rotation": {},
      "type": 1
    },
    {
      "position": {
        "x": 3274.0,
        "y": 1299.00122,
        "z": -5141.518
      },
      "rotation": {},
      "type": 2
    }
  ],
  "billboards": []
}

Billboards:

{
  "name": "Billboard Test",
  "author": "Juki",
  "gameType": "Free Roam",
  "environment": "Blue Planet",
  "location": "Space Station",
  "musicTrack": "Beautiful Catastrophe",
  "authorTimeTarget": 25.885,
  "startPosition": {
    "x": 1677.77441,
    "y": 1299.00122,
    "z": -5141.518
  },
  "startRotation": {},
  "checkpoints": [],
  "billboards": [
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 6663.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 5051.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 3452.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": 6041.0,
        "y": 1643.0,
        "z": 1846.0
      },
      "rotation": {
        "y": 270.000061
      },
      "type": "Newtons Gambit",
      "scrollSpeedOverride": 0.2
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 5051.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 6663.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 3452.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    },
    {
      "position": {
        "x": -1087.0,
        "y": 1643.0,
        "z": 1846.0
      },
      "rotation": {
        "y": 90.00012
      },
      "type": "Fly Dangerous"
    }
  ]
}

[J-The-Fox]

Have been testing it for a bit and only got 2 total crashes. Once was in the checkpoint map and the other was during the Timed Trial select screen. Didn't crash at all with the billboard only map. The errors appeared the same on both instances of the crash. Didn't crash as often as I would have liked as I would like a little more data than just 2 crashes during this run but at least it did give some data.

I would be more than willing to run a small test case to see if anything else can be found out and if something can be submitted to either Unity or Intel. There's also the crash dumps as well as the Player.log that could also be for testing.

Hopefully that helps a bit!

[jukibom]

Interesting, the way it appeared on both instances of the map and the time trial map select suggests a much deeper GPU driver issue to do with probably VRAM allocation or something really basic (thinking allocating space for the thumbnails, which are loaded in dynamically) because there's really very little special going on with the map select screen. I'll see if I can find some time to throw something together...

[J-The-Fox]

I wouldn't be surprised if something along the lines of driver issue causing unwanted behavior during VRAM allocation could be the culprit. All of the drivers should be installed for my card including the 32-bit libraries Steam requires bedsides hardware video acceleration as I don't think my Linux Distro has the package available yet for Arc. (Could be in the testing repo though). Although I don't think hardware video acceleration would be of any benefit here. I could look for an updated Mesa or Intel driver package on the testing repo to see if that may help. It is a little odd that it crashed on the Timed Trial select screen as like you said, not much going on there.

Sounds good!