[3.5.2] Crash in RasterizerStorageGLES3::sky_set_texture only on Mac mini M1

starryalley commented 1 year ago

Godot version

3.5.stable

System information

Mac mini M1 (running OS 13.5.2)

Issue description

In my app user can specfiy which panorama sky texture (.hdr) to use. After downloading the hdr and loading the hdr to the sky texture, godot will sometimes crash (at least once per 10 runs) on a Mac mini M1 (2020 version with 8GB ram) right at the part where it tries to set the loaded texture to the panorama sky.

Note that this issue only happens with GLES3.0.

First noticed this issue with godot 3.4.stable and can reproduce reliably with 3.5.2.

The Environment background mode is BG_COLOR_SKY, and the sky is a PanoramaSky. Code below will sometimes crash:

var img = Image.new()
img.load(hdr_path)
var tex = ImageTexture.new()
tex.create_from_image(img)
$Camera.environment.background_sky.panorama = tex

However it never crashes on my other macbook pro M1 (14" 2021) running the same OS version: 13.5.2 (22G91). I've added a test loop to load/unload the panorama texture and it won't crash after 500 load/unload cycles. On my Mac mini M1 however it usually crashes within 100 cycles.

The crash log looks a bit different but always from RasterizerStorageGLES3::sky_set_texture():

Thread 0::  Dispatch queue: com.apple.main-thread
0   libobjc.A.dylib                     0x7ff8131ca740 objc_msgSend + 0
1   AppleMetalOpenGLRenderer               0x11b7574c2 GLDContextRec::buildRenderPassDescriptor() + 1246
2   AppleMetalOpenGLRenderer               0x11b7577d0 GLDContextRec::beginRenderPass() + 36
3   AppleMetalOpenGLRenderer               0x11b77170d gldRenderVertexArray(GLDContextRec*, unsigned int, unsigned int, int, int, unsigned int, void const*, int, void const*) + 490
4   GLEngine                            0x7ffa2b596b53 glDrawArrays_GL3Exec + 251
5   godot.osx.opt.tools.x86_64             0x103690ea5 RasterizerStorageGLES3::sky_set_texture(RID, RID, int) + 3701 (rasterizer_storage_gles3.cpp:1919)
6   godot.osx.opt.tools.x86_64             0x104c56685 MethodBind1<Ref<Texture> const&>::call(Object*, Variant const**, int, Variant::CallError&) + 277 (method_bind.gen.inc:759)
7   godot.osx.opt.tools.x86_64             0x10532989c ClassDB::set_property(Object*, StringName const&, Variant const&, bool*) + 524 (class_db.cpp:1010)
8   godot.osx.opt.tools.x86_64             0x1053b0ae6 Object::set(StringName const&, Variant const&, bool*) + 102 (object.cpp:422)
9   godot.osx.opt.tools.x86_64             0x1054f2443 Variant::set_named(StringName const&, Variant const&, bool*) + 1107 (variant_op.cpp:1540)
10  godot.osx.opt.tools.x86_64             0x102ed6be0 GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) + 31280 (gdscript_function.cpp:613)
11  godot.osx.opt.tools.x86_64             0x102e90c5c GDScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) + 156 (gdscript.cpp:1196)
...

Another:

Thread 0::  Dispatch queue: com.apple.main-thread
0   AppleMetalOpenGLRenderer               0x119981346 GLDSamplerRec::initWithShareGroup(GLDShareGroupRec*, GLDSamplerStateRec const*, GLDPluginSamplerStateRec*) + 36
1   AppleMetalOpenGLRenderer               0x11998209e gldCreateSampler + 74
2   libGFXShared.dylib                  0x7ffa2b2e0d86 gfxCreatePluginTexture + 138
3   GLEngine                            0x7ffa2b4ecef8 gleCreateTextureObject + 249
4   GLEngine                            0x7ffa2b4ff956 gleBindTexture + 179
5   GLEngine                            0x7ffa2b4ff866 glBindTexture_Exec + 405
6   godot.osx.opt.tools.x86_64             0x1018c3a1f RasterizerStorageGLES3::sky_set_texture(RID, RID, int) + 2543 (rasterizer_storage_gles3.cpp:1846)
7   godot.osx.opt.tools.x86_64             0x102e89685 MethodBind1<Ref<Texture> const&>::call(Object*, Variant const**, int, Variant::CallError&) + 277 (method_bind.gen.inc:759)
8   godot.osx.opt.tools.x86_64             0x10355c89c ClassDB::set_property(Object*, StringName const&, Variant const&, bool*) + 524 (class_db.cpp:1010)
9   godot.osx.opt.tools.x86_64             0x1035e3ae6 Object::set(StringName const&, Variant const&, bool*) + 102 (object.cpp:422)
10  godot.osx.opt.tools.x86_64             0x103725443 Variant::set_named(StringName const&, Variant const&, bool*) + 1107 (variant_op.cpp:1540)
11  godot.osx.opt.tools.x86_64             0x101109be0 GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) + 31280 (gdscript_function.cpp:613)
...

Crash logs generated by the OS are attached too. crash_log3.txt crash_log2.txt crash_log1.txt

Steps to reproduce

Run the min reproduction project below. It will crash within 5-10sec on a Mac mini M1 with GLES3.0. By setting the project to run with GLES2.0 it will not crash.

Minimal reproduction project

sky_crash_3.zip

starryalley commented 1 year ago

Update: I've tested more with the Mac mini M1 machine and with the min reproduction project attached above, it will always crash in the second cycle of loading texture to the sky.

===> doing load/unload of panorama texture:0
Load panorama sky from:res://assets/nancy_church_1.hdr
2023-09-11 15:29:33.434 godot.osx.opt.tools.x86_64[13029:183538] GLDRendererMetal command buffer completion error: Error Domain=MTLCommandBufferErrorDomain Code=2 "Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang)" UserInfo={NSLocalizedDescription=Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang)}
Clear sky panorama texture
===> doing load/unload of panorama texture:1
Load panorama sky from:res://assets/nancy_church_1.hdr
2023-09-11 15:29:34.512 godot.osx.opt.tools.x86_64[13029:183538] GLDRendererMetal command buffer completion error: Error Domain=MTLCommandBufferErrorDomain Code=2 "Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang)" UserInfo={NSLocalizedDescription=Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang)}
2023-09-11 15:29:34.512 godot.osx.opt.tools.x86_64[13029:183550] GLDRendererMetal command buffer completion error: Error Domain=MTLCommandBufferErrorDomain Code=4 "Ignored (for causing prior/excessive GPU errors) (00000004:kIOGPUCommandBufferCallbackErrorSubmissionsIgnored)" UserInfo={NSLocalizedDescription=Ignored (for causing prior/excessive GPU errors) (00000004:kIOGPUCommandBufferCallbackErrorSubmissionsIgnored)}

With my real world app I usually got a crash within 10 cycles.

starryalley commented 1 year ago

I just did the same test on Godot 4.1.1 with the same min reproduction project above, on the mac mini M1. I have to slightly update the gdscript obviously because the property names are a bit different. It turns out 4.1.1 isn't a problem at all. Code below for godot 4:

func _ready():
    for i in range(100):
        print("===> doing load/unload of panorama texture:%d" % i)
        _setSkyPanoramaTexture("res://assets/panorama.hdr")
        _clearSkyPanoramaTexture()

func _setSkyPanoramaTexture(path: String) -> void:
    print("Load panorama sky from:%s" % path)
    var img = Image.new()
    img.load(path)
    var tex = ImageTexture.create_from_image(img)
    $WorldEnvironment.environment.sky.sky_material.panorama = tex

func _clearSkyPanoramaTexture() -> void:
    print("Clear sky panorama texture")
    $WorldEnvironment.environment.sky.sky_material.panorama = null

So this issue is specific to Godot 3. (reproducible on 3.4.x and 3.5.x. Older version not tested)

godotengine / godot