Images imported as lossless WebP crash release builds on some machines

lekoder commented 3 years ago

Godot version

3.4.stable.official, v3.5.beta.custom_build.67f19ee51

System information

Windows 8 Pro 64-bit (6.2, Build 9200), Intel(R) Core(TM) i5-3570K, AMD Radeon HD 7800 Series, GLES3

Issue description

One player reported via Steam forums application not responding during startup.

I was unable to replicate this on any of my machines. Got in contact with the player and managed to bisect a specific version of game which ceased to work, and checked the difference between working and not-working PCK files. Only relevant changes were .stex files:

Contents of '0.426.0-working/Delta-V.pck':                                                                                            | Contents of '0.426.1-broken/Delta-V.pck':
res://.import/dealer-cothon-m.png-0fcf007d7f7974b61b5d2267f4da5fa4.s3tc.stex size: 1327124                                            | res://.import/dealer-cothon-m.png-0fcf007d7f7974b61b5d2267f4da5fa4.s3tc.stex size: 8294420
res://.import/dealer-cothon-n.png-cc8cc62de14a091e1c5ce55a2035b9d0.s3tc.stex size: 8294420                                            | res://.import/dealer-cothon-n.png-cc8cc62de14a091e1c5ce55a2035b9d0.s3tc.stex size: 5530332
res://.import/dealer-eime-m.png-fdb35a3b7f5a42b5c741c4df4a7da007.s3tc.stex size: 1327124                                              | res://.import/dealer-eime-m.png-fdb35a3b7f5a42b5c741c4df4a7da007.s3tc.stex size: 8294420
res://.import/dealer-eime-n.png-0a054f30b44c22b06ac2f7dc9a3b4ecb.s3tc.stex size: 8294420                                              | res://.import/dealer-eime-n.png-0a054f30b44c22b06ac2f7dc9a3b4ecb.s3tc.stex size: 5530332
res://.import/dealer-k37-m.png-3eccf5082060f7d4ca1a2881d486021b.s3tc.stex size: 1327124                                               | res://.import/dealer-k37-m.png-3eccf5082060f7d4ca1a2881d486021b.s3tc.stex size: 8294420
res://.import/dealer-k37-n.png-f270e77360a3a9a57f52750476eeed72.s3tc.stex size: 8294420                                               | res://.import/dealer-k37-n.png-f270e77360a3a9a57f52750476eeed72.s3tc.stex size: 5530332
res://.import/dealer-prospector-m.png-eff7096a616f4318d457223019da4f01.s3tc.stex size: 1327124                                        | res://.import/dealer-prospector-m.png-eff7096a616f4318d457223019da4f01.s3tc.stex size: 8294420
res://.import/dealer-prospector-n.png-f8f1a3e530d5b7a4f8b3b300b0ee3e65.s3tc.stex size: 8294420                                        | res://.import/dealer-prospector-n.png-f8f1a3e530d5b7a4f8b3b300b0ee3e65.s3tc.stex size: 5530332
res://.import/enceladus-inner-m.png-db52c9985fe838e4dd13138f3970166d.s3tc.stex size: 1048596                                          | res://.import/enceladus-inner-m.png-db52c9985fe838e4dd13138f3970166d.s3tc.stex size: 4194324
res://.import/gammascale.png-e752fcc2cbbc3d68f3236aea2429338e.stex size: 2560                                                         | res://.import/gammascale.png-e752fcc2cbbc3d68f3236aea2429338e.stex size: 1210
res://.import/huge-hollow-rock-n.png-27cb29ac838b43555685c1e55e837f80.s3tc.stex size: 11184868                                        | res://.import/huge-hollow-rock-n.png-27cb29ac838b43555685c1e55e837f80.s3tc.stex size: 5592444
res://.import/irrs-n.png-4f4905367a8dd27df984132075f60be3.s3tc.stex size: 5592452                                                     | res://.import/irrs-n.png-4f4905367a8dd27df984132075f60be3.s3tc.stex size: 2796236
res://.import/nival-n.png-66ceab7c3a267cfbf05035ecd6348c02.s3tc.stex size: 1387124                                                    | res://.import/nival-n.png-66ceab7c3a267cfbf05035ecd6348c02.s3tc.stex size: 693572
res://.import/pistacja-n.png-ef99b52f1f193469c322647efba72414.s3tc.stex size: 17345012                                                | res://.import/pistacja-n.png-ef99b52f1f193469c322647efba72414.s3tc.stex size: 8672516
res://.import/showroom-m.png-2350f6d7956f8b52e4c901943ae10f5c.s3tc.stex size: 262164                                                  | res://.import/showroom-m.png-2350f6d7956f8b52e4c901943ae10f5c.s3tc.stex size: 3686420
res://.import/showroom-n.png-0a99f79a4e5fef35b192d89ab0e6fe34.s3tc.stex size: 1048596                                                 | res://.import/showroom-n.png-0a99f79a4e5fef35b192d89ab0e6fe34.s3tc.stex size: 4915396
res://.import/starfield2.png-6a73c44d1346abf6200fbc15535ecbf1.stex size: 6511913                                                      | res://.import/starfield2.png-6a73c44d1346abf6200fbc15535ecbf1.stex size: 4512386
res://backgrounds/starfield2.png.import size: 669                                                                                     | res://backgrounds/starfield2.png.import size: 703
res://menu/gammascale.png.import size: 663                                                                                            | res://menu/gammascale.png.import size: 697
res://ships/drone/ATLAS.tscn.converted.res size: 9140                                                                                 | res://ships/drone/ATLAS.tscn.converted.res size: 9132
res://ships/drone/ClaimBeacon.tscn.converted.res size: 7894                                                                           | res://ships/drone/ClaimBeacon.tscn.converted.res size: 7900
res://ships/drone/Drone.tscn.converted.res size: 9083                                                                                 | res://ships/drone/Drone.tscn.converted.res size: 9089
res://tests/TestStory.gdc size: 5038                                                                                                  | res://tests/TestStory.gdc size: 4974
res://tests/TestStory.tscn.converted.res size: 50944                                                                                  | res://tests/TestStory.tscn.converted.res size: 50946

I removed .import directory from last known working version and re-imported everything. It crashed.
I enabled force png in project settings, removed .import and re-imported everything. This worked.
Switched to master branch of my game and exported with force png - also worked.

In case of crash logs are cut off without any error report. This happens both for release builds and for debug builds. Here is the debug build output (run with --verbose --debug):

Godot Engine v3.5.beta.custom_build.67f19ee51 - https://godotengine.org
VisualServerWrapMT: Creating render thread
VisualServerWrapMT: Starting render thread
Using GLES3 video driver
OpenGL ES 3.0 Renderer: AMD Radeon HD 7800 Series
Async. shader compilation: OFF
OpenGL ES Batching: ON
    OPTIONS
    max_join_item_commands 16
    colored_vertex_format_threshold 0.25
    batch_buffer_size 65535
    light_scissor_area_threshold 0.2401
    item_reordering_lookahead 4
    light_max_join_items 32
    single_rect_fallback False
    debug_flash False
    diagnose_frame False
VisualServerWrapMT: Finished render thread
WASAPI: wFormatTag = 65534
WASAPI: nChannels = 2
WASAPI: nSamplesPerSec = 44100
WASAPI: nAvgBytesPerSec = 352800
WASAPI: nBlockAlign = 8
WASAPI: wBitsPerSample = 32
WASAPI: cbSize = 22
WASAPI: detected 2 channels
WASAPI: audio buffer frames: 896 calculated latency: 20ms

Loading resource: res://translations.en.translation
Loading resource: res://translations.pl.translation
Loading resource: res://translations.zh_HK.translation
[...]
Loading resource: res://AsteroidSpawner.gdc
Loading resource: res://asteroids/class-11.tscn.converted.res
Loading resource: res://asteroids/ice.phymat

Steps to reproduce

I can't reproduce it locally - the only way I was able to debug is to send builds via Steam to the player and await feedback. Here is what I gathered about the machine that was used to run:

   Operating System: Windows 8 Pro 64-bit (6.2, Build 9200) (9200.win8_gdr.151230-0600)
           Language: English (Regional Setting: English)
          Processor: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz (4 CPUs), ~3.4GHz
             Memory: 16384MB RAM
Available OS Memory: 16326MB RAM

          Card name: AMD Radeon HD 7800 Series
       Manufacturer: Advanced Micro Devices, Inc.
          Chip type: AMD Radeon Graphics Processor (0x679E)
           DAC type: Internal DAC(400MHz)
        Device Type: Full Device
         Device Key: Enum\PCI\VEN_1002&DEV_679E&SUBSYS_23281787&REV_00
     Display Memory: 9928 MB
   Dedicated Memory: 2022 MB
      Shared Memory: 7906 MB

        Output Type: HDMI
        Driver Name: aticfx64.dll,aticfx64.dll,aticfx64.dll,aticfx32,aticfx32,aticfx32,atiumd64.dll,atidxx64.dll,atidxx64.dll,atiumdag,atidxx32,atidxx32,atiumdva,atiumd6a.cap,atitmm64.dll
Driver File Version: 8.17.0010.1280 (English)
     Driver Version: 14.100.0.0
        DDI Version: 11.1
     Feature Levels: 11.1,11.0,10.1,10.0,9.3,9.2,9.1
       Driver Model: WDDM 1.2
Graphics Preemption: DMA
 Compute Preemption: DMA
  Driver Attributes: Final Retail

Minimal reproduction project

No MRP due to not even crashing on my own machine.

I know that version 0.427.2 (~~stable branch~~ godot-debug branch, branch password: godotdebugbranch) and 0.427.7 (beta branch) currently crash the affected systems and 0.428.1 (experimental branch) does not crash them. This affects both full builds and free demos. Game link. We tested both a GodotSteam build, an official Godot binary and GodotGog build with same results.

akien-mga commented 3 years ago

CC @godotengine/import @mortarroad - this is going to be fun to debug :|

@lekoder In "v3.5.beta.custom_build.67f19ee51", the hash doesn't seem to match an upstream commit, I guess it's from your fork? What's the matching upstream commit you rebased on? In particular, does it include the recently merged 724c207005b0784144d78a854a88302343864ff6 (just to see if that could have fixed an upstream libwebp bug).

If the user is still willing to assist in debugging, it would be good to figure out a way to actually get a stacktrace. Official builds should create a stacktrace, albeit an unhelpful one due to stripping debug symbols. But a debug build should have a stacktrace with symbols even on Windows, but maybe not if you compiled with MSVC?

lekoder commented 3 years ago

@lekoder In "v3.5.beta.custom_build.67f19ee51", the hash doesn't seem to match an upstream commit, I guess it's from your fork? What's the matching upstream commit you rebased on? In particular, does it include the recently merged 724c207 (just to see if that could have fixed an upstream libwebp bug).

No it's from my own fork and it does not include [724c207]. User agreed to assist in further testing, so I'm going to rebuild upstream 3.x and check if [724c207] helps.

lekoder commented 3 years ago

Unfortunately, it still fails with 888f8cea9, which seems to contain 724c207.

lekoder commented 3 years ago

If the user is still willing to assist in debugging, it would be good to figure out a way to actually get a stacktrace. Official builds should create a stacktrace, albeit an unhelpful one due to stripping debug symbols. But a debug build should have a stacktrace with symbols even on Windows, but maybe not if you compiled with MSVC?

I usually get stacktraces from the debug builds I make, so this is probably not a MSVC preventing it. In this case it did not output anything at all.

Calinou commented 3 years ago

Crash backtraces are currently not written to log files. This means that for Windows release builds (which can't output anything to standard output), you must use a debugger such as Visual Studio, WinDbg (MSVC .pdb symbols) or gdb (MinGW symbols) to get a backtrace.

lekoder commented 3 years ago

This means that for Windows release builds (which can't output anything to standard output), you must use a debugger such as Visual Studio, WinDbg (MSVC .pdb symbols) or gdb (MinGW symbols) to get a backtrace.

Sadly this means we won't be getting stack trace for this particular problem - while I can ask player to switch to a specific Steam branch, I don't think it's reasonable to expect him to install and connect a debugger.

qarmin commented 3 years ago

Can you provide example webp files?

akien-mga commented 3 years ago

Crash backtraces are currently not written to log files. This means that for Windows release builds (which can't output anything to standard output), you must use a debugger such as Visual Studio, WinDbg (MSVC .pdb symbols) or gdb (MinGW symbols) to get a backtrace.

It seems expected to me that the crash backtrace isn't in the log written by Godot (Godot crashed so it can't write a log), but shouldn't it be written to stdout (or stderr, not sure)? If so I think it might be possible to get it by redirecting output to a file, not sure what the syntax would be exactly (maybe @bruvzg would know). On Steam it could be done with something like %command% > C:\path\to\file.txt 2>&1 or similar as custom command (to be confirmed by a Windows user).

lekoder commented 3 years ago

We checked that, this is what we got from stdout/err:

Loading resource: res://menu/KeepDialogCentered.gdc
Loading resource: res://enceladus/EnceladusIndustrialTop.tscn.converted.res
Loading resource: res://menu/delta-v-logo.png
Loading resource: res://demo/DemoLabel.gdc
Loading resource: res://music/Enceladus_Intro.ogg
Loading resource: res://SaveSlotButton.gdc
Loading resource: res://enceladus/MatchPitchToEngine.gdc
Loading
C:\Program Files (x86)\Steam\steamapps\common\dV Rings of Saturn>

Note that part of the last line is cut off.

lekoder commented 3 years ago

Can you provide example webp files?

@qarmin these are png files imported into the engine using webp compression. The source is a .png or .jpg. I can provide some, but I have no idea which one caused the problem, so this will be blind guess.

lekoder commented 3 years ago

I updated the branch information in the original issue since we had an upgrade of the stable branch to publish fix; there is a branch specifically made for this issue that points to a broken build, for both the demo and full version of the game.

bruvzg commented 3 years ago

%command% > C:\path\to\file.txt 2>&1 is a correct syntax, and should work on Windows when the app is running normally. But Windows flush the stream in the strange way, and it quite possible it won't catch the crash log.

Godot crashed so it can't write a log

Stack trace is printed by our code, for release builds we should be able to write it to the file instead of stderr. https://github.com/godotengine/godot/blob/ca707562387be0034d5384b6d4bf218c7da480ab/platform/windows/crash_handler_windows.cpp#L181-L212

godotengine / godot