search

godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
90.75k stars 21.12k forks source link

Hanging when importing PNG with etc/etc2 VRAM compression enabled [etc2comp bug] #51930

Open calebROlson opened 3 years ago

calebROlson commented 3 years ago

Godot version

3.4 beta 4 (6a058cbf3984131f94c8680970f3c184ddecc801)

System information

Windows 10

Issue description

Importing a png with import settings as follows leads to hang (using Qodot clip texture as an example):

[remap]

importer="texture"
type="StreamTexture"
path.s3tc="res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.s3tc.stex"
path.etc2="res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.etc2.stex"
metadata={
"imported_formats": [ "s3tc", "etc2" ],
"vram_texture": true
}

[deps]

source_file="res://addons/qodot/textures/special/clip.png"
dest_files=[ "res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.s3tc.stex", "res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.etc2.stex" ]

[params]

compress/mode=2
compress/lossy_quality=1.0
compress/hdr_mode=0
compress/bptc_ldr=0
compress/normal_map=0
flags/repeat=1
flags/filter=false
flags/mipmaps=true
flags/anisotropic=true
flags/srgb=1
process/fix_alpha_border=false
process/premult_alpha=false
process/HDR_as_SRGB=false
process/invert_color=false
process/normal_map_invert_y=false
stream=false
size_limit=0
detect_3d=false
svg/scale=1.0

I ran MSVC profiler which has the following info:

Function Name   Total CPU [unit, %] Self CPU [unit, %]  Module  Category
 + godot.windows.tools.64.exe (PID: 20356)  289492 (100.00%)    0 (0.00%)   Multiple modules    
| + [External Code] 289440 (99.98%) 645 (0.22%) Multiple modules    Other | IO | Networking | Driver | Graphics | File System | Kernel
|| + Concurrency::details::`anonymous namespace'::_Task_scheduler_callback  270934 (93.59%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||| + Concurrency::details::_DefaultPPLTaskScheduler::_PPLTaskChore::_Callback  270934 (93.59%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||| + Concurrency::details::_TaskProcHandle::_RunChoreBridge   270932 (93.59%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||| + Concurrency::details::_PPLTaskHandle<unsigned char,Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>,Concurrency::details::_TaskProcHandle>::invoke   262563 (90.70%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||| + Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>::_Perform  262563 (90.70%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||| + Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>::_Init    262563 (90.70%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||| + Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>::_LogWorkItemAndInvokeUserLambda<std::function<unsigned char __cdecl(void)> >    262560 (90.70%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||| + std::_Func_class<unsigned char>::operator() 262551 (90.69%) 2 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||| + std::_Func_impl_no_alloc<<lambda_052e919cc0e5399df76dff3972c0cac1>,unsigned char>::_Do_call    262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||| + std::_Invoker_ret<unsigned char,0>::_Call<<lambda_052e919cc0e5399df76dff3972c0cac1> &>    262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||| + std::invoke<<lambda_052e919cc0e5399df76dff3972c0cac1> &> 262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||| + std::_Invoker_functor::_Call<<lambda_052e919cc0e5399df76dff3972c0cac1> &>   262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||| + <lambda_052e919cc0e5399df76dff3972c0cac1>::operator()  262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||| + std::_Func_class<void>::operator()    262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||| + std::_Func_impl_no_alloc<<lambda_33c0edf7e2bf8e11477062206c55a36b>,void>::_Do_call   262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||| + std::_Invoker_ret<void,1>::_Call<<lambda_33c0edf7e2bf8e11477062206c55a36b> &>   262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||| + std::invoke<<lambda_33c0edf7e2bf8e11477062206c55a36b> &>   262549 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||| + std::_Invoker_functor::_Call<<lambda_33c0edf7e2bf8e11477062206c55a36b> &> 262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||| + <lambda_33c0edf7e2bf8e11477062206c55a36b>::operator()    262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||| + std::_Packaged_state<unsigned int __cdecl(void)>::_Call_immediate   262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||| + std::_Func_class<unsigned int>::operator() 262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||||| + std::_Func_impl_no_alloc<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &>,unsigned int>::_Do_call   262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||| + std::_Invoker_ret<unsigned int,0>::_Call<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &> &>   262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||||||| + std::invoke<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &> &>   262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||| + std::_Invoker_functor::_Call<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &> &> 262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||||||||| + std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &>::operator()    262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||||| + std::_Invoke_stored<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int> 262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||||||||||| + std::_Invoke_stored_explicit<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int,0,1,2,3,4> 262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||||||| + std::invoke<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int>   262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||||||||||||| + std::_Invoker_pmf_pointer::_Call<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int> 262548 (90.69%) 0 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||||||||| + Etc::Image::IterateThroughWorstBlocks    262548 (90.69%) 1 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||||||||||||||| + Etc::Block4x4::PerformEncodingIteration 262546 (90.69%) 9 (0.00%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||||||||||| + Etc::Block4x4Encoding_RG11::PerformIteration   234767 (81.10%) 1 (0.00%)   godot.windows.tools.64.exe  Kernel
||||||||||||||||||||||||||||||||||| + Etc::Block4x4Encoding_RG11::CalculateG11  118403 (40.90%) 51463 (17.78%)  godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||||||||||||| - Etc::Block4x4Encoding::CalcPixelError    40705 (14.06%)  24172 (8.35%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||||||||||||| - Etc::Block4x4Encoding_R11::DecodePixelRed    21797 (7.53%)   21791 (7.53%)   godot.windows.tools.64.exe  Kernel
|||||||||||||||||||||||||||||||||||| - Etc::ColorFloatRGBA::ColorFloatRGBA  3745 (1.29%)    3743 (1.29%)    godot.windows.tools.64.exe  Kernel

I suspect this has to do with this change from #51402.

Setting flags/anisotropic=false or flags/mipmaps=false does not fix the hang.

Steps to reproduce

Import png with above import definition on Godot 3.4 beta 4 using etc/etc2 compression. Godot will hang at re-importing assets pop-up.

Minimal reproduction project

vram-compression-hang.zip

Calinou commented 3 years ago

@calebROlson Does this occur in 3.3.2 or 3.3.3?

calebROlson commented 3 years ago

It does not occur in 3.3.3 or 3.4 beta 3.

Only seems to happen with the beta 4, both the dev snapshot and one I built myself with Visual Studio 2019.

calebROlson commented 3 years ago

I'm seeing this on 3.3.3 stable now. This is not a regression from 3.4beta3 (it is present in 3.3.3). Sometimes the operation finishes eventually, and other times it hangs until I force restart Godot. Once the import succeeds, I don't see the issue again until I re-import. I am running the 64-bit versions.

Seems similar to https://github.com/google/etc2comp/issues/46 and https://github.com/godotengine/godot/pull/37488 ? I noticed my callstack calls Etc::Block4x4Encoding_RG11::CalculateG11 instead of the RGB8 functions. It looks like there is still an issue in that library, and that #37488 was not the whole fix.

I am running GLES3. etc 2 is "only supported when using the GLES3 renderer".

I was able to workaround by disabling etc2 vram compression in the project settings for each affected release.

calebROlson commented 3 years ago

I found a fork of etc2comp (https://github.com/abergmeier/etc2comp) which has numerous fixes, including threaded executor support. After a few tweaks, I was able to build it with 3.3.3 and this fixed the issue for me and my textures imported in a couple seconds. However, the maintainer uses c++17 and I needed to switch my SConstruct file to use that instead of c++14 (is there a downside to doing this?).

I will attach my patch file for reference.

etc2comp_fork.patch

Calinou commented 3 years ago

However, the maintainer uses c++17 and I needed to switch my SConstruct file to use that instead of c++14 (is there a downside to doing this?).

We'd like to keep 3.x buildable with a C++14 compiler, but this is not an issue for the master branch which already requires a C++14 compiler.

fire commented 2 years ago

@calebROlson Would you be interested in assistance to make this a Godot 3 pr?