High CPU usage, time spent in texture updating

[hedin-hiervard]

Godot version:

Any Godot 3.1 alpha

OS/device including version:

Issue description:

I'm using Godot in a project where a lot of sprites are packed into a spritesheet and many .tres files generated and then used for creating TextureAtlases. All went well until today I noticed the IDE became highly unresponsive. Activity Monitor shows 100% CPU usage. I investigated a bit and found out most time it's doing this:

thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00000001029d096a godot.osx.tools.64`PoolVector<unsigned char>::resize(int) [inlined] operator new(p_size=1, p_pointer=0x000000017c1ab6ba, check=1, p_description="") at memory.h:104
    frame #1: 0x00000001029d0966 godot.osx.tools.64`PoolVector<unsigned char>::resize(this=0x00007fff5fbfe118, p_size=134217728) at dvector.h:589
  * frame #2: 0x0000000100c5d7cf godot.osx.tools.64`RasterizerStorageGLES3::texture_get_data(this=0x0000000108060a10, p_texture=RID @ 0x00007fff5fbfe170, p_layer=0) const at rasterizer_storage_gles3.cpp:1054
    frame #3: 0x00000001027d447b godot.osx.tools.64`VisualServerRaster::texture_get_data(this=0x0000000106b38990, arg1=RID @ 0x00007fff5fbfe1b8, arg2=0) const at visual_server_raster.h:153
    frame #4: 0x000000010282fa5b godot.osx.tools.64`CommandQueueMT::CommandRet2<VisualServer, Ref<Image> (VisualServer::*)(RID, int) const, RID, int, Ref<Image> >::call(this=0x00000001124e5570) at command_queue_mt.h:307
    frame #5: 0x000000010282d987 godot.osx.tools.64`CommandQueueMT::flush_one(this=0x00000001124c5128, p_lock=false) at command_queue_mt.h:425
    frame #6: 0x0000000102815b64 godot.osx.tools.64`CommandQueueMT::flush_all(this=0x00000001124c5128) at command_queue_mt.h:465
    frame #7: 0x0000000102815c0f godot.osx.tools.64`VisualServerWrapMT::sync(this=0x00000001124c5010) at visual_server_wrap_mt.cpp:90
    frame #8: 0x00000001000497fd godot.osx.tools.64`Main::iteration() at main.cpp:1867
    frame #9: 0x000000010000fd7e godot.osx.tools.64`OS_OSX::run(this=0x00007fff5fbfe640) at os_osx.mm:2564
    frame #10: 0x0000000100017698 godot.osx.tools.64`main(argc=4, argv=0x00007fff5fbff998) at godot_main_osx.mm:100
    frame #11: 0x00007fffcace9235 libdyld.dylib`start + 1
    frame #12: 0x00007fffcace9235 libdyld.dylib`start + 1

So, basically it reallocates the memory for the large texture (6MB) all the time, thus the CPU usage:

frame #2: 0x0000000100c5d7cf godot.osx.tools.64`RasterizerStorageGLES3::texture_get_data(this=0x0000000108060a10, p_texture=RID @ 0x00007fff5fbfe170, p_layer=0) const at rasterizer_storage_gles3.cpp:1054
   1051 
   1052     int data_size = Image::get_image_data_size(texture->alloc_width, texture->alloc_height, real_format, texture->mipmaps > 1);
   1053 
-> 1054     data.resize(data_size * 2); //add some memory at the end, just in case for buggy drivers
   1055     PoolVector<uint8_t>::Write wb = data.write();
   1056 
   1057     glActiveTexture(GL_TEXTURE0);

Deleting the texture from the project helps. I'm not sure what is causing this.

Steps to reproduce:

Minimal reproduction project:

[hedin-hiervard]

Update: deleting the texture or all of the using .tres files helps.

[hedin-hiervard]

The problem exists in all 3.1 alphas, but less noticeable on release builds (gives about 72% of CPU load), while a custom (debug) build renders it impossible to work with.

[eon-s]

I see that there are mentions of osx on the logs but there is no specification of that on the OP, is this MacOSX related or you see the problem on many OS?

[hedin-hiervard]

I see that there are mentions of osx on the logs but there is no specification of that on the OP, is this MacOSX related or you see the problem on many OS?

Yes, this is an editor problem and I only tested in on macOS. The game itself seems to work fine (no high CPU usage) on Windows and macOS.

[hedin-hiervard]

It seems to execute some sort of sync operation very frequently. There is a heap of defines which prevents me from understanding of what's going on really. Each operation calls texture_get_data which in turns allocates the whole memory required for the texture. I don't really know if this is an intended behavior or a bug.

[eon-s]

Probably related to #24478

See https://github.com/godotengine/godot/issues/24478#issuecomment-448860054

[hedin-hiervard]

Probably related to #24478

See #24478 (comment)

Turning on/off the VSync and Scene Sync doesn't change the situation.

[hedin-hiervard]

Here is the minimal repro project: https://github.com/hedin-hiervard/godot-cpu-bug

Opening it with 3.1 alpha 4 gives 56% CPU load, opening with a custom debug build gives 100% CPU load and clutchy UI.

[hedin-hiervard]

I have more info on this problem. I noticed the problem appears and disappears randomly with the same dataset. My workflow includes rebuilding the texture and the corresponding .tres files while Godot editor is open. Usually after I switch back to the IDE it hangs for a second with "(Re-)importing" message and everything is reloaded and works just smooth. So I tried to remove the problematic texture.png file, its .corresponding texture.png.import and the whole root .import directory. The Godot was switched off while removing the files. After I restarted Godot I saw everything intact including all the sprites. Obviously the texture was cached somewhere else (I searched the entire project dir and found nothing). After some trial and Godot reloads the texture sprites disappeared at last. After that I regenerated the texture again, it got re-imported and voila, no CPU load. So, my every 5 or so texture regens causes the texture to be "bad". In case you blame macOS reading files from Trash, I used rm -Rf to delete files.

[reduz]

The problem here is that you use a huge atlas, which needs to be loaded/freed every time a single bit of an atlas texture generates a preview in a thread. As OpenGL does not allow you to load/save texture data asynchronously in there, it has to transfer back and forth this huge texture every time by blocking the main thread.

This will be improved in Vulkan, which allows more control over asynchronously uploading data and creating resources, so for now there is no fix possible. Just wait patiently until all the previews are generated and then it will work.

[Calinou]

Is this issue still relevant with the addition of built-in atlas support back in April 2019?

[Calinou]

@hedin-hiervard Can you reproduce this in Godot 3.2.1 or 3.2.2beta2?

[hedin-hiervard]

@hedin-hiervard Can you reproduce this in Godot 3.2.1 or 3.2.2beta2?

I don't use atlases anymore. I found they don't add to the performance and the current implementation of Atlases in 3.1 is cumbersome to work with, so I switched to plain sprites.

[pwnorbitals]

Knowing this, should we just close this issue ? It doesn't seem to impact anyone else either

[Calinou]

Closing per the above comments.