`ResourceSaver.save` error in tool script: `get_multiple_md5: Condition ' !f ' is true.`

giviz commented 4 years ago

Godot version:

Godot Engine v3.1.2.stable.official

OS/device including version:

Mac OS 10.14.6

Issue description:

Usage of ResourceSaver.save(name, texture) produce sometime the following error : ERROR: get_multiple_md5: Condition ' !f ' is true. Continuing..: At: core/os/file_access.cpp:622.

Which crash Godot (freeze)

Steps to reproduce:

1 - Unzip test project 2 - Start from command line 3 - Freeze of Godot, error in the terminal

Look into addons/codeandweb.texturepacker/texturepacker_import_spritesheet.gd line 115, it's where the method call is. You can also see the that file here : file on github

Minimal reproduction project: importBug.zip

To be noted that I've already spent 3 hours with the plugin author to try to debug the issue, but our conclusion is that it's a bug inside Godot and we are a bit clueless on how to move forward to fix that :D

It doesn't seems to be an issue with the plugin itself, but if so please let met know where.

Thanks !

Anutrix commented 4 years ago

Do try with latest master if can. Also, try to find the last version the addon worked fine with.

giviz commented 4 years ago

I just compiled the last master on the same platform and tested with it, same issue and same error in the console : ERROR: get_multiple_md5: Condition ' !f ' is true. Continuing..: At: core/os/file_access.cpp:667.

I'll now test previous versions of the addon as requested and report back.

giviz commented 4 years ago

So the error of the get_multiple_md5 is not related to that issue, it was related to an other issue inside the addon with a miss use of r_gen_files.

The freeze seems to be related to the fact that when there is too many files created during the import (many .tres created), there is a problem. If the number of files is reduce, the same import doesn't freeze.

I've tried to increase MAX_SERVER here from 64 to 640 but the issue is still there. Is that the correct constant to increase ? Should I look elsewhere ?

Here is the new debug project that produce the freeze without the other error importBug.zip

Now the thing is that there is no error when it's freezing, so I'm not sure where I can look to debug ?

I've also looked into previous versions of the addon, but they seems not compatible with that version of Godot.

giviz commented 4 years ago

The freeze happen after the addon return from the import method, so outside of the addon itself, and only if it import too many resources.

I've tried to use the debug (-d) option to get more info, but nada.

I'm not sure on I can dig more into that issue without debug feedback ?

giviz commented 4 years ago

I did some more testing with other Godot versions 3.0 -> WORK 3.0.6 -> WORK 3.1.stable -> FREEZE 3.1.2 -> FREEZE 3.2 -> FREEZE

Same addon folder, same assets folder to import in all cases.

Here is the project with all files that work in 3.0.x importBug-3.0.x-WORK.zip

Here is the project with all files that DO NOT work in > 3.0 importBug-3.1-3.2-FAIL.zip

deflinhec commented 4 years ago

So the error of the get_multiple_md5 is not related to that issue, it was related to an other issue inside the addon with a miss use of r_gen_files.

Great job on this one. 2020-11-22 AM3 59 02

seocwen commented 2 years ago

I'm having a similar issue with crashes using ResourceSaver.save

I'm running the following code to generate a large test texture and then save it:

func bad_texture_save():
    var image = Image.new()
    image.create(4096, 9192, false, Image.FORMAT_RGBA8)
    image.lock()
    image.fill(Color.transparent)
    image.unlock()
    var new_texture = ImageTexture.new()
    new_texture.create_from_image(image)
    ResourceSaver.save("res://resources/image/test.tres", new_texture)

The error the debugger gives me is as follows:

scene/resources/resource_format_text.cpp:804 - res://resources/image/test.tres:1 - Parse Error:
Failed loading resource: res://resources/image/test.tres. Make sure resources have been imported by opening the project in the editor at least once.

akien-mga commented 2 years ago

@seocwen You're trying to save an image with 37M pixels as a text file. It's probably HUGE and difficult to parse back. You should save it as .res so that it's in binary format, not text.

thunderk commented 1 year ago

I encountered this problem on 4.1 and 4.2beta6.

It happens in two custom importers that work on a large number of small files, one importer produces ImageTexture and another ArrayMesh. The import process (and the UI) freezes completely, with no error in stdout/stderr.

I don't know if this information can help in finding this bug, but it always freezes during or after the 8th file's processing in my case. So killing/rerunning Godot in loop will succeed in importing all of the files, 8 at a time.

Importing one file in isolation always succeeds, and is fast, with no unusual memory consumption.

thunderk commented 1 year ago

Even setting a minimal project with the texture importer and only 10 files, it freezes during import.

In this minimal project, it does not always happen, but randomly under certain conditions:

.godot/imported is empty
*.TXTR.import files do not exist yet
The importer plugin is enabled in Project Settings for the first time (interestingly it happens less if it's already enabled in project.godot)

The UI freezes fully like this:

Here is the minimal project:

bug.tar.gz

The output before freeze (from print statements in the plugin):

Godot Engine v4.2.beta6.official.64150060f - https://godotengine.org
Vulkan API 1.3.257 - Forward+ - Using Vulkan Device #0: AMD - AMD Radeon Vega 8 Graphics

WARNING: Blend file import is enabled in the project settings, but no Blender path is configured in the editor settings. Blend files will not be imported.
     at: _editor_init (modules/gltf/register_types.cpp:63)
res://data/M000.TXTR -> res://.godot/imported/M000.TXTR-4bb0fc982e0e8da4bb882b98bceb0aad.res
res://data/M001.TXTR -> res://.godot/imported/M001.TXTR-fe74a5ef0315d8fdb6f9bdc09f3f6428.res
res://data/M003.TXTR -> res://.godot/imported/M003.TXTR-967cf8977babfd44b3b1c3d40e57708c.res
res://data/M005.TXTR -> res://.godot/imported/M005.TXTR-06dad9b17abe199d2512f9c3387587af.res
res://data/M002.TXTR -> res://.godot/imported/M002.TXTR-83aa75747fe3ba7170d1863f4b39e98f.res
res://data/M004.TXTR -> res://.godot/imported/M004.TXTR-b19fd80f2afb1ffc7b8f131016cd8ab9.res
res://data/M006.TXTR -> res://.godot/imported/M006.TXTR-d45d4916580d73bdbed1c55d81a92f01.res
res://data/M007.TXTR -> res://.godot/imported/M007.TXTR-81e6b1cfd963935c1849355e686ea70f.res
res://data/M009.TXTR -> res://.godot/imported/M009.TXTR-f290dc67f49ba13700c462d305645eb5.res
res://data/M008.TXTR -> res://.godot/imported/M008.TXTR-6b69818b71410aede94f9d288f0be74f.res

The state of import files after freezing:

~/w/g/bug > find . -name '*.import' | sort                                                                                                                                                                        
./data/M000.TXTR.import
./data/M001.TXTR.import
./data/M002.TXTR.import
./data/M003.TXTR.import
./data/M004.TXTR.import
./data/M005.TXTR.import
./data/M006.TXTR.import
./data/M007.TXTR.import
./data/M009.TXTR.import
./icon.svg.import

The state of .godot/imported after freezing:

~/w/g/bug > ll .godot/imported                                                                                                                                                                      

   3288 16 nov.  09:20 icon.svg-218a8f2b3041327d8a5756f3a245f83b.ctex
     91 16 nov.  09:20 icon.svg-218a8f2b3041327d8a5756f3a245f83b.md5
     91 16 nov.  09:20 M000.TXTR-4bb0fc982e0e8da4bb882b98bceb0aad.md5
  49617 16 nov.  09:20 M000.TXTR-4bb0fc982e0e8da4bb882b98bceb0aad.res
     91 16 nov.  09:20 M001.TXTR-fe74a5ef0315d8fdb6f9bdc09f3f6428.md5
  49617 16 nov.  09:20 M001.TXTR-fe74a5ef0315d8fdb6f9bdc09f3f6428.res
     91 16 nov.  09:20 M002.TXTR-83aa75747fe3ba7170d1863f4b39e98f.md5
  49617 16 nov.  09:20 M002.TXTR-83aa75747fe3ba7170d1863f4b39e98f.res
     91 16 nov.  09:20 M003.TXTR-967cf8977babfd44b3b1c3d40e57708c.md5
  49617 16 nov.  09:20 M003.TXTR-967cf8977babfd44b3b1c3d40e57708c.res
     91 16 nov.  09:20 M004.TXTR-b19fd80f2afb1ffc7b8f131016cd8ab9.md5
  12753 16 nov.  09:20 M004.TXTR-b19fd80f2afb1ffc7b8f131016cd8ab9.res
     91 16 nov.  09:20 M005.TXTR-06dad9b17abe199d2512f9c3387587af.md5
 262610 16 nov.  09:20 M005.TXTR-06dad9b17abe199d2512f9c3387587af.res
     91 16 nov.  09:20 M006.TXTR-d45d4916580d73bdbed1c55d81a92f01.md5
 197073 16 nov.  09:20 M006.TXTR-d45d4916580d73bdbed1c55d81a92f01.res
     91 16 nov.  09:20 M007.TXTR-81e6b1cfd963935c1849355e686ea70f.md5
 197073 16 nov.  09:20 M007.TXTR-81e6b1cfd963935c1849355e686ea70f.res
      0 16 nov.  09:20 M008.TXTR-6b69818b71410aede94f9d288f0be74f.res-oHioUv
     91 16 nov.  09:20 M009.TXTR-f290dc67f49ba13700c462d305645eb5.md5
  12753 16 nov.  09:20 M009.TXTR-f290dc67f49ba13700c462d305645eb5.res

thunderk commented 1 year ago

Disabling import threads (import/use_multiple_threads=false) seems to solve this problem. So this looks like a race condition or deadlock somewhere. Still investigating.

akien-mga commented 1 year ago

@thunderk Please open a new issue, it's unlikely to be the same as an issue opened in 2020 for Godot 3.

I can reproduce your issue but it doesn't look like this one reported against 3.x. The 3.x importer doesn't use multiple threads like Godot 4.

I can still reproduce the OP's issue in 3.5.3-stable and 3.6-beta (e96c42f0e162685b95da5b721701476ec5ad07eb).

It doesn't crash for me, but it fails importing a file, and the file import is re-attempted every time the editor gets focused. On some platforms, this may lead to a loop for endless fail/reimport that may eventually crash.

Edit: Actually just after posting this and re-focusing the editor where I was testing this bug, I got an endless loop of these errors:

Failed method: FileSystemDock:_tree_thumbnail_done target ID: 8636
TOTAL BYTES: 4194264
NULL count: 0
CALL emit_signal: 2
CALL _update_callback: 2
CALL _update_minimum_size: 1
CALL _update_hover: 2
CALL update_script_classes: 1
CALL _tree_thumbnail_done: 34950
ERROR: Message queue out of memory. Try increasing 'memory/limits/message_queue/max_size_kb' in project settings.
   at: push_call (core/message_queue.cpp:54)

These can definitely lead to a crash eventually (stack overflow).

godotengine / godot

`ResourceSaver.save` error in tool script: `get_multiple_md5: Condition ' !f ' is true.` #34961