Godot 4 .ogg, .ttf and .glb files are imported with inconsistent resulting file hashes

[hhyyrylainen]

Tested versions

• Reproducible in: 4.3.stable.mono.official.77dcf97d8
• Not reproducible in: 3.5 (I'm 99% sure I didn't see this happening before the switch to Godot 4)

System information

Godot v4.3.stable.mono - Fedora Linux 40 (Workstation Edition) - Wayland - Vulkan (Forward+) - dedicated AMD Radeon RX 7900 XTX (RADV NAVI31) - AMD Ryzen 9 5950X 16-Core Processor (32 Threads)

Issue description

When importing a specific kind of file into Godot, the imported data (in the .godot folder) results in a hash that changes each time the file is imported. So even when the original file is not changed at all, Godot writes a generated file based on that that has a different hash. This is not very optimal as this increases delta compressed build sizes (for example Steam game updates). And unless this is intentional for these file types, this indicates that Godot writes some garbage / uninitialized bytes into the import files, or there is some other factor that results in (slight) randomization of the file contents.

This seems to happen to all .ttf and .ogg files in my project and some .glb files. In the case of .glb this might depend on the contents of the file as I'm not entirely confident I saw all of our .glb files changing hashes of the imported data. This has the effect that each CI build of my game now takes up multiple times more storage than a few months ago when I was still using Godot 3. That is how I noticed this problem and started to investigate.

I have hundreds of asset files in total where most of them don't experience the same issue, it's just these 3 file types. And for example .png files when imported do not experience the same issue but always import as exactly the same hash.

Here's an example comparing one imported .ogg file to itself one import cycle later. It seems that only 9 bytes out of 12 KB are different (here's them shown with the offset into the file and then the differing bytes):

00000160 72 32
00000161 64 67
00000162 70 6F
00000163 67 70
0000018E 6B 70
0000018F 31 65
00000190 6E 68
00000191 32 67
00000192 72 62

So the differences between the files are very minor but result in different hashes meaning that file identification based on hashes doesn't work and delta compression is less effective as there are a few changed bytes here and there invalidating otherwise the exact same blocks of data.

Steps to reproduce

• Open a project with .ttf, .ogg or .glb files in Godot 4.3 (in case the exact files matter I've attached a MRP with one of each type of file I noticed having this issue)
• Wait for Godot to import the files and close it
• Check the hashes of files in the .godot folder. I've provided a script that does this: ./check_import_hashes.rb
• Delete the .godot folder (or maybe just the imported folder under it is enough)
• Open Godot again to the project and wait for the import to run again
• Check the hashes of the files in the .godot folder one last time. The hashes are now different even though none of the input asset files were changed

Here's an example output of running that script again at the last step:

$ ./check_import_hashes.rb 
Calculating hashes...
Finished calculating hashes
Wrote new DB
New hash seen for file: .godot/editor/editor_layout.cfg (7ca33945be6e6237a4999673182aef680cb0ddf8654e6581590acdc58501c954)
New hash seen for file: .godot/imported/Flagellum.glb-679c8089bec3f669655bc36357199c1e.md5 (f4fc359c0c68319faf9ce8ae36906ff38aaa61184652509113f0a8f7be79d6e2)
New hash seen for file: .godot/imported/Flagellum.glb-679c8089bec3f669655bc36357199c1e.scn (c98659764d08fc0d688efcea7962c952b78869d250925897873e7fc1a394ffe9)
New hash seen for file: .godot/imported/Lato-Regular.ttf-f7608c791f23a0f20b371f48da679f3e.fontdata (e6ce0d27a7ac9f7ee28787d880968d5d78cf30ce62c8194d228941e1801bba28)
New hash seen for file: .godot/imported/Lato-Regular.ttf-f7608c791f23a0f20b371f48da679f3e.md5 (ec6642214309d0172d4beaa34919dfada52e678b7103f77fa4c90539ed5d3a85)
New hash seen for file: .godot/imported/pilus_puncture_stab.ogg-d4f2274dc40ba34ca5c9ef9878f2d4ed.md5 (758642acd4b7b5717ef3e4a805721d387abaed93b3c2496011c6eb3576fb8a7d)
New hash seen for file: .godot/imported/pilus_puncture_stab.ogg-d4f2274dc40ba34ca5c9ef9878f2d4ed.oggvorbisstr (fdb44df84b70934e0f3cd024227a3842189fe99208b80217634b82062f481521)
New hash seen for file: .godot/uid_cache.bin (871d52853101923bbd35f12343727f0240507b02a8a8852a716f041b65bf5828)

editor_layout.cfg and uid_cache.bin changing seem fine to me, but those asset files I'd expect to be consistently imported as the same hash.

For good measure doing the same steps again results in yet more hashes for all of the problematic files:

hhyyrylainen@dakara:~/test/InconsistentImport$ rm -rf .godot
hhyyrylainen@dakara:~/test/InconsistentImport$ godot --editor
Godot Engine v4.3.stable.mono.official.77dcf97d8 - https://godotengine.org
Vulkan 1.3.278 - Forward+ - Using Device #0: AMD - AMD Radeon RX 7900 XTX (RADV NAVI31)

WARNING: Ignoring face with non-finite normal in LOD generation.
     at: generate_lods (scene/resources/3d/importer_mesh.cpp:521)
hhyyrylainen@dakara:~/test/InconsistentImport$ ./check_import_hashes.rb 
Calculating hashes...
Finished calculating hashes
Wrote new DB
New hash seen for file: .godot/imported/Flagellum.glb-679c8089bec3f669655bc36357199c1e.md5 (1918d0ab5499a1921e9fa8ab5d31658878445ac75aef651c6ea1e7cd806f95b6)
New hash seen for file: .godot/imported/Flagellum.glb-679c8089bec3f669655bc36357199c1e.scn (5558bbd4275d88a85f6a44e29389ef98d2cf50187f555ab63f1eff9593d1517c)
New hash seen for file: .godot/imported/Lato-Regular.ttf-f7608c791f23a0f20b371f48da679f3e.fontdata (5a4025bbf7bddda4eeaec5ba87d3b761a9ec371eb0b5e0e246b42a4ce465a94b)
New hash seen for file: .godot/imported/Lato-Regular.ttf-f7608c791f23a0f20b371f48da679f3e.md5 (1e826d87c570b635da8f6a08fb05b57c9640198c9515db193a3753498a0f417c)
New hash seen for file: .godot/imported/pilus_puncture_stab.ogg-d4f2274dc40ba34ca5c9ef9878f2d4ed.md5 (6dfb71952b91c78d330c2d3d98467bccb145503359046bd683268d49229e100c)
New hash seen for file: .godot/imported/pilus_puncture_stab.ogg-d4f2274dc40ba34ca5c9ef9878f2d4ed.oggvorbisstr (4bee4aa12c4152f40d91213b91268a2787619a1993d3add921011c2d1e9d8064)

Minimal reproduction project (MRP)

InconsistentImport.zip

[mihe]

As noted in #97360 (which turned out to be a duplicate of this) the non-determinism stems from this piece of code:

https://github.com/godotengine/godot/blob/d5aadc38b459762fa74850777edb2ec8f6a02c16/core/io/resource_format_binary.cpp#L2317-L2348

Which is non-deterministic because of the fact that Resource::generate_scene_unique_id is generated based on timestamps, as seen here:

https://github.com/godotengine/godot/blob/d5aadc38b459762fa74850777edb2ec8f6a02c16/core/io/resource.cpp#L105-L113

From what I understand after chatting with @reduz about this very briefly, this ID shouldn't be generated for stuff that ends up in the .godot folder in the first place, since their IDs are found in the *.import files anyway.