Content in stored dictionary returning var_length of 0

[ccLouis]

I've used this addon to recovery encrypted player data for my game. Thanks!

The game's map data (unencrypted) is separate and more complex. Unfortunately when converting the map data it fires many errors when trying to determine variable types, which I believe can be traced to var var_length: int = _file_access.get_32() returning 0.

Unknown Godot 3 variant type: 65539
   At: res://addons/godot_3_to_4_file_conversion/file_access_godot3.gd:292:_convert_variant_type()

Unknown Godot 3 variant type: 1717986919
   At: res://addons/godot_3_to_4_file_conversion/file_access_godot3.gd:292:_convert_variant_type()

Unknown Godot 3 variant type: 1079764582
   At: res://addons/godot_3_to_4_file_conversion/file_access_godot3.gd:292:_convert_variant_type()

Debugger reads

E 0:00:11:0232   file_access_godot3.gd:217 @ _convert_variable(): Length of buffer cannot be smaller than 0.

E 0:00:11:0236   file_access_godot3.gd:187 @ get_var(): Condition "len < 4" is true. Returning: ERR_INVALID_DATA

E 0:00:11:0236   file_access_godot3.gd:187 @ get_var(): Error when trying to encode Variant.

The file (map_data.map) is created with store_var(), holding a Dictionary with some metadata and the central "map" key, which holds a Dictionary of metadata array keys pointing to arrays of Vector2 values (representing each instance of that metadata in the world).

I've attached a version of the test project here that reads the map_data.map and produces the same errors as seen in the game.

ValidateTestFilesGodot4.zip

[daveTheOldCoder]

Can you provide the Godot 3 code that created the file map_data.map? Or is it a single call to store_var() with a Dictionary parameter?

[ccLouis]

Right - just store_var(Dictionary) on the file.

[daveTheOldCoder]

I can reproduce the problem with your file.

What's the full version of Godot 3 that created the file?

[ccLouis]

The included file was created in 3.5.2, although I just tested with a file made in 3.5.3 and get the same result. I've included that file here if it's of use.

map_data.zip

[daveTheOldCoder]

Does the Dictionary contain anything of type Object or RID?

Was the second parameter (full_objects) passed as true in the call to store_var()?

[ccLouis]

The Dictionary shouldn't contain anything other than Dictionaries, Arrays, Strings, Vector2, Bool, int and float.

The second parameter of store_var() was set to false in Godot 3.

[daveTheOldCoder]

Okay, thanks. I'm trying to figure out what data is causing the problem. Handling nested Arrays and Dictionaries is the most complex part of this code. :)

[ccLouis]

Thanks! I figured it could be something to do with nested arrays/dictionaries - the returned dictionary results with some nested data as the first level keys, like something went out of sync.

[daveTheOldCoder]

I've done some analysis of the second file.

Here's what it contains:

Dictionary, num keys=4
    String value='game_version' size=20 data=[4, 0, 0, 0, 12, 0, 0, 0, 103, 97, 109, 101, 95, 118, 101, 114, 115, 105, 111, 110]
    String value='0.7.9i' size=16 data=[4, 0, 0, 0, 6, 0, 0, 0, 48, 46, 55, 46, 57, 105, 0, 0]
    String value='map' size=12 data=[4, 0, 0, 0, 3, 0, 0, 0, 109, 97, 112, 0]
    Dictionary, num keys=372
        Array, num elements=3
            Array, num elements=2
                int value=0 size=8 data=[2, 0, 0, 0, 0, 0, 0, 0]
                float value=36 size=8 data=[3, 0, 0, 0, 0, 0, 16, 66]
            Array, num elements=7
                String value='PineMature' size=20 data=[4, 0, 0, 0, 10, 0, 0, 0, 80, 105, 110, 101, 77, 97, 116, 117, 114, 101, 0, 0]
                bool value=false size=8 data=[1, 0, 0, 0, 0, 0, 0, 0]
                int value=0 size=8 data=[2, 0, 0, 0, 0, 0, 0, 0]
                Vector2 value=(0, 0) size=12 data=[5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                Vector2 value=(0, 0) size=12 data=[5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                String value='A' size=12 data=[4, 0, 0, 0, 1, 0, 0, 0, 65, 0, 0, 0]
                String value='' size=8 data=[4, 0, 0, 0, 0, 0, 0, 0]
            Dictionary, num keys=4
                String value='max_health' size=20 data=[4, 0, 0, 0, 10, 0, 0, 0, 109, 97, 120, 95, 104, 101, 97, 108, 116, 104, 0, 0]

The error occurs when trying to decode the next item.

Here are the next 100 bytes:
[3, 0, 1, 0, 103, 102, 102, 102, 102, 102, 87, 64, 4, 0, 0, 0, 6, 0, 0, 0, 104, 101, 97, 108, 116, 104, 0, 0, 3, 0, 1, 0, 103, 102, 102, 102, 102, 102, 87, 64, 4, 0, 0, 0, 6, 0, 0, 0, 102, 101, 108, 108, 101, 100, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0, 120, 45, 102, 108, 105, 112, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 43, 0, 0, 0, 5, 0, 0, 0]

Fact 1:
The first four of those bytes should be a Variant.Type, but the value is 65539 (hex 0x00010003), which the current code treats as invalid.

Fact 2:
If I decode its size using PackedByteArray.decode_var_size(), the result is 12 bytes.
If I decode its data using PackedByteArray.decode_var(), the result is a float with value 93.6.
The Variant.Type of a float is 3 (in both Godot 3 and Godot 4), not 0x00010003.

I don't understand the conflict between fact #1 and fact #2.

[daveTheOldCoder]

I determined that the problem is that Godot sets bit 16 of the four-byte type field of an encoded 64-bit numeric value, to distinguish it from a 32-bit numeric value. For example, setting bit 16 of 0x00000003 results in 0x00010003. My code wasn't taking that into account.

I made a tentative fix, and was able to load the Dictionary using this addon, and write it to a file using File_Access.store_var(), in Godot 4.2-rc2. I used the file you provided here: https://github.com/daveTheOldCoder/Godot3To4FileConversion/issues/3#issuecomment-1829650163

Please see if you can load the Dictionary in this file in Godot 4, using FileAccess.get_var() (without this addon): map_data_4.2-rc2.map.zip

If it's successful, I'll publish a corrected addon after I do some cleanup.

[ccLouis]

Thanks for this - can confirm the converted save works in my latest build!

[daveTheOldCoder]

This was fixed by commit https://github.com/daveTheOldCoder/Godot3To4FileConversion/commit/d8e9294f5e06fe8f2ac0a4aaa10ad25222751301

The fixed addon is in v1.0.2 https://github.com/daveTheOldCoder/Godot3To4FileConversion/releases/tag/v1.0.2

The updated version is in the Asset Library: https://godotengine.org/asset-library/asset/2307