format changes

[inolen]

Hi,

I was trying to use the 01-basic tool for the first time on a recent demo, and on parsing I get:

CUtlVectorEmbeddedNetworkVar< CNetworkedIKProceduralTargetContext >
/home/inolen/projects/butterfly/src/butterfly/private/flattened_serializer.cpp:206: Unhandled fs type.

I naively copied and pasted a bunch of seemingly similar fields thinking they may have just been renamed in the underlying code:

diff --git a/src/butterfly/private/flattened_serializer.inline b/src/butterfly/private/flattened_serializer.inline
index 7cc7a60..a9018f4 100644
--- a/src/butterfly/private/flattened_serializer.inline
+++ b/src/butterfly/private/flattened_serializer.inline
@@ -103,6 +103,45 @@ MATCH_VECTOR("CUtlVector< FowBlocker_t >"_chash, "FowBlocker_t", 1)
 MATCH_VECTOR("CUtlVector< CTransform >"_chash, "CTransform", 1)
 MATCH_VECTOR("CUtlVector< CBasePlayerWeapon >"_chash, "CBasePlayerWeapon", 1)

+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CAnimationLayer >"_chash, "CAnimationLayer", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CDOTASubChallengeInfo >"_chash, "CDOTASubChallengeInfo", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CDOTACustomShopItemInfo >"_chash, "CDOTACustomShopItemInfo", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CDOTACustomShopInfo >"_chash, "CDOTACustomShopInfo", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CDOTA_ItemStockInfo >"_chash, "CDOTA_ItemStockInfo", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CDOTA_PlayerChallengeInfo >"_chash, "CDOTA_PlayerChallengeInfo", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CHeroStatueLiked >"_chash, "CHeroStatueLiked", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CHeroesPerPlayer >"_chash, "CHeroesPerPlayer", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< CNetworkedIKProceduralTargetContext >"_chash, "CNetworkedIKProceduralTargetContext", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< DataTeamPlayer_t >"_chash, "DataTeamPlayer_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< PlayerResourceBroadcasterData_t >"_chash, "PlayerResourceBroadcasterData_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< PlayerResourcePlayerData_t >"_chash, "PlayerResourcePlayerData_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< PlayerResourcePlayerEventData_t >"_chash, "PlayerResourcePlayerEventData_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< PlayerResourcePlayerTeamData_t >"_chash, "PlayerResourcePlayerTeamData_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< TempViewerInfo_t >"_chash, "TempViewerInfo_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< TreeModelReplacement_t >"_chash, "TreeModelReplacement_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< InGamePredictionData_t >"_chash, "InGamePredictionData_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< PingConfirmationState_t >"_chash, "PingConfirmationState_t", 1)
+MATCH_VECTOR("CUtlVectorEmbeddedNetworkVar< DOTAThreatLevelInfo_t >"_chash, "DOTAThreatLevelInfo_t", 1)
+
+MATCH_VECTOR("CNetworkUtlVectorBase< NeutralSpawnBoxes_t >"_chash, "NeutralSpawnBoxes_t", 1)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CUtlSymbolLarge >"_chash, prop_decode_string, 10)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CHandle< CBaseEntity > >"_chash, prop_decode_varint, 10)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CHandle< CBaseFlex > >"_chash, prop_decode_varint, 2)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CHandle< CBasePlayer > >"_chash, prop_decode_varint, 64)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CHandle< CEconWearable > >"_chash, prop_decode_varint, 100)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CHandle< CIngameEvent_Base > >"_chash, prop_decode_varint, 10)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CHandle< CPostProcessingVolume > >"_chash, prop_decode_varint, 10)
+MATCH_SIMPLE("CNetworkUtlVectorBase< CHandle< CBaseModelEntity > >"_chash, prop_decode_varint, 100)
+MATCH_VECTOR("CNetworkUtlVectorBase< CHandle< CBasePlayerWeapon > >"_chash, prop_decode_varint, 100)
+MATCH_VECTOR("CNetworkUtlVectorBase< float32 >"_chash, prop_decode_float, 1)
+MATCH_VECTOR("CNetworkUtlVectorBase< int32 >"_chash, prop_decode_varint, 1)
+MATCH_VECTOR("CNetworkUtlVectorBase< uint8 >"_chash, prop_decode_varint, 1)
+MATCH_VECTOR("CNetworkUtlVectorBase< uint32 >"_chash, prop_decode_varint, 1)
+MATCH_VECTOR("CNetworkUtlVectorBase< Vector >"_chash, prop_decode_vector, 1)
+MATCH_VECTOR("CNetworkUtlVectorBase< bool >"_chash, prop_decode_bool, 1)
+MATCH_VECTOR("CNetworkUtlVectorBase< QAngle >"_chash, prop_decode_qangle, 0)
+MATCH_VECTOR("CNetworkUtlVectorBase< FowBlocker_t >"_chash, "FowBlocker_t", 1)
+MATCH_VECTOR("CNetworkUtlVectorBase< CTransform >"_chash, "CTransform", 1)

 // float
 MATCH_SIMPLE("float32"_chash, prop_decode_float, 0)
@@ -123,6 +162,8 @@ MATCH_OBJECT("CPlayerLocalData"_chash, "CPlayerLocalData", 0)
 MATCH_VECTOR("m_SpeechBubbles"_chash, "m_SpeechBubbles", 1)
 MATCH_OBJECT("ECNYBeastAbility[24]"_chash, "ECNYBeastAbility", 24)
 MATCH_OBJECT("CBaseVRHandAttachmentHandle"_chash, "CBaseVRHandAttachmentHandle", 0)
+MATCH_VECTOR("WeightedSuggestion_t[3]"_chash, "WeightedSuggestion_t", 3)
+MATCH_VECTOR("WeightedSuggestion_t[5]"_chash, "WeightedSuggestion_t", 5)
 MATCH_VECTOR("WeightedSuggestion_t[15]"_chash, "WeightedSuggestion_t", 15)
 MATCH_VECTOR("AttachmentHandle_t[10]"_chash, "AttachmentHandle_t", 10)
 MATCH_OBJECT("PingConfirmationIconType"_chash, "PingConfirmationIconType", 0)

until I hit:

/home/inolen/projects/butterfly/src/butterfly/public/butterfly/util_bitstream.hpp:139: Expected n < size - pos: Bitstream overflow.

Breakpoint 1, 0x00007ffff779fc80 in exit () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.33-20.fc34.x86_64 libgcc-11.2.1-1.fc34.x86_64 libstdc++-11.2.1-1.fc34.x86_64 protobuf-3.14.0-3.fc34.x86_64 snappy-1.1.8-5.fc34.x86_64 zlib-1.2.11-26.fc34.x86_64
(gdb) bt 10
#0  0x00007ffff779fc80 in exit () from /lib64/libc.so.6
#1  0x000000000043844d in bf_assert (format=0x53b1ce "%s:%d: Expected %s < %s: %s.\n", format=0x53b1ce "%s:%d: Expected %s < %s: %s.\n") at /home/inolen/projects/butterfly/src/butterfly/public/butterfly/util_assert.hpp:53
#2  0x0000000000438c69 in butterfly::prop_decode_varint(butterfly::bitstream&, butterfly::fs_info*, butterfly::property*) () at /home/inolen/projects/butterfly/src/butterfly/public/butterfly/util_bitstream.hpp:140
#3  0x000000000041cb64 in butterfly::entity::parse (this=0x7ffff769d090, b=...) at /home/inolen/projects/butterfly/src/butterfly/private/entity.cpp:133
#4  0x00000000004100e0 in butterfly::parser::svc_handle_entities (this=0x7fffffffccd0, data=<optimized out>, size=<optimized out>, v=<optimized out>) at /usr/include/c++/11/bits/stl_vector.h:1043
#5  0x0000000000410c71 in butterfly::parser::dem_handle_packet (this=0x7fffffffccd0, bs=..., v=0x7fffffffccb0) at /home/inolen/projects/butterfly/src/butterfly/private/parser.cpp:362
#6  0x0000000000411104 in butterfly::parser::parse (this=0x7fffffffccd0, v=0x7fffffffccb0) at /home/inolen/projects/butterfly/src/butterfly/private/parser.cpp:158
#7  0x000000000041123b in butterfly::parser::parse_all (this=0x7fffffffccd0, v=0x7fffffffccb0) at /home/inolen/projects/butterfly/src/butterfly/private/parser.cpp:181
#8  0x000000000040b797 in main (argc=<optimized out>, argv=0x7fffffffcf88) at /home/inolen/projects/butterfly/examples/cpp/01-basic.cpp:42
(gdb) frame 3
#3  0x000000000041cb64 in butterfly::entity::parse (this=0x7ffff769d090, b=...) at /home/inolen/projects/butterfly/src/butterfly/private/entity.cpp:133
133                 prop->decoder( b, prop->info, p );
(gdb) p *prop
$1 = {properties = std::vector of length 0, capacity 0, decoder = 0x438bb0 <butterfly::prop_decode_varint(butterfly::bitstream&, butterfly::fs_info*, butterfly::property*)>, info = 0x8ffc30, name = "m_nMinGPULevel", 
  hash = 3042826424764152715}

Have I done something wrong here, or are these just new changes to the replay format that aren't yet supported?

Thanks!

[Noxville]

Hey there.

Over time there are more entity types added - and yes it's in flattened_serializer.inline where you specify how these are decoded - you got it spot on. If you see the git history I've had the same maintenance to do over time.

At some point in the past there's been something which has gone wonky in the decoding, here's decoding status grouped by patch (for pro games):

 patch_name | num_good | num_bad
------------+------+------
 6.84       |  169 |   13
 6.85       | 1560 |   30
 6.86       | 2349 |   54
 6.87       | 1427 |   18
 6.88       | 3381 |   72
 7.00       |  132 |   31
 7.01       |  641 |   10
 7.02       |  414 |   38
 7.03       |  147 |    1
 7.04       |  235 |    6
 7.05       |  750 |   12
 7.06       | 2032 |   74
 7.07       | 1602 |   20
 7.08       |  445 |    5
 7.09       |  181 |    2
 7.10       |  193 |    2
 7.11       |  416 |    6
 7.12       |  213 |    2
 7.13       |  280 |    4
 7.14       |  121 |    1
 7.15       |  175 |    1
 7.16       |   97 |    6
 7.17       |  246 |    1
 7.18       |  129 |    1
 7.19       | 1102 |   65
 7.20       |  674 |    9
 7.21       | 1924 |   25
 7.22       | 1812 |   21
 7.23       |  463 |    8
 7.24       |  563 |   40
 7.25       |  569 |   46
 7.26       | 1393 |  191
 7.27       | 2100 | 1199
 7.28       |    0 | 1703
 7.29       |    0 | 2015
 7.30       |    0 | 1376

You can see that in 7.24 things started going quite pear-shaped.

Because I only use butterfly for audio-parsing (and not general replay parsing), it's not been a super high priority for me to fix it exactly, however I did discuss this with @spheenik a while ago and we said we'd look into it when we got time (he's currently in the process of moving).

It's certainly a bug I want to fix - firstly so I can reparse the broken audio, but also because I'd like to recompile the Javascript lib. I'm just not sure when I will have time to look into it right now, maybe spheenik has some time over Christmas break and we can fix it up. Definitely would be interested if you come across any ideas / solutions.

[inolen]

@Noxville thanks for the heads up! Is that data set you're testing against available for download anywhere?

[Noxville]

It's just the pro matches on datdota.com - if you're having trouble getting some ping me on Discord (on discord.gg/datdota).

[RyanAfrish7]

@inolen Had any luck with parsing? I'm facing the same issue as well.