search

skadistats / clarity

Comically fast Dota 2, CSGO and CS2 replay parser written in Java.
BSD 3-Clause "New" or "Revised" License
627 stars 119 forks source link

Unknown Message Types #58

Open spheenik opened 8 years ago

spheenik commented 8 years ago

If you come here, you know why. Thanks for your help!

Decoud commented 8 years ago

Match ID: 1782181262 Running clarity-examples tick package. Happens on many ticks:

tick 57378, synthetic true, had 0 messages unknown embedded message of kind SOURCE2/47. Please report this in the correspon ding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/47. Please report this in the correspon ding issue: https://github.com/skadistats/clarity/issues/58

Decoud commented 8 years ago

Same replay, I don't have the ticks but can get them if you want:

unknown embedded message of kind SOURCE2/497. Please report this in the correspo nding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/128. Please report this in the correspo nding issue: https://github.com/skadistats/clarity/issues/58

spheenik commented 8 years ago

uploaded a new snapshot, which has all those...

Decoud commented 8 years ago

unknown embedded message of kind SOURCE2/510. Please report this in the correspo nding issue: https://github.com/skadistats/clarity/issues/58

spheenik commented 8 years ago

new snapshot uploaded

Decoud commented 8 years ago

What type of messages are these things?

unknown embedded message of kind SOURCE2/62. Please report this in the correspon ding issue: https://github.com/skadistats/clarity/issues/58

howardchung commented 8 years ago

My guess is that @spheenik tried to remove as many of the messages from the generated Java classes as possible to reduce code size and speed up compile times.

spheenik commented 8 years ago

@howardchung: correct.

@Decoud: 62 is svc_UserMessage... In S1, UserMessages had their own container, in S2, they were, until now, encoded as Embedded Messages. The latest protobufs on SteamKit do not have the message definition (yet). Someone has to extract it from the client. I'll look into it tonight...

manveru commented 8 years ago

@spheenik latest protobufs are always available at https://github.com/SteamDatabase/GameTracking/tree/master/Protobufs/dota

howardchung commented 8 years ago

+1 for GameTracking :)

spheenik commented 8 years ago

@manveru: thx!

So, their netmessages say that it's CSVCMsg_HLTVStatus. I'll release a new protobuf snapshot tonight...

spheenik commented 8 years ago

new snapshot uploading as I type...

Decoud commented 8 years ago

SOURCE2/554

howardchung commented 8 years ago

I think this is the new CombatLogDataHLTV. @spheenik added support for it last night, try updating your snapshot?

Decoud commented 8 years ago

Works fine, my bad. Should have checked for the update first. Will have to take a closer look at some of those combat log changes.

spheenik commented 8 years ago

@Decoud: have a look at the comments in issue https://github.com/skadistats/clarity/issues/61 for info on what changed.

1zb commented 8 years ago

kind 534 or 544

file name: frankfurt_major\1856290165_1750727379.dem unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 league id: 3671 total time taken: 0.158s

file name: frankfurt_major\1858282173_1016472045.dem unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 league id: 3671 total time taken: 0.535s

file name: frankfurt_major\1859146913_1174254792.dem unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 league id: 3671 total time taken: 0.582s

file name: frankfurt_major\1860271085_922706473.dem unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 league id: 3671 total time taken: 0.601s

file name: frankfurt_major\1861004655_314559210.dem unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 league id: 3671 total time taken: 0.511s

file name: frankfurt_major\1861405896_2145933448.dem unknown embedded message of kind SOURCE2/544. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/544. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 league id: 3671 total time taken: 0.321s

file name: frankfurt_major\1861422181_1394172680.dem unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58 league id: 3671 total time taken: 0.213s

spheenik commented 8 years ago

Hi, thx for reporting!

Deployed a new snapshot of clarity protobuf that has 534,

The 544 is the new combatlog - I already included it a week ago. So it seems your snapshots are not up to date. mvn -U ... should do the trick.

spheenik commented 8 years ago

Bullshit - 544 was new too - added.

howardchung commented 8 years ago 
unknown embedded message of kind SOURCE2/557. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

According to proto definition appears to be some kind of match metadata. Type is bytes so it's probably binary encoded in some way.

spheenik commented 8 years ago

I have a fix, but I cannot test it. Can you provide a D/L link to a replay?

howardchung commented 8 years ago

https://s3-us-west-2.amazonaws.com/manta.dotabuff/2159568145.dem

spheenik commented 8 years ago

Thx, will push a fix tonight.

spheenik commented 8 years ago

Fix is out.

howardchung commented 8 years ago

unknown embedded message of kind SOURCE2/558. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

(lower priority, I don't think this crashes the parser)

TodorovicF commented 8 years ago

I also got the same error as @howardchung

unknown embedded message of kind SOURCE2/558. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

It seemed to have ran alright, no errors, though it does seem like it took longer to complete.

spheenik commented 8 years ago

added 558 (S2DotaGcCommon.CMsgDOTAMatch) which contains a lot of detailed information about the match.

PatheticNeatness commented 8 years ago

Can you help me with

18:36:10.003 [clarity-runner] WARN  s.c.p.reader.InputSourceProcessor - unknown usermessage message of kind SOURCE1/15. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
18:36:10.003 [clarity-runner] WARN  s.c.p.reader.InputSourceProcessor - unknown usermessage message of kind SOURCE1/27. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
spheenik commented 8 years ago

added those 2 with this commit and pushed a new snapshot.

TodorovicF commented 8 years ago

I've been getting crashes lately with this code:

unknown top level message of kind SOURCE2/1171. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

Exception in thread "clarity-runner" java.lang.ClassCastException: java.io.IOException cannot be cast to java.lang.RuntimeException
    at skadistats.clarity.event.Event.raise(Event.java:25)
    at skadistats.clarity.processor.runner.AbstractRunner.runWith(AbstractRunner.java:78)
    at skadistats.clarity.processor.runner.ControllableRunner.access$1401(ControllableRunner.java:13)
    at skadistats.clarity.processor.runner.ControllableRunner$3.run(ControllableRunner.java:208)
    at java.lang.Thread.run(Unknown Source)
spheenik commented 8 years ago

I did not update the clarity protobufs for quite a while, that's why you got the "unknown top level"-message, which is only a warning.

The exception you get is from a fucked up change I did sometime ago. So the real problem you have is in that IOException. I will revert the change soon...

WikiNeo commented 7 years ago

Match id 1513129245

unknown top level message of kind SOURCE2/5142. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
unknown top level message of kind SOURCE2/-1030568052. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
unknown top level message of kind SOURCE2/26. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
unknown top level message of kind SOURCE2/-2121819872. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
unknown top level message of kind SOURCE2/1781417. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
unknown top level message of kind SOURCE2/34. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
unknown top level message of kind SOURCE2/572190. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
unknown top level message of kind SOURCE2/244830986. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
spheenik commented 7 years ago

Most definitely corrupt replay. Does it crash afterwards?

WikiNeo commented 7 years ago

The program just suspends there, there is no crash or any responding.

spheenik commented 7 years ago

The match id is less than the earliest source 2 replay in the public domain. Where did you get it from?

WikiNeo commented 7 years ago

Using the https://github.com/Arcana/node-dota2, I can get the replay download url for a league match, then download it.

spheenik commented 7 years ago

But are you sure the match id you posted is correct?

spheenik commented 7 years ago

Just tried downloading, client says "Source 1 replays are unavailable in Source 2".

WikiNeo commented 7 years ago

I have the file locally, how can i share it with u?

WikiNeo commented 7 years ago

https://www.dropbox.com/s/0isd7l8gh7h9aos/1513129245.dem?dl=0

spheenik commented 7 years ago

dropbox, or send to github@martin.schrodt.org. No time to tool at it immediately though, maybe tonight. (CET)

spheenik commented 7 years ago

oh, there it is :)

spheenik commented 7 years ago

will look into into it later...

WikiNeo commented 7 years ago

Sure~ Have a good day :)

WikiNeo commented 7 years ago

match id： 1613735059 https://www.dropbox.com/s/1y0opwnik2ez7hc/1613735059.dem?dl=0

unknown top level message of kind SOURCE2/51. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: malformed varint detected
    at skadistats.clarity.event.Event.raise(Event.java:25)
    at skadistats.clarity.processor.runner.AbstractFileRunner.initAndRunWith(AbstractFileRunner.java:23)
    at skadistats.clarity.processor.runner.SimpleRunner.runWith(SimpleRunner.java:36)
    at risewinter.Parser.<init>(Parser.java:72)
    at risewinter.ProcessEntry.<init>(ProcessEntry.java:103)
    at risewinter.Analyser.<init>(Analyser.java:166)
    at risewinter.Analyser.main(Analyser.java:1813)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.io.IOException: malformed varint detected
    at skadistats.clarity.source.Source.readVarInt32(Source.java:106)
    at skadistats.clarity.processor.reader.InputSourceProcessor.processSource(InputSourceProcessor.java:110)
    at skadistats.clarity.event.AbstractInvocationPoint.invoke(AbstractInvocationPoint.java:72)
    at skadistats.clarity.event.Event.raise(Event.java:23)
    ... 11 more
spheenik commented 7 years ago

almost definitely corrupt replay data...

WikiNeo commented 7 years ago

Got it, thanks.

mephory commented 7 years ago

I get this message since the last update for every replay I checked: unknown embedded message of kind SOURCE2/563. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

Here's an example replay: http://mephory.com/2845882562.dem

spheenik commented 7 years ago

Yes, this is because I did not upgrade the protobufs yet. Will do so tomorrow.

spheenik commented 7 years ago

Headsup: New version (4.3) of the protobufs built and deployed to Maven Central.