Errors parsing games since Patch 1.1

[GoogleCodeExporter]

As of Patch 1.1 events in Replay Files aren't parsed properly.
Error on the upload Script: 
Version: 1.1.0.16561

Warning: unpack() [function.unpack]: Type C: not enough input, need 1, have 0 
in /opt/lampp/htdocs/ooaemkee/SC2Replay/mpqfile.php on line 87

Warning: unpack() [function.unpack]: Type C: not enough input, need 1, have 0 
in /opt/lampp/htdocs/ooaemkee/SC2Replay/mpqfile.php on line 87

Original issue reported on code.google.com by thorsten...@gmail.com on 23 Sep 2010 at 4:53

[GoogleCodeExporter]

Confirmed here with same error.

Original comment by webmaste...@gtempaccount.com on 24 Sep 2010 at 3:56

[GoogleCodeExporter]

Same error at our site.

Original comment by YellEl...@gmail.com on 25 Sep 2010 at 10:06

[GoogleCodeExporter]

Patch 1.1 completely changed a few of the events in the replay.game.events file 
and so the parseGameEventsFile() function is completely broken.

As a workaround you can add "return 0;" at the beginning of that function or 
comment out the function call on lines 90-95 in sc2replay.php. This will get 
rid of the errors, but some of the functionality will be unavailable. 
Specifically this will break build order lists, apm table, total apm, winner 
parsing, unit type lists and race detection. The locale-specific 'lrace' and 
four-letter 'srace' will still work, however.

Due to having to having to figure out a lot of the format again, there is 
currently no ETA for a fix.

Original comment by lauri.vi...@gmail.com on 25 Sep 2010 at 10:48

• Changed state: Started
• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Woops, that is to say, add "if ($this->build >= 16561) return 0;" to the 
beginning of the function. That way it will not interfere with older replays.

Original comment by lauri.vi...@gmail.com on 25 Sep 2010 at 11:32

[GoogleCodeExporter]

Also, I think apm in the replay become abnormally low, is this also one result 
attribute to the new version 1.1.0.16561?

Original comment by elohim...@gmail.com on 26 Sep 2010 at 7:18

[GoogleCodeExporter]

Since apm depends on parsed events i believe that no events mean no apm. 
Timestamps seem to be messed after a few seconds so apm graphs are messed too. 
If I understood more if hex values i would like to help but right now my 
knowledge of this stuff is way too less.

Original comment by thorsten...@gmail.com on 26 Sep 2010 at 9:27

[GoogleCodeExporter]

There seems to be two events that got changed (at least what I've noticed when 
I spent some time debugging this). One of which is the select event. I don't 
think the select event was changed completely since some of the select event 
still respects the "known" format. I think this event just got another special 
case. The other event seems to be the cast spell event. I'll continue to work 
on this some more but I don't see myself figuring out the select event without 
help.

Original comment by jdela...@gmail.com on 28 Sep 2010 at 1:13

[GoogleCodeExporter]

Seems like all the unit type codes got changed too? Can someone confirm?

Original comment by jeanmarc...@gmail.com on 28 Sep 2010 at 5:07

[GoogleCodeExporter]

lauri.virkamaki,
Hey lauri, we'd like to thank for your job and wanna make a decent proposal of 
paying for your job through some freelancer web services if you agree to devote 
more time for this project thus have it done more quick.
I mean we could award a private project for, say 40-50$, at freelancer.com if 
we could get updated parser earlier.
Regards

Original comment by YellEl...@gmail.com on 28 Sep 2010 at 9:21

[GoogleCodeExporter]

Sorry for offtop. 

Dear Alexey aka YellElite, of course, it's really good idea to donate for 
developer, but there is many other project, which could offer more than 40-50$. 
My project too, for example. And if everyone will pay to get the code first... 
or hold auctions after every patch, it will be nonsense. Hello, GPLv3.

So, let's be honest with other developers and projects, and let's make 
transparent donations instead doing something unfair.

Regards.

Original comment by sMKInGh...@gmail.com on 28 Sep 2010 at 11:02

[GoogleCodeExporter]

I am coding this library for 4 reasons: 1) I like Starcraft, 2) I like coding, 
3) I like figuring stuff out and 4) I like helping people.

As such I am already devoting whatever time I can to this project. It is 
unfortunate that it has already taken this long to figure out the changes to 
the format, but it will take as long as it takes. I do appreciate the offers, 
though, but knowing that the project is useful to other people is incentive 
enough.

I am hoping that I can get a working version up during the weekend, or if 
everything goes very well even sooner. I can't make any promises at this point, 
though.

Original comment by lauri.vi...@gmail.com on 28 Sep 2010 at 9:07

[GoogleCodeExporter]

Hi Lauri,

Do you need help? Do you mind sharing where you're at right now?

It looks like the ability codes did not change, but instead are shifted somehow 
in the sequence. I haven't figured out the new order yet.

Original comment by jeanmarc...@gmail.com on 28 Sep 2010 at 10:27

[GoogleCodeExporter]

Guys, don't get me wrong. The only point in my proposal was to motivate project 
owners and now i see it doesn't really matter.
So thanks a lot and waiting for 1.1.1 patch parser release soon =)

Original comment by YellEl...@gmail.com on 29 Sep 2010 at 9:53

[GoogleCodeExporter]

There's still a weird bit shift I haven't quite been able to put my finger on 
in the selection events, so my earlier estimates of being done this weekend 
won't be accurate. I think I have managed to tackle the two other problems the 
new patch created, so expect to have a fixed version sometime soon (depending 
on my time constraints and whether something else weird pops up or not).

I haven't been able to take a look at ability codes yet, so if you've managed 
to figure something out that would make things quicker once I do get there.

Original comment by lauri.vi...@gmail.com on 3 Oct 2010 at 9:30

[GoogleCodeExporter]

abilities seem to have a bit shift at least for buildings and unit builds:
ex: train zealot:
                   bit1       bit2      bit3
(old) 090B00 :  0000 1001  0000 1011  0000 0000
(new) 021B00 :  0010 0010  0001 1011  0000 0000

It seems that you have to do a couple things to translate to the old ability:
bit1_new = (bit1 << 2) + (bit2 & 0x30) >> 4
bit2_new = (bit2 & 0x0F)
bit3_new = (bit3 & 0x0F)

bit3_new & 0xF0 seems to be a flag of some kind but i haven't figured it out.

I am not 100% sure about this but I verified with a few units of protoss and 
terran and it works.

Original comment by jeanmarc...@gmail.com on 4 Oct 2010 at 4:37

[GoogleCodeExporter]

Lauri, one thing I want to bring up is how you plan on supporting multiple 
versions. Because I am sure you have realized that version bumps tend to break 
the parser quite often. It happened many times during the beta and has already 
happened in the short 2 month release of the released version. 

One method I found quite successful during the beta is to create subclasses. 
You can have a base class for the v1.0 replays and then create a subclass for 
the v1.1. The subclass will only need to implement the changed logic and it 
will still call the existing logic. This methods prevents the need to add if 
statements everyway checking what version of the replay is being parsed.

I only brought this up as merely a polite suggestion and hopefully as a way to 
learn a different method I wasn't aware of. :)

Original comment by jdela...@gmail.com on 4 Oct 2010 at 11:29

[GoogleCodeExporter]

I've added the stuff figured out so far on a wiki page at 
http://code.google.com/p/phpsc2replay/wiki/EventChanges_110. I will add the 
current test code to the repository tomorrow. It is still incomplete but parses 
most of the events correctly now and may help someone else at figuring out the 
rest.

Implementing major changes in subclasses sounds like it might work better than 
an if-clause hell. Once the code for 1.10 is working correctly I will have to 
take a look at refactoring the code as well as adding additional checks so that 
changes in the replay format do not break the entire parser in the future. In 
other words rather than outputting 230130 error messages the changed 
functionality will just fail silently.

Original comment by lauri.vi...@gmail.com on 5 Oct 2010 at 9:21

[GoogleCodeExporter]

Lauri I think I figured out the selection event. Just need to do some more 
testing to confirm

Original comment by jeanmarc...@gmail.com on 6 Oct 2010 at 1:50

[GoogleCodeExporter]

There also seems to be a new event (eventype:1, eventcode:0x06). 20 bytes.

Original comment by jeanmarc...@gmail.com on 6 Oct 2010 at 7:39

[GoogleCodeExporter]

Hello there!

I'm the author of Sc2gears, you probably know my project:
http://sites.google.com/site/sc2gears/

I put some notes regarding to the 1.1 replay format.

*First to the Select action:*

The "total number of units data" is not what it is. Here is a counter example 
of an action selection 6 drones:

`00 00 01 21 00 05 06 02 b1 00 00 01 b2 00 00 01 b3 00 00 01 b4 00 00 01 b5 00 
00 01 b6 00 00 01`

The "supposed" total number of units here is 2, which is incorrect, it should 
be 6. But you can calculate the "total number of units" by adding the counts of 
different unit types.

Also the extra bytes/bits in the deselection part does not seem to depend only 
on the high 4 bits of the flag. Here are some counter examples (each line is 
one select/deselect action). In these examples the high 4 bits of the flag are 
zeros, yet an extra byte occurs:
{{{
00 02 09 00 01 08 02 01 01 01 f0 00 00 01
00 03 05 03 00 00
00 03 09 02 00 00
00 03 0d 00 00 00
00 05 83 00 01 11 00 01 01 03 d4 00 00 01
00 05 fb 01 00 00
00 09 02 03 01 08 05 01 04 04 2d 00 00 07 3c 80 00 02 40 00 00 01 40 40 00 01
00 09 fc 03 01 21 01 01 01 01 f1 00 00 01
00 11 fc fe 02 00 00
}}}

*Comment regarding to jeanmarc.soum's ability conversion:*

The 2 example he brought up (old ability code: 0x090B00, new one: 0x021B00) is 
flawed. The binary representation does not equal to the listed hexa values. The 
algorithm only seems to work on train commands (haven't checked all though).

Original comment by icz...@gmail.com on 6 Oct 2010 at 8:14

[GoogleCodeExporter]

One more note. I've seen several cases where the opcode of 0x1b was use for the 
Action event instead of 0x0b. The format was the same. I don't know 0x1b was 
used instead of 0x0b, but I suggest to handle both the same way.

Original comment by icz...@gmail.com on 6 Oct 2010 at 10:07

[GoogleCodeExporter]

Also regarding to the Select event:

I'm not even sure it's bit-shifted anymore. Examples I've come across were byte 
aligned. I don't know yet how to interpret the deselection count and 
deselection map, but everything after that seems byte aligned:

the types count followed by the unit types and their counts followed by +1 byte 
(unknown) and then the 4-byte unit IDs.

Original comment by icz...@gmail.com on 6 Oct 2010 at 11:15

[GoogleCodeExporter]

I am pretty sure that I am correct about the bit shift, since it seems to work 
for most(all?) cases and explains the value before unit IDs. Let's take your 
example again:

00 00 01 21 00 05 06 02 b1 00 00 01 b2 00 00 01 b3 00 00 01 b4 00 00 01 b5 00 
00 01 b6 00 00 01

the first two zeroes are irrelevant, the following 01 is the number of unit 
type IDs. After this comes the unit type ID, which is bytes 21 00 05. The 
actual unit type id value for probe would thus be 21 00 01, since only the 
bottom two bits of the last byte go to the unit type id. 

Let's consider the three bytes before unit ids, which are 05 06 02. Applying 
the bitmask (byte1 & 0xFC) | (byte2 & 3) results in two bytes with a value of 
6, the first being the number of units for the unit type id and the second one 
being the total number of units. This bitmask seems to work flawlessly for flag 
bytes 0x00 and 0x03 (a flag byte of 0x04/0x08 actually means flag byte 00 with 
4/8 unit type ids plus the 2 low bits of the next byte so the bitshift starts 
there already).

The issue is that sometimes (with different flags and deselection byte values) 
different masks are required. Sometimes the mask is 0xFE, while sometimes a 
mask of 0xF8 gives the correct values and sometimes the values are 
byte-aligned. I have tried to experiment a lot but so far the answer has been 
rather elusive. I can of course not be 100% certain, but many replays I've 
parsed with my debugging code now only have a few erroneous selection events so 
I am pretty certain of the conclusion that a bitshift exists.

I have also seen an opcode of 0x8B instead of 0x0B so I guess all values 
between 0x0B...0x9B are possible. You are also quite correct about the 
deselection part not being only dependant on the high bits and I did not mean 
to say they were, perhaps the way in which I said it was unclear. The point was 
that there is no mystery to the high bits, they always add as many bits to the 
deselection part as the number they represent.

Original comment by lauri.vi...@gmail.com on 6 Oct 2010 at 7:44

[GoogleCodeExporter]

Guys here is what I figured out for the Selection event:

Bit1 : Flags
Bit2 : Deselect flag (2 low bits:0x03) will dictate how to read the following 
bytes
     0x00: No deselection. Bit shift of 2 for the whole message
     0x03: Deselect all units. Bit shift of 2
     0x01: Deselect some units. Variable bit shift. Bit mask length = Bit2 & 0xFC + Bit3 & 0x03. Ex: 05 13 = 00000101 00010011 => length = 00000111 = 7. unit mask starts at Bit3. The bit shift is calculated with the formula (mask_length - 6) modulo 8 (ex with bit mask length of 7, 6 bits will be in Bit3, 1 bit from Bit4 => bit shift = 1)
     0x02: Deselect some units, but no bit mask. Instead gives a unit count (Bit2 &0xFC + Bit3&0x03). Bit shift is 2. Ex: 02 21 01 => flag=0x02 unit_count=0x20 + 0x01=33

Everything else is read with the new bit shift. The ability codes might need 
some extra logic, because I couldn't match it to the old ones. But the unit 
counts match. per group unit count now add up to the total unit count.
Let's take this example:
0c 23 ac 00 05 20 01 02 03 01 01 01 20 50 00 01
05 20 => flag:0x01, bit_length:0x04 (4),bit_mask:ZZ1000XX, bit_shift is now 2? 
or maybe 6? (ZZ unused bits from Bit2)
01 => 1 group
02 03 01 => 02 03 01 or 02 01 03 => anyway its unknown
01 units in group
01 units total
etc...

Now take Andras's example:
00 00 01 21 00 05 06 02 b1 00 00 01 b2 00 00 01 b3 00 00 01 b4 00 00 01 b5 00 
00 01 b6 00 00 01

00 flag
00 deselect flag => bit shift of 2
   01 unit types
   21 00 05 06 => 01 20 01 06 => unit type 012001 x 6 count
   02 => 06 with previous byte (00000100) = 6 total units
   start of unit ids...

Now to decode action and hotkeys, and that new event EC1 ET:0x06

Original comment by jeanmarc...@gmail.com on 6 Oct 2010 at 9:06

[GoogleCodeExporter]

I want to correct something in my comment. The Action code I saw was 0x8b just 
what you found, not 0x1b. So at this point we've only seen 0x0b and 0x8b.

Original comment by icz...@gmail.com on 7 Oct 2010 at 7:02

[GoogleCodeExporter]

I've added your implementation of the selection event and it seems to work 
perfectly. I think I also have the correct logic for parsing the hotkey events 
and action events, but the specific meaning of all values is not clear yet.

As far as the ability codes and unit codes go, I will likely redo them all this 
weekend and see if there's a consistent logic so that I can just add a few 
lines with the correct bitshifts. If there isn't, I'll just map the new ability 
codes to the old ones in a new array so that all of the data isn't duplicated 
unnecessarily.

I also didn't see EC: 1 ET: 0x06 event in any of the replays I parsed (I made a 
quick batch script and ran about 40 replays through the parser), so if you have 
one I'd like to have a look at it.

I will leave this issue ticket open until the ability and unit codes are done, 
after which I will close it unless new patch 1.1 -related problems come up.

Original comment by lauri.vi...@gmail.com on 8 Oct 2010 at 9:49

[GoogleCodeExporter]

Your logic for hotkeys seem really complex.

Original comment by jeanmarc...@gmail.com on 9 Oct 2010 at 12:33

[GoogleCodeExporter]

when can we expect some solution?

// http://hasugamers.pl/replaye/

Original comment by imp4e...@gmail.com on 10 Oct 2010 at 1:39

[GoogleCodeExporter]

As much as I don't like the comment above since it doesn't seem very polite I 
think it's gret to see some feedback of projects using the script!

This is how far your tool brought us:

http://www.solstice-gaming.eu/sc2_replays/
http://www.solstice-gaming.eu/sc2_replays/320/

I really appreciate your work and if I had any knowledge of this bitshifting 
stuff I would really like to help!!

Thx for your work from me and my mates!!

Original comment by thorsten...@gmail.com on 10 Oct 2010 at 8:22

[GoogleCodeExporter]

A download has now been added to the download section that contains a working 
(tested with ~50 different replays) version of the parser as well as most of 
the changed ability codes. 

As this means that all major problems with version 1.10 have been fixed, I am 
closing this issue. If there are any new missing ability codes or replays that 
do not parse properly, please open a new issue ticket.

Original comment by lauri.vi...@gmail.com on 10 Oct 2010 at 9:20

• Changed state: Fixed

[GoogleCodeExporter]

Nice stuff, Lauri!

Original comment by zsol.z...@gmail.com on 10 Oct 2010 at 9:22

[GoogleCodeExporter]

THX!!

Original comment by imp4e...@gmail.com on 11 Oct 2010 at 12:35

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I've found a backwards compatibility bug in 1.10:

I parsed a replay vs. Hard Computer AI from version 1 (16291) that gaves the 
human player an overall max APM of 0.
The previous version of phpsc2replay (1.00) gives the human player a max APM of 
181.
Furthermore all actions and other stuff is not available...

It's reproducable...

Original comment by hostmas...@gab-ev.de on 11 Oct 2010 at 1:16

[GoogleCodeExporter]

Manuall hotkey use event (0x2D) still broken

Hex data from replay:
offset 0x0001E8DD: 18 21 2D 0E 09 05 06 0A 1D 13 3C 45 41 0B 18 61 21 2D 56 2C 
0F 00 61 21 2D 46 E1 02

Debug info:
Selection Change
Player 2
DEBUG L2: Timestamp: 1910, Frames: 14, Type: 1, Global: 0, Player ID: 2 
(Pheard), Event code: AC Byte: 0001E8DD

Byte1: 14, Byte2: 9, Numbyte: 1E8EC, Extra bytes: 1
DEBUG L2: Timestamp: 1910, Frames: 6, Type: 1, Global: 0, Player ID: 1 
(nekolisk), Event code: 2D Byte: 0001E8EC

Unknown event: Timestamp: 6177, Type: 1, Global: 0, Player ID: 13 (), Event 
code: 56 Byte: 0001E8F0

Original comment by ph3...@gmail.com on 11 Oct 2010 at 4:16