All activity is displayed in top right for 1v1

dsjoerg commented 8 years ago

From @gravelweb on April 10, 2016 22:36

The mini-map feature is not working properly for 1v1. All activity seems to be happening at the top right, outside the boundaries of the map. This doesn't seem to be an issue in 2v2.

Copied from original issue: dsjoerg/ggtracker#19

dsjoerg commented 8 years ago

Whoa wow that is fascinating! I had assumed that the bug affected all game modes. The fact that it only happens in 1v1 is a massive clue and should make the bug a lot easier to fix. Thanks!

dsjoerg commented 8 years ago

OK, I've narrowed down the problem quite a bit (while streaming my session on twitch which has been fun :) )

The problem is that for certain maps (maybe all 1v1 maps? not sure) the coordinate system as reported by replay.map.map_info.camera_left, replay.map.mapinfo.camera* is no longer the same as the coordinate system used in the camera events.

Here's a session where I determined this:

was:sc2reader david$ python
Python 2.7.9 (default, Apr  7 2015, 07:58:25) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sc2reader
>>> replay = sc2reader.load_replay('/Users/David/Downloads/ggtracker_6586434.SC2Replay')
>>> replay.load_map()
>>> replay.map.map_info
<sc2reader.objects.MapInfo object at 0x101a1f310>
>>> replay.map.map_info.camera_right
134217728
>>> replay.map.map_info.camera_left
1768189509
>>> replay.map.map_info.camera_top
3489660928
>>> replay.map.map_info.camera_bottom
134245999
>>> camera_events = [e for e in replay.events if e.name == 'CameraEvent']
>>> len(camera_events)
3370
>>> camera_events[2500].x
44.25
>>> camera_events[2500].y
152.61328125

here's what a normal replay is supposed to be like:

>>> replay = sc2reader.load_replay('/Users/David/Downloads/ggtracker_6586262.SC2Replay')
yo
yo
yo
yo
yo
yo
yo
yo
yo
>>> replay.load_map()
>>> replay.map.map_info.camera_left
9
>>> replay.map.map_info.camera_right
175
>>> replay.map.map_info.camera_bottom
4
>>> replay.map.map_info.camera_top
168
>>> camera_events = [e for e in replay.events if e.name == 'CameraEvent']
>>> len(camera_events)
4828
>>> camera_events[2000].x
142.1796875

dsjoerg commented 8 years ago

So why are certain maps reporting really weird camera coordinates? I have no clue at the moment. Maybe there has been a change in the map data format.

In the absence of other ideas, maybe the sc2replaystats guy Anthony Martin will know. I believe he is parsing some map information as well. Also the code for SCELight is open source, András might know. And I can always ask Bliz.

dsjoerg commented 8 years ago

The relevant function starts at https://github.com/GraylinKim/sc2reader/blob/master/sc2reader/objects.py#L478. With Heroes of the Storm they easily could have added stuff to the map format which would throw off the reader.

dsjoerg commented 8 years ago

OK, asked Anthony, will post if he knows anything.

dsjoerg commented 8 years ago

Anthony doesnt have a way, he does it manually from the Galaxy Map Editor TT @gravelweb would you like to take a crack at it from here? Two ways to go might be 1) look at the byte stream and maybe the correct bytes are obviously sitting there at a different offset, depending on which map version it is 2) look at SCELight / SC2Gears (both of which have code available) and see if either of those parses the map and if so, how.

dsjoerg commented 8 years ago

Anthony reminded me of another possibility; we could take a different approach completely and use some empirical magic to make a good guess. We can use the camera events themselves to estimate the min-max range.

dsjoerg commented 8 years ago

From @gravelweb on April 11, 2016 16:32

@dsjoerg I'm happy to look into it. Seems like a nice way to get my feet wet here :) Thanks for the details. Will check it out this evening.

dsjoerg commented 8 years ago

From @gravelweb on April 18, 2016 2:25

Partial fix pushed to ggtracker/sc2reader#11. Details on what I mean by "partial":

This works well with the following maps:

Invader LE
Korhal Carnage
Frozen Temple
Ruins of Endion
Dusk Towers

Unfortunately, I'm having issues with Prion Terraces now. The symptoms are the same, but obviously the cause isn't, since my data dump is reporting seemingly correct camera info:

02:15:05 python.1 | 2016-04-18 02:15:05 2939 WARNING 10 8 150 160

I'll continue to investigate, but if you have any ideas off the top of your head.... happy to hear them!

dsjoerg commented 8 years ago

ok cool! no ideas off the top of my head, basically what i do to diagnose these things is start in the client and work backward until i find bad data, and then look at the parsing code and see why its creating bad data. i'll take a look at this monday or tuesday night and see what i can figure out.

that said, if ggtracker/sc2reader#11 works on the current ladder maps at least as well as the current code, then i'm inclined to push it into production and enjoy its benefits. have you tried it on the ladder maps? (not just 1v1)

dsjoerg commented 8 years ago

From @gravelweb on April 18, 2016 2:34

I have tried all of the 1v1 ladder maps, since those were the ones affected... But I suppose it would be nice to verify that the other modes still work too 8)

dsjoerg commented 8 years ago

From @gravelweb on April 18, 2016 2:51

1v1: Newly Fixed:

Dusk Towers
Frozen Temple
Ruins of Endion
Ulrena
Invader LE
Korhal Carnage Knockout LE

Still Broken

Prion Terraces

2v2: Still Working:

Infestation
Omicron
Sludge City

Newly Fixed:

Fields of death
Magma Mines
Distant Plane
Moebius Facility XX-1

3v3: Not tested:

Avalon Labs
Backcountry
Doraelus Hills
Ice Cliffs
New Richmond
New Williamsberg
Slaying Field

4v4: Not tested:

Abandoned Parish
Asper Mountain
Hidden Caves
Kimeran Refuge
Moebius Facility XX-1
Research Site JD-2
Void Zone

dsjoerg commented 8 years ago

From @gravelweb on April 18, 2016 2:53

Looks pretty good :) I wanted to note, however, that this feature seems to break when reaching a certain part of the replay. I'm not sure if this is related to #16

dsjoerg commented 8 years ago

From @gravelweb on April 18, 2016 2:55

The above reported issue is likely related to the timescale changes, as they do not seem to exist in production.

dsjoerg commented 8 years ago

From @gravelweb on April 18, 2016 3:1

One more thing. The existing replays don't seem to get reparsed, so this fix will not affect existing replays, unless the user re-uploads them. Is that standard in ggtracker?

dsjoerg commented 8 years ago

@gravelweb we do map processing only once, the first time we see that map. My plan was to manually somehow force map re-processing in production once you've got this working to your satisfaction. For your dev testing purposes you can blow away the esdb_sc2_maps row to force a map re-process; however that will screw up any existing parsed esdb_matches rows that refer to the old map_id.

dsjoerg commented 8 years ago

Current status: waiting for 3v3 and 4v4 ladder maps to be tested and then I'll merge in https://github.com/ggtracker/sc2reader/pull/11, deploy to production, force map reprocessing and that'll be victory.

gravelweb commented 8 years ago

All done. I made a table:

Map	Mode	Status
Dusk Towers	1v1	Fixed
Frozen Temple	1v1	Fixed
Ruins of Endion	1v1	Fixed
Ulrena	1v1	Fixed
Invader LE	1v1	Fixed
Korhal Carnage Knockout LE	1v1	Fixed
Prion Terraces	1v1	Still Broken

Infestation	2v2	Still Working
Omicron	2v2	Still Working
Sludge City	2v2	Still Working
Fields of Death	2v2	Fixed
Magma Mines	2v2	Fixed
Distant Plane	2v2	Fixed
Moebius Facility XX-1	2v2	Fixed

Avalon Labs	3v3	Fixed
Backcountry	3v3	Still Working
Doraelus Hills	3v3	Fixed
Ice Cliffs	3v3	Fixed
New Richmond	3v3	Fixed
New Williamsberg	3v3	Still Working
Slaying Field	3v3	Fixed

Abandoned Parish	4v4	Still Working
Asper Mountain	4v4	Fixed
Hidden Caves	4v4	Fixed
Kimeran Refuge	4v4	Still Working
Moebius Facility XX-1	4v4	Fixed
Research Site JD-2	4v4	Still Working
Void Zone	4v4	Fixed

Still rattled about Prion though... Oh well

gravelweb commented 8 years ago

In case you're curious, scelight does the same rigamarole as we do for map parsing. They do it slightly differently, in that from position 0x20 they read "string, uint32, string", but we're now reading "string, string, uint32". These are all unknowns, but the way we're doing it preserves the value 1 in the uint32 unknown.

1v1:

00000000  49 70 61 4d 26 00 00 00  7b 5c 01 00 00 00 00 00  |IpaM&...{\......|
00000010  a0 00 00 00 a0 00 00 00  01 00 00 00 01 00 00 00  |................|
00000020  00 00 01 00 00 00 00 00  00 00 44 61 72 6b 00 4b  |..........Dark.K|
00000030  6f 72 68 61 6c 43 69 74  79 45 78 32 00 06 00 00  |orhalCityEx2....|

2v2:

00000000  49 70 61 4d 22 00 00 00  11 dc 01 00 00 00 00 00  |IpaM"...........|
00000010  d8 00 00 00 d8 00 00 00  01 00 00 00 01 00 00 00  |................|
00000020  00 01 00 00 00 00 00 00  00 44 61 72 6b 00 4b 6f  |.........Dark.Ko|
00000030  72 68 61 6c 43 69 74 79  00 11 00 00 00 10 00 00  |rhalCity........|

Not sure how much of a "maintenance hell" awaits if we need to update MapInfo parse every time blizzard releases a new map format, but maybe that's just on blizzard to release some libraries for us to use :) Or we could investigate this "empirical magic" that you briefly mentioned..

dsjoerg commented 8 years ago

Outstanding work! Releasing this to production will proceed once problems noted in #28 are resolved.

dsjoerg commented 8 years ago

deployed and fixed in production. and somehow even Prion Terraces is working! see http://ggtracker.com/matches/6617813 as an example

gravelweb commented 8 years ago

Well I'll be darned...

gravelweb commented 8 years ago

Woohoo!

ggtracker / ggtrackerstack

All activity is displayed in top right for 1v1 #7