GraylinKim / sc2reader

A python library that extracts data from various Starcraft II resources to power tools and services for the SC2 community. Who doesn't want to hack on the games they play?
http://sc2reader.readthedocs.org
MIT License
413 stars 85 forks source link

some HotS replays die with MPQError: ('Unable to extract file: replay.message.events', error('Error -3 while decompressing data: incorrect header check',)) #102

Closed dsjoerg closed 11 years ago

dsjoerg commented 11 years ago

not sure if the problem exists in 2.0.4 as well. I'll add more details to my server log so that in the future I can easily answer that question.

the replays are here: https://www.dropbox.com/sh/k9nl1w4lox9smsu/VzWFYJiJ_w/sc2replays/2.0.3.24764/ProblematicReplays

08:58:02 python.1 | File "/Users/david/Dropbox/Programming/esdb/esdb/vendor/ggpyjobs/src/sc2reader/sc2reader/factories.py", line 137, in load 08:58:02 python.1 | return self._load(cls, resource, filename=filename, options=options) 08:58:02 python.1 | File "/Users/david/Dropbox/Programming/esdb/esdb/vendor/ggpyjobs/src/sc2reader/sc2reader/factories.py", line 147, in _load 08:58:02 python.1 | obj = cls(resource, filename=filename, factory=self, **options) 08:58:02 python.1 | File "/Users/david/Dropbox/Programming/esdb/esdb/vendor/ggpyjobs/src/sc2reader/sc2reader/resources.py", line 259, in init 08:58:02 python.1 | self._read_data(data_file, self._get_reader(data_file)) 08:58:02 python.1 | File "/Users/david/Dropbox/Programming/esdb/esdb/vendor/ggpyjobs/src/sc2reader/sc2reader/resources.py", line 590, in _read_data 08:58:02 python.1 | data = utils.extract_data_file(data_file,self.archive) 08:58:02 python.1 | File "/Users/david/Dropbox/Programming/esdb/esdb/vendor/ggpyjobs/src/sc2reader/sc2reader/utils.py", line 460, in extract_data_file 08:58:02 python.1 | file_data = archive.read_file(data_file, force_decompress=True) 08:58:02 python.1 | File "/Users/david/local/lib/python2.7/site-packages/mpyq.py", line 235, in read_file 08:58:02 python.1 | file_data = decompress(file_data) 08:58:02 python.1 | File "/Users/david/local/lib/python2.7/site-packages/mpyq.py", line 187, in decompress 08:58:02 python.1 | return zlib.decompress(data[1:], 15) 08:58:02 python.1 | MPQError: ('Unable to extract file: replay.message.events', error('Error -3 while decompressing data: incorrect header check',))

GraylinKim commented 11 years ago

As way of explanation I'll include the comment from the commit. In short, a hack to overcome corrupted replay files had a bug in it.


# Some replays tampered with by 3rd party software report
# block sizes wrong. They can either over report or under
# report. If they over report a non-compressed file might
# attempt decompression. If they under report a compressed
# file might bypass decompression. So do this:
#
#  * Attempt to do things normally.
#  * Force Decompression and fall back to original exception
#  * mpyq doesn't allow you to ignore decompression, it has
 #    not been a problem yet though.

See arkx/mpyq#7 and arkx/mpyq#12 for more information.