Open rmoe opened 6 years ago
Where did you get the demo you're parsing? I'd like to run it and see what's going on.
I checked on different demos. On demos from gotv and from player. For example. https://www.hltv.org/download/demo/43506
Oh, I see the problem. I'm guessing you installed the 0.0.1 package. I committed updated protobuf files (the cause of the error you're getting) and some bug fixes a few days ago and haven't had time to make a new release. I'm going to try to do that this weekend. If you install from source you'll be able to parse these demos.
How could I possibly do that ? Just download and throw in the folder with the module ?
Check out the source and then in the in the top-level directory (with setup.py) run:
$ pip install -r requirements.txt
$ pip install .
λ pip install .
Processing c:\users\t1moxa\desktop\demoparser-master
Complete output from command python setup.py egg_info:
Compiling demoparser/util.pyx because it changed.
Compiling demoparser/props.pyx because it changed.
Compiling demoparser/bitbuffer.pyx because it changed.
Compiling demoparser/demofile.pyx because it changed.
[1/4] Cythonizing demoparser/bitbuffer.pyx
[2/4] Cythonizing demoparser/demofile.pyx
[3/4] Cythonizing demoparser/props.pyx
[4/4] Cythonizing demoparser/util.pyx
c:\users\t1moxa\appdata\local\programs\python\python37-32\lib\site-packages\setuptools\pep425tags.py:89: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
warn=(impl == 'cp')):
c:\users\t1moxa\appdata\local\programs\python\python37-32\lib\site-packages\setuptools\pep425tags.py:93: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
warn=(impl == 'cp')):
ERROR:root:Error parsing
Traceback (most recent call last):
File "c:\users\t1moxa\appdata\local\temp\pip-req-build-zdxr0gi_\.eggs\pbr-4.2.0-py3.7.egg\pbr\core.py", line 96, in pbr
attrs = util.cfg_to_args(path, dist.script_args)
File "c:\users\t1moxa\appdata\local\temp\pip-req-build-zdxr0gi_\.eggs\pbr-4.2.0-py3.7.egg\pbr\util.py", line 259, in cfg_to_args
pbr.hooks.setup_hook(config)
File "c:\users\t1moxa\appdata\local\temp\pip-req-build-zdxr0gi_\.eggs\pbr-4.2.0-py3.7.egg\pbr\hooks\__init__.py", line 25, in setup_hook
metadata_config.run()
File "c:\users\t1moxa\appdata\local\temp\pip-req-build-zdxr0gi_\.eggs\pbr-4.2.0-py3.7.egg\pbr\hooks\base.py", line 27, in run
self.hook()
File "c:\users\t1moxa\appdata\local\temp\pip-req-build-zdxr0gi_\.eggs\pbr-4.2.0-py3.7.egg\pbr\hooks\metadata.py", line 26, in hook
self.config['name'], self.config.get('version', None))
File "c:\users\t1moxa\appdata\local\temp\pip-req-build-zdxr0gi_\.eggs\pbr-4.2.0-py3.7.egg\pbr\packaging.py", line 839, in get_version
name=package_name))
Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name csgo-demoparser was given, but was not able to be found.
error in setup command: Error parsing C:\Users\T1MOXA\AppData\Local\Temp\pip-req-build-zdxr0gi_\setup.cfg: Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name csgo-demoparser was given, but was not able to be found.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\T1MOXA\AppData\Local\Temp\pip-req-build-zdxr0gi_\
Did you clone this repo and then run the install command from inside that?
I downloaded zip, unpacked it to desktop folder, then i went to directory and run commands you gave me.
I tried to make a git clone
and seems it helped.
Successfully installed csgo-demoparser-0.0.1.dev22
Another bunch of errors
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
Exception ignored in: 'demoparser.demofile.DemoFile.handle_user_message'
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
Exception ignored in: 'demoparser.demofile.DemoFile.handle_user_message'
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
And with another demo
KeyError: 27
Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event'
KeyError: (27,)
KeyError: 27
Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event'
KeyError: (27,)
Traceback (most recent call last):
File "demoparser\demofile.pyx", line 518, in demoparser.demofile.DemoFile.class_by_net_message_type
AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage'
Exception ignored in: 'demoparser.demofile.DemoFile.handle_demo_packet'
Traceback (most recent call last):
File "demoparser\demofile.pyx", line 518, in demoparser.demofile.DemoFile.class_by_net_message_type
AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage'
Traceback (most recent call last):
File "1.py", line 10, in <module>
df.parse()
File "demoparser\demofile.pyx", line 120, in demoparser.demofile.DemoFile.parse
File "demoparser\demofile.pyx", line 155, in demoparser.demofile.DemoFile.parse
demoparser.demofile.CommandError: Unrecognized command
Can you give me a link to that demo?
Well ?
And with another demo
KeyError: 27 Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event' KeyError: (27,) KeyError: 27 Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event' KeyError: (27,) Traceback (most recent call last): File "demoparser\demofile.pyx", line 518, in demoparser.demofile.DemoFile.class_by_net_message_type AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage' Exception ignored in: 'demoparser.demofile.DemoFile.handle_demo_packet' Traceback (most recent call last): File "demoparser\demofile.pyx", line 518, in demoparser.demofile.DemoFile.class_by_net_message_type AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage' Traceback (most recent call last): File "1.py", line 10, in <module> df.parse() File "demoparser\demofile.pyx", line 120, in demoparser.demofile.DemoFile.parse File "demoparser\demofile.pyx", line 155, in demoparser.demofile.DemoFile.parse demoparser.demofile.CommandError: Unrecognized command
Is this with the original demo in OP? did you update the protobuf files?
Can you try appending this into netmessages.proto and rebuilding?
message CSVCMsg_EntityMessage {
optional int32 ent_index = 1;
optional int32 class_id = 2;
optional bytes ent_data = 3;
}
1) I don't understand what you mean. 2) No, I did not touch the sources of the library.
OK, I'll try
KeyError: 27
Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event'
KeyError: (27,)
KeyError: 27
Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event'
KeyError: (27,)
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
Exception ignored in: 'demoparser.demofile.DemoFile.handle_user_message'
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
Exception ignored in: 'demoparser.demofile.DemoFile.handle_user_message'
Traceback (most recent call last):
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString
self.MergeFromString(serialized)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1120, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 502, in DecodeField
field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
File "C:\Users\T1MOXA\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 469, in _ConvertToUnicode
return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 137: 'utf-8' codec can't decode byte 0xd0 in position 137: unexpected end of data in field: CCSUsrMsg_SayText2.msg_name
from demoparser.demofile import DemoFile
data = open('2.dem', 'rb').read()
df = DemoFile(data)
df.parse()
demo files >
1.dem >
can you confirm the svc_entitymsg exception is fixed in dem1? Can you also upload the file on google drive? cant seem to access it from my end.
2.dem >
If you don't care about user messages in your parsing, an easy fix would be to use df.deregister_callback('svc_UserMessage')
And with another demo
KeyError: 27 Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event' KeyError: (27,) KeyError: 27 Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event' KeyError: (27,) Traceback (most recent call last): File "demoparser\demofile.pyx", line 518, in demoparser.demofile.DemoFile.class_by_net_message_type AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage' Exception ignored in: 'demoparser.demofile.DemoFile.handle_demo_packet' Traceback (most recent call last): File "demoparser\demofile.pyx", line 518, in demoparser.demofile.DemoFile.class_by_net_message_type AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage' Traceback (most recent call last): File "1.py", line 10, in <module> df.parse() File "demoparser\demofile.pyx", line 120, in demoparser.demofile.DemoFile.parse File "demoparser\demofile.pyx", line 155, in demoparser.demofile.DemoFile.parse demoparser.demofile.CommandError: Unrecognized command
Is this with the original demo in OP? did you update the protobuf files?
Can you try appending this into netmessages.proto and rebuilding?
message CSVCMsg_EntityMessage { optional int32 ent_index = 1; optional int32 class_id = 2; optional bytes ent_data = 3; }
I had the same problem that seems to occur with demos from recent match (i tried with several match demo from GOTV demo from hltv).
To compile the source (pip install .) i think you need to have a recent version of protoc (protocol compiler from https://github.com/protocolbuffers/protobuf/releases) as it didn't work with the one shipped with ubuntu (version 2.6.1 compared to the recent 3.6.1).
@kpei Where did you find that protobuf snippet? Is there a newer version of that file somewhere or did you just write what was missing? I've been using this https://github.com/SteamDatabase/Protobufs to track the protobuf files. That snippet isn't there though.
@adrz protoc 2.6.1 should work (it's what gets installed in the Travis CI job). If 2.6.1 isn't working you can open another issue and I'll take a look at it.
@kpei Where did you find that protobuf snippet? Is there a newer version of that file somewhere or did you just write what was missing? I've been using this https://github.com/SteamDatabase/Protobufs to track the protobuf files. That snippet isn't there though.
It isn't but it seems like the error at line 518 is trying to locate CSVCMsg_EntityMsg but with the class name CSVCMsg_EntityMessage in the protobuf file so i think appending the struct itself would probably solve the issue. It seems like it worked and the exception no longer shows up.
demo files >
1.dem >
Can you also upload the file on google drive? cant seem to access it from my end.
This issue occurs because CS:GO are allowing new methods on the Protobuf handler. I'm going to PR the updated Protobuf, and also a better error handling for those cases for future.
Actually, the Key 69 means the WeaponSound CS_UM_WeaponSound = 69;
@kpei
message CSVCMsg_EntityMessage {
optional int32 ent_index = 1;
optional int32 class_id = 2;
optional bytes ent_data = 3;
}
fixes these errors:
Traceback (most recent call last):
File "demoparser\demofile.pyx", line 519, in demoparser.demofile.DemoFile.class_by_net_message_type
AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage'
Exception ignored in: 'demoparser.demofile.DemoFile.handle_demo_packet'
Traceback (most recent call last):
File "demoparser\demofile.pyx", line 519, in demoparser.demofile.DemoFile.class_by_net_message_type
AttributeError: module 'demoparser.protobufs.netmessages_pb2' has no attribute 'CSVCMsg_EntityMessage'
Traceback (most recent call last):
File "1.py", line 43, in <module>
d.parse()
File "demoparser\demofile.pyx", line 120, in demoparser.demofile.DemoFile.parse
File "demoparser\demofile.pyx", line 155, in demoparser.demofile.DemoFile.parse
demoparser.demofile.CommandError: Unrecognized command
Maybe you should send a PR ?
KeyError: 27
Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event'
KeyError: (27,)
KeyError: 27
Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event'
KeyError: (27,)
Errors like these still exist.
KeyError: 27 Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event' KeyError: (27,) KeyError: 27 Exception ignored in: 'demoparser.demofile.DemoFile.handle_game_event' KeyError: (27,)
Errors like these still exist.
Right now I made a clone of the repository and got these errors.
Are you trying to work with POV demos? I have never used this to parse a POV demo. I wonder if those are formatted differently.
Yes, POV
I found more errors, can you tell me why they occur ?
code:
Perhaps I should create a new issue for this?
Originally posted by @T1MOXA in https://github.com/ibm-dev-incubator/demoparser/issues/1#issuecomment-423605415