How to parse the .tcr replay file

[biug]

Thank you for publishing the starcraft data. I'm a developer of broodwar bot and have great interest in deep learning. I have download your data in dumped_replays and decompress the data, got lots of .tcr file, but I don't find any furthur guidance of how to analyze the dumped file. I have deploy the torchcraft, and I wonder if you can give me some hint about the processing procedure. Thank you very much!

[jgehring]

Hi @biug, you can load the .tcr files via TorchCraft's Replayer class. In the extract_stats.cpp example in this repository you can find the following lines: (from https://github.com/TorchCraft/StarData/blob/master/extract_stats.cpp#L75-L77):

std::ifstream inRep(fname);
Replayer r;
inRep >> r;

Similar functionality is available in the Python (replayer.load(fname)) and Lua (replayer.loadReplayer(fname)) wrappers. The resulting Replayer object will contain all the information from the .tcr file.

[biug]

Thanks, I'll try it

[biug]

Hi @jgehring , I have tried to compile the extract_stats.cpp and try to parse some .tcr files in 0 directory, but it terminates with 'invalid map size' error, I wonder if I have made some wrong operation or files have some problem. I also try to use lua to parse the file, but receive a 'units nPlayer < 0' error

[ebetica]

Hi @biug, can you confirm that Torchcraft is compiled with zstd? I believe it won't find it by default, unless you've installed / compiled libzstd 1.1.4+.

Another thing you can try is to simply uncompress it with zstd before reading it to test whether torchcraft has zstd.

[biug]

Hi @ebetica , I found that I miss the zstd, so I install it from source code. Then I re-build the whole TorchCraft project and re-compile the extractstats.cpp. But it still show me the 'invalid map size' error. I'm into confusion @@ By the way, my OS is Ubuntu 16.04 32bit, and I follow https://github.com/TorchCraft/TorchCraft/blob/master/docs/user/installation.md#installing-the-torch-client-linux to install torchcraft Thank you very much.

[ebetica]

Are you using the right version of torchcraft? https://github.com/TorchCraft/TorchCraft/tree/90e3b9b12ece9fb7c5cbcc0e08a1e525a48a160d

FYI a Python client also exists, simply install zmq and zstd from https://github.com/TorchCraft/TorchCraft/tree/90e3b9b12ece9fb7c5cbcc0e08a1e525a48a160d/py and run pip install . from the root directory. It should crash if you don't have zstd, so it's an easy way to ensure you have it. Despite the beta tag, we've been using it for a while so it might be more stable than the torch version :D

[biug]

Thank for your patient reply @ebetica :D I checkout my version to 90e3b9, reinstall the torchcraft, and pip instal is OK. But error still there. I install pip from root so pip install . also run from root, maybe I should install pip from local? Does the problem occur here?

[ebetica]

I can confirm that I can read this replay, if you cannot maybe something went wrong with the download... https://www.dropbox.com/s/xow9cfg9r0rheuy/bwrep_00r3w.tcr?dl=0.

[ebetica]

Closing, feel free to reopen or submit a new issue if you have questions.

[andrewsilva9]

I am able to load replays in with Python and extract information from them, but the information appears to be only partially complete (units never lose health and no replay I've opened has a frame where is_terminal is 1), and I notice that attempting to run the extract_units or get_corrupted_replays scripts fails with exception: Corrupted replay: invalid map size. I'm trying this on the replay you linked above, so it should be a valid replay (and it is at least loading with the Python bindings). Do you know what might be going wrong?

[ebetica]

Sorry, is_terminal was actually a left over "bug" from when we were using this for micro battles in the same game. The replays do not actually record who wins, and it's not trivial to extract this data (since players may resign even if they seem stronger), so we do not provide it here. However, just checking the player with more supply at the end of the game is a very accurate heuristic.

I'm not entirely sure why extract_units and get_corrupt_replays fail... sorry. The most common cause of this error is a mismatch between version of torchcraft installed and version of the replay used.

[andrewsilva9]

Oh okay. Another question, do you know of a way to tell when a unit dies? It seems like they don't drop to 0 health, instead it's just whatever their health was at last before they took a killing blow. Is there a flag or any way to tell besides some heuristics involving movement and health? (also sorry if this should be a new issue or something)

[jgehring]

@andrewsilva9 the Python state has a deaths array that is populated with the IDs of all units that died between the previous frame and the current one.

[andrewsilva9]

@jgehring how do I access that? Is it in the replay object?

[rrz5013]

Was wondering if there were any updates on this? I am also getting an 'invalid map size' error and it looks like the width comes out to 0 for this file. I'm not sure about other maps because I don't know which replay files use different maps, but I tried a few in the 0 directory and they all came out to the same numbers.

[andrewsilva9]

"Invalid map size" is the error that always gets thrown when loading the replay fails for some reason. So it isn't necessarily the map, it could be that your path is wrong, something isn't built properly, some packages are missing, etc.