Unable to parse demos

[nkashyap14]

I cloned the latest version of the repo and ran the install. I then tried to open a jupyter notebook file and parse a demo but whenever I do so I get this error.

Command I am running:

from csgo.parser import DemoParser

# Create parser object
# Set log=True above if you want to produce a logfile for the parser
demo_parser = DemoParser(demofile = "og-vs-natus-vincere-m1-dust2.dem", demo_id = "OG-NaVi-BLAST2020", parse_rate=128)

# Parse the demofile, output results to dictionary
data = demo_parser.parse()

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-1-7eb1ec812df7> in <module>
      7 
      8 # Parse the demofile, output results to dictionary
----> 9 data = demo_parser.parse()

~\desk\CS_DEMOS\parser\csgo\csgo\parser\demoparser.py in parse(self, return_type)
    146             A dictionary of output
    147         """
--> 148         self._parse_demo()
    149         self._read_json()
    150         if self.json:

~\desk\CS_DEMOS\parser\csgo\csgo\parser\demoparser.py in _parse_demo(self)
    115         stdout = proc.stdout.read().splitlines()
    116         output = [event.decode("utf-8") for event in stdout]
--> 117         self.output_file = output[0]
    118         if os.path.isfile(self.output_file):
    119             self.logger.info("Wrote demo parse output to " + self.output_file)

IndexError: list index out of range

I know this is a similar issue to what another person raised but in his case the log files at least listed something. These are my logs with the last message indicator being that it is looking for a demo file.

Go version>=1.14.0
18:56:00 [INFO] Go version>=1.14.0
Initialized CSGODemoParser with demofile C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
18:56:00 [INFO] Initialized CSGODemoParser with demofile C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
Setting demo id to OG-NaVi-BLAST2020
18:56:00 [INFO] Setting demo id to OG-NaVi-BLAST2020
Setting parse rate to 128
18:56:00 [INFO] Setting parse rate to 128
Running Golang parser from C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\
18:56:00 [INFO] Running Golang parser from C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\
Looking for file at C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
18:56:00 [INFO] Looking for file at C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
Go version>=1.14.0
18:58:07 [INFO] Go version>=1.14.0
Initialized CSGODemoParser with demofile C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\natus-vincere-vs-astralis-m1-dust2.dem
18:58:07 [INFO] Initialized CSGODemoParser with demofile C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\natus-vincere-vs-astralis-m1-dust2.dem
Setting demo id to na-vi-astralis
18:58:07 [INFO] Setting demo id to na-vi-astralis
Setting parse rate to 64
18:58:07 [INFO] Setting parse rate to 64
Running Golang parser from C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\
18:58:07 [INFO] Running Golang parser from C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\
Looking for file at C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\natus-vincere-vs-astralis-m1-dust2.dem
18:58:07 [INFO] Looking for file at C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\natus-vincere-vs-astralis-m1-dust2.dem
Go version>=1.14.0
18:58:08 [INFO] Go version>=1.14.0
Initialized CSGODemoParser with demofile C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
18:58:08 [INFO] Initialized CSGODemoParser with demofile C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
Setting demo id to OG-NaVi-BLAST2020
18:58:08 [INFO] Setting demo id to OG-NaVi-BLAST2020
Setting parse rate to 128
18:58:08 [INFO] Setting parse rate to 128
Running Golang parser from C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\
18:58:08 [INFO] Running Golang parser from C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\
Looking for file at C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
18:58:08 [INFO] Looking for file at C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem

Go + Python version: go version go1.16.2 windows/386 Python 3.8.4

The environment I'm working in is local jupyter notebooks. I have even tried running the example notebooks you ship with the repo along with the specific demo that you test with and I am getting the same error. Any ideas?

[pnxenopoulos]

Hi @nkashyap14, thanks for checking out the library. I reran the code locally, and it executed on my Linux machine. Quick question -- do you see a file named og-vs-natus-vincere_de_dust2.json in the same directory where you ran the parser?

In any case, my guess is that the error for these lines:

    115         stdout = proc.stdout.read().splitlines()
    116         output = [event.decode("utf-8") for event in stdout]

has to do with Windows. What is happening in these lines is that the Go code is writing the name of the JSON file to standard out. I have an idea for how to fix this, and I think my original code may not be optimal/incompatible with Windows. I mull it over the next day or so, and get back to you with a fix. If you have access to a Mac or Linux machine, I suggest trying the code on there as well, to double-check.

[nkashyap14]

Thanks for the quick response. I'll definitely spin up a different environment and test the code there as well.

[pnxenopoulos]

Here's what I just pushed:

I changed the code to write out a json called <demo_id>.json. The old version had map name in the title, but that is redundant since the map name is one of the top-level keys in the json.

The build ran, so give it a try, and let me know if it fixed the problem.

[nkashyap14]

Command Run:

demo_parser = DemoParser(demofile = "og-vs-natus-vincere-m1-dust2.dem", demo_id = "OG-NaVi-BLAST2020", parse_rate=128, log=True)

Jupyter Cell Output:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-3-cd037ba95003> in <module>
      7 
      8 # Parse the demofile, output results to dictionary
----> 9 data = demo_parser.parse()

~\desk\CS_DEMOS\parser\csgo\csgo\parser\demoparser.py in parse(self, return_type)
    146         """
    147         self._parse_demo()
--> 148         self._read_json()
    149         if self.json:
    150             self.logger.info("Successfully parsed JSON output")

~\desk\CS_DEMOS\parser\csgo\csgo\parser\demoparser.py in _read_json(self)
    128         json_path = self.outpath + "/" + self.output_file
    129         self.logger.info("Reading in JSON from " + self.output_file)
--> 130         with open(json_path) as f:
    131             demo_data = json.load(f)
    132         self.json = demo_data

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\nikhi\\desk\\CS_DEMOS\\parser\\csgo/OG-NaVi-BLAST2020.json'

Log:

Go version>=1.14.0
Initialized CSGODemoParser with demofile C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
Setting demo id to OG-NaVi-BLAST2020
Setting parse rate to 128
Running Golang parser from C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\
Looking for file at C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\og-vs-natus-vincere-m1-dust2.dem
No file produced, error in calling Golang
Reading in JSON from OG-NaVi-BLAST2020.json

As you can see it says no file produced error in calling Golang which essentially means the issue I'm getting is at 117 in demoparser.py in _parse_demo where the output file does not exist as a json file. I also do not see the json file being created. Does this mean there is an issue with golang on my end? I don't think it should when I run the go version command via cli I get go version go1.16.2 windows/386 as output. Does this mean that there is an error in running the go parser as a process in the python program? Is this an environment issue and do you recommend I just switch to a linux environment?

[pnxenopoulos]

Do you see a demoparser.log in C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\csgo\parser\ ? If so, what are its contents?

I would also recommend a Mac or Linux environment, as that's where I develop/test most of the package. However, it's important we also sort out this error on your Windows environment, as well.

[nkashyap14]

No I see a csgo_demoparser.log in C:\Users\nikhi\desk\CS_DEMOS\parser\csgo\ which is where I moved the example workbooks and am running it from as well. That is where the log I posted from my last comment is coming from as well. I don't see a demoparser.log in the location you mentioned.

[nkashyap14]

Managed to spin up a ubuntu instance via wsl and set up a python 3.6 golang 1.16.1 environment on it and I'm parsing smoothly there. As such take your time on this issue as I have found a workaround. I would appreciate you figuring out the issues with windows environments however as that is typically what I use. Thanks for your prompt responses so far.

[pnxenopoulos]

@nkashyap14 could you try pulling the new release and checking if the problem on Windows persists?

[nkashyap14]

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-3-cd037ba95003> in <module>
      7 
      8 # Parse the demofile, output results to dictionary
----> 9 data = demo_parser.parse()

~\desk\CS_DEMOS\parser\csgo\csgo\parser\demoparser.py in parse(self, return_type)
    146         """
    147         self._parse_demo()
--> 148         self._read_json()
    149         if self.json:
    150             self.logger.info("Successfully parsed JSON output")

~\desk\CS_DEMOS\parser\csgo\csgo\parser\demoparser.py in _read_json(self)
    128         json_path = self.outpath + "/" + self.output_file
    129         self.logger.info("Reading in JSON from " + self.output_file)
--> 130         with open(json_path) as f:
    131             demo_data = json.load(f)
    132         self.json = demo_data

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\nikhi\\desk\\CS_DEMOS\\parser\\csgo/OG-NaVi-BLAST2020.json'

Still getting an error where it can't find the json file

[pnxenopoulos]

Perhaps it has to do with the pathing. C:\\Users\\nikhi\\desk\\CS_DEMOS\\parser\\csgo/OG-NaVi-BLAST2020.json -- there's a "/" in there. Can you try opening the json file in Python by doing

import json

with open("C:\\Users\\nikhi\\desk\\CS_DEMOS\\parser\\csgo\\OG-NaVi-BLAST2020.json") as f:
    data = json.load(f)

and see if that works?

[nkashyap14]

sorry for the late reply. Turns out in windows for some reason that json file is not even being created. It constantly evaluates to false at line 121 for

if os.path.isfile(self.output_file):

For some reason its not producing that json file on my windows environment. However on the linux environment its working fine.

[pnxenopoulos]

I see. Looking back at the log you produced, I see No file produced, error in calling Golang. Perhaps it has to do with the pathing in the Go file. Could you move the demofile to C:\\Users\\nikhi\\desk\\CS_DEMOS\\parser\\csgo\\parser, run go run parse_demo.go --demo <demofile name> --parserate 128 --demoid test --out ., and show the contents?

[nkashyap14]

I get this response "go tool: no such tool "asm""

I tried installing it via tools but those commands give me the same answer. I don't have much experience with golang so nothing at first sight comes to mind. I'm digging in on stack overflow still. As I said though I still have the repo running on wsl

[LaihoE]

This library is just what I needed, if only I could get it to work. I think it's related to what is discussed above. Tried on windows and ubuntu, same error.

go: github.com/markus-wa/demoinfocs-golang/v2@v2.8.1: missing go.sum entry; to add it:
    go mod download github.com/markus-wa/demoinfocs-golang/v2
Traceback (most recent call last):
  File "C:/Users/emill/PycharmProjects/csgoparse/csgo/main_fl.py", line 11, in <module>
    data = demo_parser.parse()
  File "C:\Users\emill\PycharmProjects\csgoparse\csgo\csgo\parser\demoparser.py", line 154, in parse
    self._read_json()
  File "C:\Users\emill\PycharmProjects\csgoparse\csgo\csgo\parser\demoparser.py", line 136, in _read_json
    with open(json_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\emill\\PycharmProjects\\csgoparse\\csgo/benu.json'

[LaihoE]

Downgraded Go to 1.15 and got it working on Windows

[pnxenopoulos]

@nkashyap14 have you seen this Stack Overflow post? Does it help? And if not, did downgrading Go to 1.15 work?

@LaihoE I'm not sure the issues are the same -- can you open a separate issue and output any *.log files that were generated? If they are indeed the same then I will simply close the issue you created.

[nkashyap14]

@nkashyap14 have you seen this Stack Overflow post? Does it help? And if not, did downgrading Go to 1.15 work?

@LaihoE I'm not sure the issues are the same -- can you open a separate issue and output any *.log files that were generated? If they are indeed the same then I will simply close the issue you created.

Will try later today and see.

[pnxenopoulos]

Hi @nkashyap14 and @LaihoE -- last night @arjun-22 successfully installed the package on Windows. He had Go 1.16 (installed following the instructions on the Golang site) and Python>3.7

Could you both try pulling the latest repository code (made some updates last night), ensure you have Go 1.16 and then try installing?

[nkashyap14]

@pnxenopoulos Sorry I realized I forgot to reply. I downgraded to golang 1.15 and had gotten it working on windows!