RangeError when parsing ESEA demos

[tugamars]

Hey, I got an error when parsing huge demo files (120mb).

I tried same code with a 34mb (old) demo and with a new demo from ESL with 140mb, and in the second case it throws a error on bytebuffer-node:

bytebuffer-node.js:499 throw RangeError("Illegal offset: 0 <= "+offset+" (+"+length+") <= "+this.buffer.length); ^

RangeError: Illegal offset: 0 <= 137980764 (+152) <= 137980880 at RangeError (native) at module.exports.ByteBufferPrototype.readBytes (C:\Users\marce\Desktop\csgo_demoaparser__NEW\node_modules\bytebuffer\dist\bytebuffer-node.js:499:23) at DemoFile._handleDemoPacket (C:\Users\marce\Desktop\csgo_demoaparser__NEW\demo.js:156:44) at DemoFile._parseRecurse (C:\Users\marce\Desktop\csgo_demoaparser__NEW\demo.js:271:14) at tryOnImmediate (timers.js:543:15) at processImmediate [as _immediateCallback] (timers.js:523:5)

Any ideia on how to solve it?

[tugamars]

Apparently it just happens when logging all the events.

[saul]

If you can provide a minimal reproduction and the demo itself I'll fix the bug that's causing this

[tugamars]

I think it just happens when you have to much data on the demo and try to print everything on the console.

I dont think it is a bug. I can send you demo file.

And the code is as basic as this:

demoFile.stringTables.on('update', e => { console.log(util.inspect(e, false, null)); });

I am having serious problems on other things, if you could just accept me on skype would be great (marcelo.silva.12). I already have my code working with jsgo, but i noticed it doesnt work with new demos and now I just cant adapt it to work with your module :/

[tugamars]

UPDATE:

Its not a problem on my code after all, it is an error that happens with new demos.

Code and error: http://i.imgur.com/88DKcMc.png?1

Demo used: http://www.hltv.org/?pageid=28&demoid=25801

[mavrick]

getting the same error demofile@0.3.1 using demo file from: http://www.hltv.org/?pageid=28&demoid=25901

C:\dfrag\demos\node_modules\bytebuffer\dist\bytebuffer-node.js:499
                throw RangeError("Illegal offset: 0 <= "+offset+" (+"+length+") <= "+this.buffer.length);
                ^

RangeError: Illegal offset: 0 <= 163429178 (+152) <= 163429328
    at RangeError (native)
    at ByteBuffer.module.exports.ByteBufferPrototype.readBytes (C:\dfrag\demos\node_modules\bytebuffer\dist\bytebuffer-node.js:499:23)
    at DemoFile._handleDemoPacket (C:\dfrag\demos\node_modules\demofile\demo.js:156:44)
    at DemoFile._parseRecurse (C:\dfrag\demos\node_modules\demofile\demo.js:271:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

code:

fs.readFile('./eclipse-vs-denial-cache.dem', function (err, buffer) 
{
    var demo = new demofile.DemoFile();

    demo.on('start', function() 
    {
        console.log('start');
    });

    demo.on('end', function() 
    {
        console.log('end');
    });

    demo.parse(buffer);
});

Also tested with 0.3.0 and 0.2.11 - same issue, appears to be an update to csgo that has caused this.

demos@1.0.0 C:\dfrag\demos
`-- demofile@0.2.11

npm WARN demos@1.0.0 No description

C:\dfrag\demos>node demo
start
C:\dfrag\demos\node_modules\bytebuffer\dist\bytebuffer-node.js:499
                throw RangeError("Illegal offset: 0 <= "+offset+" (+"+length+") <= "+this.buffer.length);
                ^

RangeError: Illegal offset: 0 <= 163429178 (+152) <= 163429328
    at RangeError (native)
    at ByteBuffer.module.exports.ByteBufferPrototype.readBytes (C:\dfrag\demos\node_modules\bytebuffer\dist\bytebuffer-node.js:499:23)
    at DemoFile._handleDemoPacket (C:\dfrag\demos\node_modules\demofile\demo.js:151:31)
    at DemoFile._parseRecurse (C:\dfrag\demos\node_modules\demofile\demo.js:231:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

[saul]

Thanks for the repro - I'll take a stab at fixing this tonight.

[saul]

I don't think this is something we're doing wrong as it looks like the format's changed/the demo file hasn't been fully written out. In the mean time you can ignore the exception - this is all happening just before parsing finishes anyway.

I've sent an email to @Migdalskiy hoping to shed some light on this from Valve's side (if the format's changed)

[tugamars]

Alright, so we cant use the "end" event, because it shuts down before making to it, right ?

[saul]

After speaking with one of the engine developers at Valve, it appears more likely that these demos are corrupted/not saved correctly.

Demos from random official public competitive matches appear to work fine.

[mavrick]

Interesting, it must be hltv.org that are either uploading or zipping the demos incorrectly on their end. I'm going to test today with a few more that I have on hand.

[tugamars]

The problem is on the demo it self, I just downloaded same demo from HLTV and ESEA website, and the output is the following (from esea website): http://pastebin.com/mbzKYJhn

Its probably due to ESEA system shuts the server down before tv_stoprecord and then the demo gets corrupted ?

[saul]

@tugamars that could possibly be it - you're best off getting in touch with ESEA/HLTV because it looks like demos on official servers are fine.

Please re-open the issue if you find out that this isn't the case.

[mavrick]

Just spoke with ESL/ESEA Australia, demo files from HLTV and directly from ESEA AU work fine

http://www.hltv.org/?pageid=188&matchid=41747 https://play.esea.net/index.php?s=servers&d=download_replay&id=7280835

I've spoken with HLTV reps and they are aware of the issue but will give me more information soon when they have something.

Looking into why OCE works vs NA doesn't, it sounds like they record their GOTV demos differently. (i dont have much clarity on this yet) From what I understand, OCE relies on the server gotv recording but NA/EU have someone recording the gotv? I still need clarification on this.

[pedrofornaza]

Im having the same problem. Tested with demos from november/2016 and from april/2017. Both from oficial MM SA.

                throw RangeError("Illegal offset: 0 <= "+offset+" (+"+1+") <= "+this.buffer.length);
                ^

RangeError: Illegal offset: 0 <= 90300877 (+1) <= 90300877
    at RangeError (native)
    at ByteBuffer.module.exports.ByteBufferPrototype.readUint8 (/vagrant/demoparser/node_modules/bytebuffer/dist/bytebuffer-node.js:646:23)
    at DemoFile._parseRecurse (/vagrant/demoparser/node_modules/demofile/demo.js:301:33)
    at runCallback (timers.js:666:20)
    at tryOnImmediate (timers.js:639:5)
    at processImmediate [as _immediateCallback] (timers.js:611:5)

Demo for testing: https://drive.google.com/open?id=0B-wmy64XD63ISm1RdTR2aDJ5dHM

[busheezy]

I have been having problems with a bunch of retakes server demos. I was trying to use this library but just figured our demos were the problem. I added some code to try and guarantee a tv_stoprecord but I didn't solve anything with my methods.

                throw RangeError("Illegal offset: 0 <= "+offset+" (+"+length+") <= "+this.buffer.length);
                ^

RangeError: Illegal offset: 0 <= 6945171 (+813) <= 6945744
    at ByteBuffer.module.exports.ByteBufferPrototype.readBytes (C:\Users\ryan9\Desktop\hsf\hsf-demos\demo-parser\node_modules\bytebuffer\dist\bytebuffer-node.js:499:23)
    at ByteBuffer.readIBytes (C:\Users\ryan9\Desktop\hsf\hsf-demos\demo-parser\node_modules\demofile\ext\bytebuffer.js:21:15)
    at DemoFile._handleDemoPacket (C:\Users\ryan9\Desktop\hsf\hsf-demos\demo-parser\node_modules\demofile\demo.js:203:31)
    at DemoFile._parseRecurse (C:\Users\ryan9\Desktop\hsf\hsf-demos\demo-parser\node_modules\demofile\demo.js:314:14)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

This is the demo. https://hsfactory.net/gotv/retakes/auto-20170507-1741-de_inferno-HSFactory_%7c_Retake_%234_%5bFR%5d.dem

Thanks a bunch.

[saul]

@pedrofornaza I've fixed the issue you reported with the cbble demo. I've release 0.4.8 to fix that.

@busheezy maybe it's retakes of demos that are causing issues? Exactly how are you retaking the demo?

[pedrofornaza]

Thanks @saul. I have a bunch of demos that are resulting in the same exception. Will test them after the release.

[busheezy]

I was told by someone that tv_autorecord doesn't work but honestly haven't tested. They are using a plugin that executes tv_record <filename> 5 seconds after the map starts. I tried adding tv_stoprecord to the OnMapEnd event provided by sourcemod. I tried the updates you pushed just to see if anything changed without luck.

[pedrofornaza]

@saul, cbble demo is working fine by now but i have an issue with another demo.

`/vagrant/demoparser/node_modules/bytebuffer/dist/bytebuffer-node.js:499 throw RangeError("Illegal offset: 0 <= "+offset+" (+"+length+") <= "+this.buffer.length); ^

RangeError: Illegal offset: 0 <= 46135530 (+1179) <= 46136272 at RangeError (native) at ByteBuffer.module.exports.ByteBufferPrototype.readBytes (/vagrant/demoparser/node_modules/bytebuffer/dist/bytebuffer-node.js:499:23) at ByteBuffer.readIBytes (/vagrant/demoparser/node_modules/demofile/ext/bytebuffer.js:21:15) at DemoFile._handleDemoPacket (/vagrant/demoparser/node_modules/demofile/demo.js:203:31) at DemoFile._parseRecurse (/vagrant/demoparser/node_modules/demofile/demo.js:319:14) at runCallback (timers.js:666:20) at tryOnImmediate (timers.js:639:5) at processImmediate [as _immediateCallback] (timers.js:611:5) make: *** [parse] Error 1`

Demo for tests: https://drive.google.com/file/d/0B-wmy64XD63INl90QktkWmRublk

[saul]

@pedrofornaza how did you record the demo that is failing? Does it play fully in the game client without any errors in the console?

[pedrofornaza]

@saul, its an official MM bro.

[necuk]

hello every1, got same issue here with a bunch of very recent demos, all from different tournaments. example: https://www.hltv.org/matches/2311763/mask-off-vs-ajax-esea-premier-season-25-north-america code to reproduce: examples/dumpfile.js

@saul, could you please throw some light on what might be causing the problem? even if it's caused by the server-side somehow, what might be the reason?

I also randomly bumped into some server plugin that pretends to "fix demo crash caused by tv_autorecord 1", not sure if it helps — but @busheezy mentioned it above. Anyway here is the link https://github.com/Thiry1/DemoAutoRecordFix

[necuk]

another example: https://www.hltv.org/matches/2311585/outlaws-vs-spraynpray-hellcase-cup-5 what is interesting is that there are two demos from the same game. inferno one fails while de_train works fine

[saul]

@necuk I've got a feeling that it's due to server-side. It looks like the end of the demo is simply missing data - it's as if the demo file hasn't been flushed correctly.

Thinking about it now I wonder if it's something to do with tv_delay? If tv_delay is set to 15 for example, potentially at the end of the game the demo writer is terminated before all the buffered data is written.

[Friss]

ESEA demos in particular I've found (while working on my own parser) don't emit the stop / end event code. To get around this I just check if a team has won at the end of a round and if that is true stop parsing.

Another thing to do is to check that you are at the end of the buffer before trying to read because at that point you have reached the end of the file and the game must have ended.

[saul]

It looks like the game is ending before the demo is fully written. I think you'll notice that all the playback variables in the header are 0. This is because these values are overwritten when the demo is finished.

I believe this is an issue with the engine. If somebody can give me a demo that demonstrates this issue that has been sourced from an official MM game, I can forward it to Valve to investigate.

[tugamars]

On ESL/ESEA demos what I did is run the function of endding the parser (function where the stats are sended to database) on exception, because i noticed every ESL/ESEA demo ends with an error/exception.

[tugamars]

Btw saul, last ebot demos with 500mb/600mb are working perfectly fine, so i think its really a esea problem.

[zyphie]

On ESEA demos, if I catch the exception, everything else has been running like normal in the few demos I have tried (incl. 'end' event).

[saul]

I've sent an email off to the ESEA guys so hopefully I'll get a response! Thanks for the reports guys.

[saul]

Closing this as won't fix as it's a game server issue - will let you know if I hear from ESEA.

As a workaround you can catch the exception and then you should get an end event anyway.

[necuk]

sorry for using a closed ticket. I had to quickly parse a bunch of demos and made a very quick patch to bypass the problem, here is the pull request https://github.com/saul/demofile/pull/34.

guys, maybe someone has a better solution? what I noticed is that if you dont put any event listener it still crushes. also the buggy demo generates not only RangeErrors but some other as well. I suppose we'd better move to the pull request in case some1 has any thoughts. thanks