Memory error with large replays

[captain-majid]

Screp is unble to parse the cmds of this replay, definitely it's a huge one, but can this be solved ? https://drive.google.com/file/d/1rCG0MSX7lRj2s9-RmB5nYWzvmvTZMpCd/view?usp=sharing

[icza]

screp can hande this rep without any issues. What problems do you have? Are you referring to https://repmastered.app not being able to handle it? There's a 1MB replay file limit at repmastered.app, but screp can analyze this replay.

[icza]

Note: I could extract commands from the replay with the following command:

screp -cmds 2hv2AI\ 1hourStruggle@BGH.rep >a.txt

The result is an indented JSON file being approximately 400 MB. To make it more compact, you may disable JSON indenting:

screp -cmds -indent=false 2hv2AI\ 1hourStruggle@BGH.rep >b.txt

This will result in a file approximately 180 MB.

[captain-majid]

Note: I could extract commands from the replay with the following command:

screp -cmds 2hv2AI\ 1hourStruggle@BGH.rep >a.txt

Minifying json (-indent switch) works, thanks, but without it I get the following error, I've 8GB RAM with plenty of that free:

runtime: out of memory: cannot allocate 552992768-byte block (1703903232 in use)
fatal error: out of memory

runtime stack: runtime.throw(0x51eb9d, 0xd) c:/go/src/runtime/panic.go:616 +0x6b runtime.largeAlloc(0x20f5ddc3, 0x101, 0x124c140) c:/go/src/runtime/malloc.go:828 +0xdc runtime.mallocgc.func1() c:/go/src/runtime/malloc.go:721 +0x39 runtime.systemstack(0x0) c:/go/src/runtime/asm_386.s:464 +0x5e runtime.mstart() c:/go/src/runtime/proc.go:1175

goroutine 1 [running]: runtime.systemstack_switch() c:/go/src/runtime/asm_386.s:418 fp=0x13255da8 sp=0x13255da4 pc=0x446040 runtime.mallocgc(0x20f5ddc3, 0x4fc720, 0x13255e01, 0x40d7cd) c:/go/src/runtime/malloc.go:720 +0x67b fp=0x13255dfc sp=0x13255da8 pc=0x40cdfb runtime.makeslice(0x4fc720, 0x20f5ddc3, 0x20f5ddc3, 0x528614, 0x12684330, 0x40d848) c:/go/src/runtime/slice.go:61 +0x59 fp=0x13255e10 sp=0x13255dfc pc=0x434ed9 bytes.makeSlice(0x20f5ddc3, 0x0, 0x0, 0x0) c:/go/src/bytes/buffer.go:230 +0x61 fp=0x13255e2c sp=0x13255e10 pc=0x467171 bytes.(Buffer).grow(0x13710000, 0x1, 0x2) c:/go/src/bytes/buffer.go:144 +0x112 fp=0x13255e54 sp=0x13255e2c pc=0x466c72 bytes.(Buffer).WriteByte(0x13710000, 0x51c40a, 0x2, 0x4aa0d4) c:/go/src/bytes/buffer.go:268 +0x78 fp=0x13255e64 sp=0x13255e54 pc=0x467368 encoding/json.newline(0x13710000, 0x0, 0x0, 0x51c437, 0x2, 0x5) c:/go/src/encoding/json/indent.go:61 +0x26 fp=0x13255e84 sp=0x13255e64 pc=0x4a9606 encoding/json.Indent(0x13710000, 0x32c00000, 0xaff853a, 0x124d465f, 0x0, 0x0, 0x51c437, 0x2, 0x0, 0x0) c:/go/src/encoding/json/indent.go:97 +0x30a fp=0x13255ebc sp=0x13255e84 pc=0x4a996a encoding/json.(*Encoder).Encode(0x13255f9c, 0x512040, 0x1265bd00, 0x0, 0x1265bd00) c:/go/src/encoding/json/stream.go:219 +0x118 fp=0x13255f08 sp=0x13255ebc pc=0x4ab908 main.main() c:/go/src/github.com/icza/screp/cmd/screp/screp.go:189 +0x330 fp=0x13255fc8 sp=0x13255f08 pc=0x4ddf20 runtime.main() c:/go/src/runtime/proc.go:198 +0x1ea fp=0x13255ff0 sp=0x13255fc8 pc=0x426e3a runtime.goexit() c:/go/src/runtime/asm_386.s:1665 +0x1 fp=0x13255ff4 sp=0x13255ff0 pc=0x447b01

[icza]

I could probably optimize the memory usage, but since this replay is one in a million, I won't take any action on this. You may use the -indent=false option, and if you do need indented JSON, you can do that on the result JSON with a tool.

[captain-majid]

Is there is a size limit that screp can handle normally. (3MB .rep maybe?)

[icza]

There is no size limit. Probably the free memory is what limits what can be handled. There is no direct correlation that I can post, and it would probably change from release to release (also depending on the Go version used to build the binary, and the target platform).