shenwei356 / rush

A cross-platform command-line tool for executing jobs in parallel
https://github.com/shenwei356/rush
MIT License
846 stars 63 forks source link

Occasional crash when using the immediate-output flag #52

Open jarrodharvey opened 8 months ago

jarrodharvey commented 8 months ago

Prerequisites

Checking new version... You are using the latest version of rush

Describe your issue

The error looks like the below.

panic: runtime error: slice bounds out of range [7998:7994]

goroutine 81 [running]:
github.com/shenwei356/rush/process.(*ImmediateLineWriter).WritePrefixedLines(0xc0003d0000, {0xc0003ba000, 0x1f3e}, 0xc0000ca008)
        /home/shenwei/go/src/github.com/shenwei356/rush/process/process.go:378 +0x625
github.com/shenwei356/rush/process.ImmediateWriter.Write(...)
        /home/shenwei/go/src/github.com/shenwei356/rush/process/process.go:422
io.copyBuffer({0x1408100, 0xc00038c070}, {0x14084e0, 0xc0003e6008}, {0x0, 0x0, 0x0})
        /usr/local/go/src/io/io.go:429 +0x204
io.Copy(...)
        /usr/local/go/src/io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1()
        /usr/local/go/src/os/exec/exec.go:560 +0x3a
os/exec.(*Cmd).Start.func2(0x0?)
        /usr/local/go/src/os/exec/exec.go:717 +0x32
created by os/exec.(*Cmd).Start
        /usr/local/go/src/os/exec/exec.go:716 +0xab3

Unfortunately, the python script that rush is calling is my company's IP so I cannot provide it to support repro. I can, however, provide the rush command that I'm using if that helps.

cat parallel.txt | rush 'python main.py find_results/{}.txt extraction_results/{} debug' -c -I --eta

Note that it seems to work well for a while after running this command before failing at a seemingly random point in time.

shenwei356 commented 8 months ago

Here's a quick fix.

jarrodharvey commented 8 months ago

Amazing thank you Wei that was fast. Appreciate the effort.

Unfortunately I'm on Windows, and the python script only works on windows, so can't make use of these. Looking forward to the windows fix once it's ready.

shenwei356 commented 8 months ago

@jarrodharvey Oh, I'm sorry for forgetting the Windows version. It's updated now.