search

TylerBrock / saw

Fast, multi-purpose tool for AWS CloudWatch Logs
MIT License
1.4k stars 77 forks source link

Panic when using prefix #25

Open ktilcu opened 5 years ago

ktilcu commented 5 years ago

Love the tool. I keep running into this error. The prefix definitely exists. 

-› saw get tag --prefix production
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1390c4c]

goroutine 1 [running]:
github.com/TylerBrock/saw/config.(*Configuration).TopStreamNames.func1(0x1, 0x0, 0x0)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/config/configuration.go:97 +0x2c
sort.doPivot_func(0xc000217b60, 0xc0002651e0, 0x0, 0x1a54, 0x0, 0x8)
    /usr/lib/go/src/sort/zfuncversion.go:83 +0xde
sort.quickSort_func(0xc000217b60, 0xc0002651e0, 0x0, 0x1a54, 0x1a)
    /usr/lib/go/src/sort/zfuncversion.go:143 +0x9a
sort.Slice(0x1460b20, 0xc0002651c0, 0xc000217b60)
    /usr/lib/go/src/sort/slice.go:21 +0x12f
github.com/TylerBrock/saw/config.(*Configuration).TopStreamNames(0x19063e0, 0xc000a45930, 0x136d160, 0xc00000c0f0)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/config/configuration.go:96 +0x82
github.com/TylerBrock/saw/config.(*Configuration).FilterLogEventsInput(0x19063e0, 0xc000860700)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/config/configuration.go:67 +0x35e
github.com/TylerBrock/saw/blade.(*Blade).GetEvents(0xc00000ae80)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/blade/blade.go:95 +0x54
github.com/TylerBrock/saw/cmd.glob..func2(0x19004e0, 0xc0000fc180, 0x1, 0x8)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/cmd/get.go:38 +0xaa
github.com/TylerBrock/saw/vendor/github.com/spf13/cobra.(*Command).execute(0x19004e0, 0xc0000fc080, 0x8, 0x8, 0x19004e0, 0xc0000fc080)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/vendor/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/TylerBrock/saw/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x19009a0, 0xc000080058, 0x0, 0x23)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/TylerBrock/saw/vendor/github.com/spf13/cobra.(*Command).Execute(0x19009a0, 0xc000113f88, 0xc000080058)
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
    /home/tbrock/Code/Go/src/github.com/TylerBrock/saw/saw.go:10 +0x2d
TylerBrock commented 5 years ago

Thanks for reporting this. It would be nice if we had a test suite so we could capture this as a regression test.

gary-cowell commented 5 years ago

Just to add a datum, I also see this but only for some prefixes.

ktilcu commented 5 years ago

I found some additional data. Saw seems to panic when there are too many matching streams. E.G. we have some ECS tasks that started thrashing. Each new task created a new stream. After a night of it, we have a few thousand streams. Now saw panics on every call.

gary-cowell commented 5 years ago

Confirmed. The ones that panic for me are the prefixes with many matches. I haven't been able to narrow down the exact breakpoint yet.

A working prefix for me has 1612 matches.

A failing one has 2611 matches.

TylerBrock commented 5 years ago

Thanks for the data points everyone! I think this might be happening because we are getting rate limited and we need to exponentially back-off when you have a large number of streams. Does that theory track?