Panic on querying

[gouthamve]

panic: runtime error: index out of range [44032] with length 44032
goroutine 35 [running]:
github.com/logv/sybil/src/lib.(*TableBlock).unpackIntCol(0xc0001ec120, 0x6307c0, 0xc000171960, 0xc00000ac00, 0xc000174c60, 0xc000175e00)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/column_store_io.go:705 +0x9a2
github.com/logv/sybil/src/lib.(*Table).LoadBlockFromDir(0xc0000d6000, 0xc000252a80, 0x24, 0xc0001fdf80, 0x0, 0x0)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_block_io.go:256 +0x58e
github.com/logv/sybil/src/lib.(*Table).LoadAndQueryRecords.func1(0xc000165550, 0xc0000d6000, 0xc000252a80, 0x24, 0xc00023c000, 0xc000165c90, 0xc0001fdf80, 0xc000165c68, 0xc000165c98, 0xc000246480, ...)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_query.go:127 +0xaf9
created by github.com/logv/sybil/src/lib.(*Table).LoadAndQueryRecords
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_query.go:105 +0x958

This was seen at commit 0cf4402c2b76d21302cf4811f250c59fd68184b7

[gouthamve]

Another panic, but slightly different line:

panic: runtime error: index out of range [43008] with length 43008

goroutine 246 [running]:
github.com/logv/sybil/src/lib.(*TableBlock).unpackIntCol(0xc000510120, 0x6307c0, 0xc0004061a0, 0xc00000a800, 0xc0003163f0, 0xc0002be5a0)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/column_store_io.go:729 +0x982
github.com/logv/sybil/src/lib.(*Table).LoadBlockFromDir(0xc0000f2000, 0xc000464780, 0x24, 0xc00052c340, 0x0, 0x0)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_block_io.go:256 +0x58e
github.com/logv/sybil/src/lib.(*Table).LoadAndQueryRecords.func1(0xc000497540, 0xc0000f2000, 0xc000464780, 0x24, 0xc0000f0100, 0xc000497d40, 0xc00052c340, 0xc000497d18, 0xc000497d48, 0xc0004b9860, ...)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_query.go:127 +0xaf9
created by github.com/logv/sybil/src/lib.(*Table).LoadAndQueryRecords
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_query.go:105 +0x958\n"

[gouthamve]

Even more difference, and I think likely different issue:

panic: runtime error: index out of range [46506] with length 43008

goroutine 539 [running]:
github.com/logv/sybil/src/lib.(*TableBlock).unpackStrCol(0xc000f2a360, 0x6307c0, 0xc000e04020, 0xc00000a800, 0xc000be1410, 0xc000be1d40)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/column_store_io.go:567 +0xd74
github.com/logv/sybil/src/lib.(*Table).LoadBlockFromDir(0xc0000ea000, 0xc0013cf7a0, 0x24, 0xc00184c180, 0x0, 0x0)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_block_io.go:252 +0x69e
github.com/logv/sybil/src/lib.(*Table).LoadAndQueryRecords.func1(0xc000220180, 0xc0000ea000, 0xc0013cf7a0, 0x24, 0xc0000e8100, 0xc0004c0bf0, 0xc00184c180, 0xc0004c0b88, 0xc0004c0bf8, 0xc000c54f60, ...)
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_query.go:127 +0xaf9
created by github.com/logv/sybil/src/lib.(*Table).LoadAndQueryRecords
    /home/goutham/go/src/github.com/logv/sybil/src/lib/table_query.go:105 +0x958

[tmc]

Can you write a test case exposing this?

[okayzed]

we synced offline, it has to do with a block changing size during query because a digestion happened simultaneously

there's a few ways to fix it

goutham has been really helpful in exposing bugs as the data becomes large (500+mm records) so i have a few things I'll diff on this weekend and next

On Sat, Jan 25, 2020, 11:10 AM Travis Cline notifications@github.com wrote:

Can you write a test case exposing this?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/logv/sybil/issues/107?email_source=notifications&email_token=AAAYCOMD26EYD7ASODPZCLTQ7SFCPA5CNFSM4KLIJ5W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ5DJ5Y#issuecomment-578434295, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAYCOLNSZKSH5I556Y3H6LQ7SFCPANCNFSM4KLIJ5WQ .

[okayzed]

https://github.com/logv/sybil/commit/8a7f0edfa1f4bf6abe7bdec637578afb3a84cefd should fix this.

It addresses all columns and both delta encoded and non-delta encoded scenarios with tests. Just figuring out when to push it to master - would be nice if you can test it locally and let me know if it works before I push to master, otherwise I'll push it in a few days

[okayzed]

I haven't heard a response here, but I believe this fixes the problem still