search

abdullin / cellar

Append-only storage in Go designed for the analytical workloads
https://abdullin.com/bitgn/real-time-analytics/
BSD 3-Clause "New" or "Revised" License
62 stars 6 forks source link

update to github.com/pierrec/lz4@v2.0.7 #5

Closed KaiserKarel closed 5 years ago

KaiserKarel commented 5 years ago

Seems that there have been some improvements in the compression algo (we are on v1.0.1). However updating requires changing two things

three references to CompressionLevel need to change

// current version
zw := lz4.NewWriter(w)
zw.Header.HighCompression = true

// new version
zw := lz4.NewWriter(w)
zw.Header.CompressionLevel = CompressionLevel

// where CompressionLevel is derived from a global var which may be set
var CompressionLevel int

// compression.go
func init()  {
    viper.BindEnv("LZ4_COMPRESSION_LEVEL")
    viper.SetDefault("LZ4_COMPRESSION_LEVEL", 40)

    CompressionLevel = viper.GetInt("LZ4_COMPRESSION_LEVEL")
}

Also test need to be updated. This alteration will cause writer test to periodically fail with either:

--- FAIL: TestFuzz (0.98s)
    writer_test.go:270: maxVal 92765; maxBuffer 26264139; seed 1539774761052879507
panic: Read 4194304 bytes but expected 26221599 [recovered]
        panic: Read 4194304 bytes but expected 26221599

goroutine 6 [running]:
testing.tRunner.func1(0xc0000e8200)
        /usr/local/go1.11.1.linux-amd64/src/testing/testing.go:792 +0x387
panic(0x8760e0, 0xc0000921c0)
        /usr/local/go1.11.1.linux-amd64/src/runtime/panic.go:513 +0x1b9
log.Panicf(0x921509, 0x1d, 0xc00088da28, 0x2, 0x2)
        /usr/local/go1.11.1.linux-amd64/src/log/log.go:333 +0xda
github.com/carapace/cellar.loadChunkIntoBuffer(0xc0000a2300, 0x2d, 0xc0009d0080, 0x10, 0x10, 0x1901c1f, 0xc004d04000, 0x1901c1f, 0x1901c1f, 0x0, ...)
        /home/karel/go/src/github.com/abdullin/cellar/reader.go:283 +0x484
github.com/carapace/cellar.(*Reader).Scan(0xc00088def0, 0xc00088de68, 0x0, 0x0)
        /home/karel/go/src/github.com/abdullin/cellar/reader.go:135 +0x4ba
github.com/carapace/cellar.TestFuzz(0xc0000e8200)
        /home/karel/go/src/github.com/abdullin/cellar/writer_test.go:311 +0x3e5
testing.tRunner(0xc0000e8200, 0x9330a0)
        /usr/local/go1.11.1.linux-amd64/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
        /usr/local/go1.11.1.linux-amd64/src/testing/testing.go:878 +0x353
FAIL    github.com/carapace/cellar      1.205s

Or 

--- FAIL: TestFuzz (0.00s)
    writer_test.go:270: maxVal 110817; maxBuffer 30725211; seed 1539774415500550257
panic: runtime error: index out of range [recovered]
        panic: runtime error: index out of range

goroutine 8 [running]:
testing.tRunner.func1(0xc0000ce200)
        /usr/local/go1.11.1.linux-amd64/src/testing/testing.go:792 +0x387
panic(0x8a2480, 0xe88f90)
        /usr/local/go1.11.1.linux-amd64/src/runtime/panic.go:513 +0x1b9
github.com/carapace/cellar.TestFuzz(0xc0000ce200)
        /home/karel/go/src/github.com/abdullin/cellar/writer_test.go:297 +0xb28
testing.tRunner(0xc0000ce200, 0x9330a0)
        /usr/local/go1.11.1.linux-amd64/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
        /usr/local/go1.11.1.linux-amd64/src/testing/testing.go:878 +0x353
FAIL    github.com/carapace/cellar      0.193s

I don't really have enough knowledge of the db to fix the tests/verify if it is compatible with lz4@v2.

abdullin commented 5 years ago

That should be fixed now. By default the compression level is 10. You could override it from your code by setting the cellar.SetCompressionLevel(level)

KaiserKarel commented 5 years ago

I'm still getting 

    writer_test.go:270: maxVal 95684; maxBuffer 20267182; seed 1540218988936092485
panic: runtime error: index out of range [recovered]
    panic: runtime error: index out of range
goroutine 11 [running]:
testing.tRunner.func1(0xc0000be100)
    /home/travis/.gimme/versions/go1.11.linux.amd64/src/testing/testing.go:792 +0x6a7
panic(0x70f0a0, 0xb19e10)
    /home/travis/.gimme/versions/go1.11.linux.amd64/src/runtime/panic.go:513 +0x1b9
github.com/carapace/cellar.TestFuzz(0xc0000be100)
    /home/travis/gopath/src/github.com/carapace/cellar/writer_test.go:297 +0xe11
testing.tRunner(0xc0000be100, 0x74ff20)
    /home/travis/.gimme/versions/go1.11.linux.amd64/src/testing/testing.go:827 +0x163
created by testing.(*T).Run
    /home/travis/.gimme/versions/go1.11.linux.amd64/src/testing/testing.go:878 +0x651
FAIL    github.com/carapace/cellar  1.772s

when trying to merge with my branch. Any clue why this is happening?