Concurrent Map Read / Write

[khaf]

I'm getting this error while registering the driver for ql:

fatal error: concurrent map read and map write

goroutine 611292 [running]:
runtime.throw(0xd18e15, 0x21)
    /usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc427d15650 sp=0xc427d15630
runtime.mapaccess1_fast64(0xc1f8a0, 0xc426254ff0, 0x18, 0xc4249c9440)
    /usr/local/go/src/runtime/hashmap_fast.go:119 +0x1a8 fp=0xc427d15678 sp=0xc427d15650
github.com/citrusleaf/amc/vendor/github.com/cznic/lldb.(*bitFiler).ReadAt(0xc42700cb80, 0xc428a2dcc0, 0x10, 0x10, 0x18600, 0xc424922e80, 0xc420018000, 0x7acc14)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/lldb/xact.go:148 +0x105 fp=0xc427d15728 sp=0xc427d15678
github.com/citrusleaf/amc/vendor/github.com/cznic/lldb.(*RollbackFiler).ReadAt(0xc42012bd80, 0xc428a2dcc0, 0x10, 0x10, 0x18600, 0x0, 0x0, 0x0)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/lldb/xact.go:538 +0x15b fp=0xc427d15780 sp=0xc427d15728
github.com/citrusleaf/amc/vendor/github.com/cznic/lldb.(*Allocator).read(0xc4200106e0, 0xc428a2dcc0, 0x10, 0x10, 0x18600, 0x185a, 0x40d9df)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/lldb/falloc.go:973 +0x65 fp=0xc427d157e8 sp=0xc427d15780
github.com/citrusleaf/amc/vendor/github.com/cznic/lldb.(*Allocator).Get(0xc4200106e0, 0x0, 0x0, 0x0, 0x185a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/lldb/falloc.go:697 +0x2b2 fp=0xc427d15920 sp=0xc427d157e8
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*file).Read(0xc420068180, 0x0, 0x0, 0x0, 0x185a, 0xc425b2d680, 0xb, 0xb, 0x1, 0x2, ...)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/file.go:812 +0x6b fp=0xc427d15a10 sp=0xc427d15920
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*table).row0(0xc42475c000, 0xc426732720, 0x185a, 0x185a, 0x0, 0x0, 0x8fce80, 0xc428fc3b00)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/storage.go:769 +0x96 fp=0xc427d15ac8 sp=0xc427d15a10
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*table).row(0xc42475c000, 0xc426732720, 0x185a, 0xc4246fdc50, 0x0, 0xc428a4bb90, 0x0, 0x0, 0x0)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/storage.go:782 +0x43 fp=0xc427d15b18 sp=0xc427d15ac8
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*indexPlan).doEq(0xc422239860, 0xc426732720, 0xc428fc3a40, 0x40, 0xc8f780)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/plan.go:430 +0x23f fp=0xc427d15bd8 sp=0xc427d15b18
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*indexPlan).do(0xc422239860, 0xc426732720, 0xc428fc3a40, 0xb, 0xc427e6e600)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/plan.go:596 +0x134 fp=0xc427d15c20 sp=0xc427d15bd8
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*filterDefaultPlan).do(0xc428fc20c0, 0xc426732720, 0xc4240dfb20, 0xc4259bbf00, 0x2)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/plan.go:1460 +0x15e fp=0xc427d15c80 sp=0xc427d15c20
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*orderByDefaultPlan).do(0xc425c3a820, 0xc426732720, 0xc428fc3640, 0x0, 0x0)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/plan.go:1926 +0x35f fp=0xc427d15d90 sp=0xc427d15c80
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*limitDefaultPlan).do(0xc428fc2100, 0xc426732720, 0xc4259bb6c0, 0xb, 0xc422d9b6a8)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/plan.go:1794 +0x170 fp=0xc427d15de8 sp=0xc427d15d90
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*DB).do(0xc420110c60, 0xc426732720, 0x10bd780, 0xc428fc2100, 0x0, 0xc427e568d0, 0x0, 0x0)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/ql.go:1419 +0x15f fp=0xc427d15e48 sp=0xc427d15de8
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.recordset.Do(0xc426732720, 0x10bd780, 0xc428fc2100, 0x0, 0x0, 0xc427e568d0, 0x1, 0xc427e568d0)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/ql.go:75 +0x68 fp=0xc427d15e98 sp=0xc427d15e48
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.(*recordset).Do(0xc426732a60, 0xc427e56800, 0xc427e568d0, 0xc42a1fcc88, 0xbcc1c0)
    <autogenerated>:60 +0x7b fp=0xc427d15ee8 sp=0xc427d15e98
github.com/citrusleaf/amc/vendor/github.com/cznic/ql.newdriverRows.func1(0xc426732ac0)
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/driver.go:424 +0xa6 fp=0xc427d15fd8 sp=0xc427d15ee8
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc427d15fe0 sp=0xc427d15fd8
created by github.com/citrusleaf/amc/vendor/github.com/cznic/ql.newdriverRows
    /work/source/src/github.com/citrusleaf/amc/vendor/github.com/cznic/ql/driver.go:432 +0xf1

This happens under no real load; there may be record being read, but I'm not even sure of that.

Is there something I'm doing wrong on my end? Is this known? Is there a work around until the issue is found and solved?

[cznic]

Is there something I'm doing wrong on my end? Is this known?

To answer that a reproduction case would be very helpful. Can you please provide a small, self-contained one?

[khaf]

This is happening on a customer's server. As far as I can tell, the server is waiting for connections, and no database activity is going on.

I can't reproduce this on my end. From the logs, the particular report I just posted above happens right after the server comes up and the app tries to open and read the database.

[cznic]

Any chance to use a -race enabled build? That could possibly reveal the other part of the race condition.

[cznic]

@khaf Gentle ping.

[khaf]

We worked around this issue by asking the customer to delete their db file and trying again.

It seems that a db corruption was causing this, but we couldn’t investigate more.

Thanks for your support. I’ll let you know if I get more info. I think we can close this ticket for now.

On May 7, 2018, at 4:22 PM, cznic notifications@github.com wrote:

@khaf Gentle ping.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cznic]

Ok, thanks for the feedback. Feel free to (re)open if needed.