cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

stability: crash with zero transaction timestamp in EvalContext #5333

Closed mberhault closed 8 years ago

mberhault commented 8 years ago

build sha (latest as of time of this issue): 1df05137c01a75933b462ee717e9e035afde5b55 just started up 4 clean nodes and launched photos app. within seconds, first two nodes (also the order in which the photos apps get started talking to localhost) died with:

I160317 00:57:08.950299 storage/replica_command.go:1615  initiating a split of range=6 [/Table/50-/Max) at key /Table/51
W160317 00:57:08.963951 storage/intent_resolver.go:272  [node=1,store=1,range=1] failed to push during intent resolution: failed to push "storage/replica_command.go:1617 (*Replica).AdminSplit" id=586591b9 key=/Local/Range/"\xbb"/RangeDescriptor rw=false pri=0.00310461 iso=SERIALIZABLE stat=PENDING epo=0 ts=1458176228.950367949,0 orig=1458176228.950367949,0 max=1458176228.950367949,0
I160317 00:57:09.047590 storage/replica_command.go:1297  range 7: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 405048h57m9.979284286s
I160317 00:57:09.053194 storage/replica_command.go:1615  initiating a split of range=7 [/Table/51-/Max) at key /Table/52
W160317 00:57:09.064218 storage/intent_resolver.go:272  [node=1,store=1,range=1] failed to push during intent resolution: failed to push "storage/replica_command.go:1617 (*Replica).AdminSplit" id=9e609eee key=/Local/Range/"\xbc"/RangeDescriptor rw=false pri=0.06811023 iso=SERIALIZABLE stat=PENDING epo=0 ts=1458176229.053275702,0 orig=1458176229.053275702,0 max=1458176229.053275702,0
panic: zero transaction timestamp in EvalContext

To see all goroutines, visit https://github.com/maruel/panicparse#GOTRACEBACK

1: running [Created by util.ServeHandler.func3 @ net.go:125]
           panic.go:464     panic(0x1497420, 0xc820a39b10)
    parser eval.go:767      (*EvalContext).GetTxnTimestamp(0xc820885120, 0, 0xc800000000)
    parser builtins.go:1096 glob.func56(0x1, 0, 0, 0, 0, 0, #2, #7, #6, 0x26c0420, ...)
    parser eval.go:1170     (*FuncExpr).Eval(0xc820a2e7e0, 0x1, 0, 0, 0, 0, 0, #2, #7, #6, ...)
    parser eval.go:1413     evalExprs(0x1, 0, 0, 0, 0, 0, #2, #7, #6, 0xc820a4c8c0, ...)
    parser eval.go:1429     (*Tuple).Eval(0xc820885690, 0x1, 0, 0, 0, 0, 0, #2, #7, #6, ...)
    sql    values.go:68     (*planner).ValuesClause(#4, #8, 0, 0, 0x1)
    sql    plan.go:177      (*planner).makePlan(#4, 0x7f1f21cb74c0, #8, 0x7f1f21cb7400, 0, 0, 0x8)
    sql    insert.go:116    (*planner).Insert(#4, #5, 0xe984f100, 0, 0, 0xc800000001)
    sql    plan.go:189      (*planner).prepare(#4, 0x7f1f21cb7378, #5, 0, 0, 0)
    sql    executor.go:302  (*Executor).Prepare(0xc82000cc40, 0xc820809032, 0x4, 0xc8205b8707, 0x40, 0xc8200c44c0, #6, 0, 0, 0, ...)
    pgwire v3.go:326        (*v3Conn).handleParse(#1, 0xc8200c43a8, 0, 0)
    pgwire v3.go:251        (*v3Conn).serve(#1, 0xc82030b0e0, 0, 0)
    pgwire server.go:159    (*Server).ServeConn(0xc8202b1bf8, 0x7f1f21cc0a58, 0xc820784600, 0, 0)
    server server.go:292    (*Server).Start.func4.1(#9, #3)
    util   net.go:123       ServeHandler.func3.1(0xc820122600, #9, #3, 0xc8202ac490)

Full node logs: node0.log.parse.txt node1.log.parse.txt node2.log.parse.txt node3.log.parse.txt

tbg commented 8 years ago

Was just about to suggest @knz as well.