search

travisjeffery / jocko

Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)
https://twitter.com/travisjeffery
MIT License
4.94k stars 363 forks source link

Errors in `topic create` don't propagate to client #135

Open candlerb opened 6 years ago

candlerb commented 6 years ago

If an error occurs in topic create, the client thinks it was successful. A simple way to trigger this is to create a topic with empty name:

# cmd/jocko/jocko topic create --topic ""
created topic:
# echo $?
0

But at the server side you get:

2018-06-22T05:56:26.119Z    ERROR   fsm/commands.go:97  EnsurePartition failed  {"id": 0, "broker addr": "0.0.0.0:9092", "serf addr": "0.0.0.0:9094", "raft addr": "127.0.0.1:9093", "id": 0, "raft addr": "127.0.0.1:9093", "error": "failed inserting partition: object missing primary index"}
github.com/travisjeffery/jocko/log.(*logger).Error
    /root/go/src/github.com/travisjeffery/jocko/log/logger.go:38
github.com/travisjeffery/jocko/jocko/fsm.(*FSM).applyRegisterPartition
    /root/go/src/github.com/travisjeffery/jocko/jocko/fsm/commands.go:97
github.com/travisjeffery/jocko/jocko/fsm.New.func1
    /root/go/src/github.com/travisjeffery/jocko/jocko/fsm/fsm.go:81
github.com/travisjeffery/jocko/jocko/fsm.(*FSM).Apply
    /root/go/src/github.com/travisjeffery/jocko/jocko/fsm/fsm.go:98
github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1
    /root/go/src/github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft/fsm.go:57
github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft.(*Raft).runFSM
    /root/go/src/github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft/fsm.go:120
github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft.(*Raft).(github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft.runFSM)-fm
    /root/go/src/github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft/api.go:506
github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1
    /root/go/src/github.com/travisjeffery/jocko/vendor/github.com/hashicorp/raft/state.go:146

Furthermore, this error is persistent: each time you kill and restart the broker, the same error appears again (presumably it is re-running the commit log?)

I don't know the correct approach to this problem. Is raft Apply required to succeed always? In which case, maybe the only solution is additional pre-flight checks before sending the change to raft. If so, what are the rules for validating the topic?

But I think there is a deeper issue with error handling. For example: topic create has a pre-flight check to reject a topic which already exists. But this leads to a race condition: what happens if two clients try to create the topic at the same time, both pre-flight checks pass, and then both commit?