How you can contribute today

travisjeffery commented 6 years ago

Thank you for wanting to hack on Jocko and for your contributions.

Why you should hack on Jocko:

You'll learn a lot. I learned a ton writing the initial version of Jocko. A helluva more than had I spent the same time reading what someone else had made or put in a textbook, likely having never made anything themselves. When you do need to look up something in a textbook, and you'll want to at times, you'll have a real need for the knowledge - it'll make the difference in your favor of solving your problem at hand.
It's good for the resume. In the first few months of Jocko's life a contributor leveled up in their career and hacking on Jocko helped make that happen. You'll have Go, Kafka, distributed, and big data systems experience.
It's fun. Especially if you're looking for a non-Java big data project to hack on, you have few options. I dunno about you but I'd rather have someone slam my head in a door than hack on Java in my spare time – especially on a big old decrepit code base. Here we've got a young Go code base, nimble, and hungry to be developed into something big.

Be sure to join the #jocko room in the Gopher's Slack.

Some tracts of work you can hack on:

Tests

Who should work on this? If you want something to get started hacking, something quick, and learn how Jocko works.

Adding more unit and performance tests in-particular would be peachy.
We need a test tool like kafka-producer-perf-test.
Squash the bugs.
(Documentation too where needed, but try to write code clear enough it doesn't need docs.)

API versioning

Who should work on this? After you've hacked on tests and know Jocko's code, this is the next step.

This way we can handle other clients. Jocko was developed to test against Sarama but works well with kafkacat too.
An example to look at how to implement is how Sarama does it.

Consumer groups

Who should work on this? Don't go here unless you've got the time and desire to dig deep. This is a good chunk of work.

Detailed design doc.

Replication

What I'm current working on.
So far I've got follower brokers consuming/replicating from the leaders.
The current work I'm doing is on managing/updating ISRs.

How to hack on Jocko

Before developing a feature, read the relevant Kafka design proposal, which you'll often find by Googling "Kafka design proposal the feature" or searching for it on this wiki. Reading the code will also help.
Run unit tests goes as usual, just go test.
To spin up Jocko and run/test a client against it: First spin up a cluster as documented. Then point your client at it. You'll likely find kafkacat useful.

Example:

# Spin up a cluster
$ go run cmd/jocko/main.go --debug \
          --logdir="/tmp/jocko1" \
          --brokeraddr=127.0.0.1:9001 \
          --raftaddr=127.0.0.1:9002 \
          --serfaddr=127.0.0.1:9003 \
          --id=1

$ go run cmd/jocko/main.go --debug \
          --logdir="/tmp/jocko2" \
          --brokeraddr=127.0.0.1:9101 \
          --raftaddr=127.0.0.1:9102 \
          --serfaddr=127.0.0.1:9103 \
          --serfmembers=127.0.0.1:9003 \
          --id=2

$ go run cmd/jocko/main.go --debug \
          --logdir="/tmp/jocko3" \
          --brokeraddr=127.0.0.1:9201 \
          --raftaddr=127.0.0.1:9202 \
          --serfaddr=127.0.0.1:9203 \
          --serfmembers=127.0.0.1:9003 \
          --id=3

# Create a topic
$  go run cmd/createtopic/main.go --brokeraddr localhost:9001 --topic test_topic --partitions 8

# Make requests you need (e.g. get the metadata)
$ kafkacat -L -b localhost:9201 -t test_topic

# Does it work???

skotchpine commented 6 years ago

I've found the Kafka design docs to be a good intro. @travisjeffery, how close to this do you plan to implement?

travisjeffery commented 6 years ago

@skotchpine closely, I want Kafka clients/libaries to be compatible with Jocko.

skotchpine commented 6 years ago

OH boy. Is it reasonable to integration test some of these possible systems?

alexellis commented 6 years ago

@travisjeffery can you ©get away with a single node for testing publishing and receiving messages? What would that look like as an example?

travisjeffery commented 6 years ago

@alexellis yeah you can get away with it. I'll put up an example tomorrow. What's the use case? Testing or a small dev project?

alexellis commented 6 years ago

This would be for testing the Kakfa connector for OpenFaaS - I'm presenting OpenFaaS KubeCon next week. One of the things we're building is a Kakfa connector/trigger for functions. It would be great to collaborate on the connector with Jocko. For production deployments I think people will have Kakfa in-place already.

travisjeffery / jocko