mirrormaker recipe - Githubissues

jhohertz commented 8 years ago

For your consideration, this pull request has one main feature, which is the addition of a recipe to allow an instance of mirror maker to be configured and launched.

In addition, to reduce duplicity in code, the existing recipes were refactored to put the common elements into a common recipe, and the install has been isolated into it's own recipe.

Because we need a second suite to effectively test the setup, this has been added to the test kitchen, which provisions a second kafka instance as well as a mirror. A test has been added that will check for message propagation through the mirror to the second instance. This requires using kitchen verify (which leaves prior suites running) vs. kitchen test (which does not). The alternative is to remove the tests, or allow multiple instances of kafka on a single node, neither of which was deemed appropriate.

Very open to any feedback/changes to get this fork into a condition to merge to the main project. Just let me know what you'd like to see different.

Thanks!

bbaugher commented 8 years ago

Could we point the source and target to the same kafka cluster for testing?

jhohertz commented 8 years ago

Thanks for the comments! I will review and update or prep a new cleaner pull as appropriate, I think some of it is regression as I had split things out well before current... sorry about that.

Re: one cluster for source+target, the complication there I think we'd need some function in mirror maker to prefix or otherwise translate the topic name sourced to a different topic name, which I haven't yet seen. (though it may be there somewhere). It would also need to filter on the same prefix lest a wildcard whitelist create endless loops of mirroring into recursively prefixed topics.

bbaugher commented 8 years ago

I wouldn't worry about an endless loop and point it at some non existing topic. We just want to verify we can configure and run it in our tests

mkwhitacre commented 8 years ago

@jhohertz can you include adding yourself to the contributors files as well with your updated pull?

[1] - https://github.com/cerner/cerner_kafka#contributing

jhohertz commented 8 years ago

All the minor stuff is in-branch now. What I have left to address:

[ ] tests / kitchen simplification (one suite)
[ ] pull the config assertions out?
[ ] attribute namespace changes (this will be a little while yet as I'll need to be updating our own wrappers/configs in tandem)

cerner / cerner_kafka

mirrormaker recipe #35