WIP: lesson on message bus; new kafka/zookeeper images

[wouyang666]

High-Level Overview of this Curriculum Contribution:

< This is a brand new lesson to introduce kafka message bus. plan to have examples including: using local command line to produce and consume messages; using simple python script to produce and consume messages. No overlap. with this , for future lesson expansion, close loop automation can be added. ideally for basic introduction for automation engineers looking for a taste of kafka message bus>

Which issue(s) this PR fixes:

Fixes #

Is This Ready to Review?:

No. works-in-progress Draft. To test if the containers working fine. < No >

[wouyang666]

Apache Kafka is a distributed streaming platform.

work in progress. The final goal is to provide a close loop automation use case with some network deivce, an Analytics tool, Kafka message bus. while Healthbot may be too heavey to use here, I will look for other opensoure light weigh tool.

To begin with, I will start with introduction of Kafka, and then expand from there.

The intended content for kafka:

create topics in Kafka:

$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --topic test

list the current topics:

We'll begin by running kafka cli commands locally and see how the messages get produced and published. Let's start a kafka producer:

$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9093 --topic test

Run this snippet

Now, let's start a kafka consumer:

$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic test --from-beginning

Run this snippet

Now try to produce any message you want by typing in the window and observe in the consumer window to check what is received

hello kafka

Run this snippet

Now, go to the consumer tab to see if the message is received correctly.

Now let's try to see how can we do python to produce and consumer messages:

python3
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers=['ip:9092'])
data = b'{"host":"vmx101","ident":"cscript","message":"RPM_TEST_RESULTS: test-owner=northstar-ifl test-name=ge-0/0/6.0 loss=0 min-rtt=8 max-rtt=14 avgerage-rtt=19.67 jitter=29.2"}''
producer.send('test', value=data)

python3
from kafka import KafkaConsumer
consumer = KafkaConsumer(
    'test',
     auto_offset_reset='earliest',
     bootstrap_servers=['ip:9092']
     )
for message in consumer:
    message = message.value
    print(message)

[Mierdin]

Thanks for the above, but looks more like the content for the stage0 lesson guide, instead of a higher-level outline. Could you please update this PR description with a list of topics you want to address, separated into distinct stages, as a bulleted list.

[wouyang666]

Thanks for the comments.

I was trying to go through the things I went through when I was learning Kafka. Install Kafka --> Test/Run Kafka with Kafka native command --> Using Python to interactive with Kafka. Here we save users' effort in installing Kafka and go to step2 and step3 directly.

Those scripts actually come with Kafka installation. And they are the first few steps in the Kafka Quick start https://kafka.apache.org/quickstart . The script are not interacting with docker. Then run natively on the Kafka server and give users a quick view of how to produce and consume messages. If those steps are not desired, I can just focus on the Python-Kafka interaction part.

[Mierdin]

All I know is what I saw in the scripts; they seemed to heavily rely on access to the docker socket. In any case I think the last thing you said is a good plan. Focus on providing a solid, turnkey Kafka setup that's ready to go once the lesson starts, and the lesson itself can focus on interacting with it through Python, at least for starters.

I'd still like to see a high-level outline in the description though. From what I've seen thus far, Stage 0 should something like "Basic consumer/producer examples in Python". I'd bet we can get at least two more stages that build on these basic concepts, though - perhaps bridging into more advanced closed-loop automation?

[wouyang666]

I just tried building the kafka container, it looks there is no container inside the kafka docker container. The script under images/kafka are used for building the kafka image they are from the repo https://github.com/wurstmeister/kafka-docker . I tried to start the kafka container in kubernetes and it also works

[nre-labs-preview[bot]]

I've constructed a preview of your change, and it is available at https://preview-fgvthuc.nrelabs.io/

This is a fully-deployed version of the full NRE Labs site, so navigate to the lesson catalog to find your lesson.

Please note that this preview is only active for 12 hours

Preview Service Documentatation

Details

- **Commit**: cfd99fd89fafb31b15c671d9f4cf4cf7a8be78d1 - **Logs**: https://gist.github.com/18021ad077b4ffb91fde73b3fb4602b4 - **Inspect Traces**: https://inspect-preview.nrelabs.io/search?lookback=12h&operation=api_livelesson_request&service=preview-fgvthuc - **LivePreviewID**: fgvthuc

[Mierdin]

What's the latest on this? I would like to see more in the way of a detailed outline for what kind of content this lesson will tackle. I think we have a good start in the direction of using Python to interact with a message bus but there needs to be more meat there.

[wouyang666]

hi, Matt. I have updated the lesson content and changed it to python interaction only.

[Mierdin]

I'm not sure we're on the same page. I've asked a few times for a detailed outline of the concepts you intend to cover across the lesson, and thus far I have seen only a single stage's lesson guide, and even that is a bit anemic (covering only a handful of Python commands).

While I appreciate the work that has gone into building the images, a lesson needs to be much more than running a handful of commands. I am not convinced the learner will come away from this lesson having learned much of anything, in its current state.

I encourage you to read the Curriculum Quality Standards; that is where we maintain the set of principles that guide the development of the curriculum.