This is a Kafka Producer that generates mock Clickstream data for an imaginary e-commerce site. The scenario is that a user logs into the site, lands on the home page, browses the product catalog, lands on individual product pages, and either adds the product to cart or not and conitues to do that until the user either creates an order and does order checkout or abandons the session.
The producer generates the events in a user session in sequence but runs multiple threads to simulate multiple users hitting the site. The number of threads can be specified by using parameters. It utilizes a Schema Registry and generates Avro encoded events. It supports both the AWS Glue Schema Registry and a 3rd party Schema Registry.
For the AWS Glue Schema Registry, the producer accepts parameters to use a specific registry, pre-created schema name, schema description, compatibility mode (to check compatibility for schema evolution) and whether to turn on auto registration of schemas. If those parameters are not specified but using the AWS Glue Schema registry is specified, it uses the default schema registry. Some of the parameters may need to be specified if others are not. The AWS Glue Schema Registry provides open sourced serde libraries for serialization and deserialization which use the AWS default credentials chain (by default) for credentials and the region to construct an endpoint. For further information see the AWS Glue Schema Registry documentation.
For the 3rd party Schema Registry, the location of the Schema Registry needs to be specified in a producer.properties_msk file.
For each event generated, the Producer assigns a Global seq number that is unique across the threads and sequential. The Producer assigns a UserId as the partition key for the events sent to Apache Kafka which means that the events for the same User always go to the same Kafka partition which would allow stateful processing of user events in order. However, the Global seq numbers are spread out across multiple partitions in Apache Kafka. The highest Global seq number received by the consumer at any point can be utilized to figure out how far behind the producer, the consumer is or if the consumer is caught up with the producer.
This producer supports TLS in-transit encryption, TLS mutual authentication and SASL/SCRAM authentication with Amazon MSK. See the relevant parameters to enable them below.
This producer can be used to generate other types of events by modifying the RunProducer and Events classes.
This consumer depends on another library to get secrets from AWS Secrets Manager for SAS/SCRAM authentication with Amazon MSK. The library needs to be installed first before creating the jar file for the consumer.
git clone https://github.com/aws-samples/sasl-scram-secrets-manager-client-for-msk.git
cd sasl-scram-secrets-manager-client-for-msk
mvn clean install -f pom.xml
mvn clean package -f pom.xml
--gsrCompatibilitySetting or (-gcs): The compatibility setting for AWS Glue Schema Registry schema evolution. If not specified, uses the default setting for AWS Glue Schema Registry.
java -jar KafkaClickstreamClient-1.0-SNAPSHOT.jar -t ExampleTopic -pfp /tmp/kafka/producer.properties_msk -nt 8 -rf 300 -mtls
java -jar KafkaClickstreamClient-1.0-SNAPSHOT.jar -t ExampleTopic -pfp /tmp/kafka/producer.properties_msk -nt 8 -rf 300 -sse ssu nancy
java -jar KafkaClickstreamClient-1.0-SNAPSHOT.jar -t ExampleTopic -pfp /tmp/kafka/producer.properties_msk -nt 8 -rf 300 -iam
docker build . -t clickstream-producer:latest
docker run --rm -p 3800:3800 -e REGION=us-east-1 -e BROKERS=localhost:9092 -e TOPIC=click-stream -e BATCHSIZE=16384 -e LINGERMS=0 -e BUFFERMEMORY=33554432 -e NMTRD=1 clickstream-producer