search

Yolean / kafka-test-failover

A test utility for https://github.com/Yolean/kubernetes-kafka/pull/78
1 stars 2 forks source link

Produce proper JSON formatted logs #1

Open solsson opened 7 years ago

solsson commented 7 years ago

This repo is partly an experiment for me with a Java microservice...

Currently evaluating https://github.com/jacek99/structlog4j. Alternatives include http://mvnrepository.com/artifact/net.logstash.log4j/jsonevent-layout and https://github.com/savoirtech/slf4j-json-logger.

We must allow kafka libs to log using regular slf4j, which might be an issue with structlog4j. And for now I haven't spotted how to get timestamp, level, class etc into the JSON object.

solsson commented 7 years ago

Current log output

[main] INFO org.apache.kafka.clients.producer.ProducerConfig - ProducerConfig values: 
    acks = all
    ...
    value.serializer = class org.apache.kafka.common.serialization.StringSerializer

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.11.0.1
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : c2a0d5f9b1f45bf5
[main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Discovered coordinator kafka-0.broker.kafka.svc.cluster.local:9092 (id: 2147483647 rack: null) for group test.
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking previously assigned partitions [] for group test
[main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining group test
[main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully joined group test with generation 1
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly assigned partitions [test-produce-consume-latency-0] for group test
[main] WARN se.yolean.kafka.test.failover.ProducerConsumerRun - {"message":"Interval insufficient","index":1,"duration":1750,"target":1000}
[main] INFO se.yolean.kafka.test.failover.ProducerConsumerRun - {"message":"consumed","offset":1,"timestamp":1508610385263,"key":"KTcjnmgcliy000000001","value":"1508610385262/2017-10-21T18:26:25.262+0000"}
[main] INFO se.yolean.kafka.test.failover.ProducerConsumerRun - {"message":"consumed","offset":2,"timestamp":1508610386263,"key":"KTcjnmgcliy000000002","value":"1508610386262/2017-10-21T18:26:26.262+0000"}

The multiline config printout is interesting, and very useful to humans using kubectl logs.