Add support for MDC output in logs

Apache Kafka 2.3 added support for MDC in logs. For reasons of backward compatibility, it is disabled in log output by default.

This PR adds a new env variable to the Kafka Connect base image so that users can enable the MDC output in logs.

Changes

CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN added as an optional environment variable. If present, the value provided is used in log4j.appender.stdout.layout.ConversionPattern of connect-log4j.properties

Testing done

test 01 : No change to existing environment variables

Log config connect-log4j.properties generated exactly as before

$ docker-compose up -d kafka-connect-01; sleep 30; docker exec kafka-connect-01 cat /etc/kafka/connect-log4j.properties
zookeeper is up-to-date
kafka is up-to-date
schema-registry is up-to-date
Recreating kafka-connect-01 ... done

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

# loggers from CONNECT_LOG4J_LOGGERS env variable
log4j.logger.org.reflections=ERROR
log4j.logger.org.apache.kafka.connect.runtime.rest=WARN

test 02 : Add env var `CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN`

      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"

Log config connect-log4j.properties generated with specified pattern

$ docker-compose up -d kafka-connect-01; sleep 30; docker exec kafka-connect-01 cat /etc/kafka/connect-log4j.properties
zookeeper is up-to-date
kafka is up-to-date
schema-registry is up-to-date
Recreating kafka-connect-01 ... done

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %X{connector.context}%m (%c:%L)%n

# loggers from CONNECT_LOG4J_LOGGERS env variable
log4j.logger.org.reflections=ERROR
log4j.logger.org.apache.kafka.connect.runtime.rest=WARN

MDC information present in generated log

[2019-07-16 14:04:49,706] INFO [sink-elastic-orders-00|task-0] Kafka version: 5.3.0-ccs-SNAPSHOT (org.apache.kafka.common.utils.AppInfoParser:117)
[2019-07-16 14:04:49,707] INFO [sink-elastic-orders-00|task-0] Kafka commitId: 8df177bde86678d9 (org.apache.kafka.common.utils.AppInfoParser:118)
[2019-07-16 14:04:49,707] INFO [sink-elastic-orders-00|task-0] Kafka startTimeMs: 1563285889702 (org.apache.kafka.common.utils.AppInfoParser:119)

confluentinc / cp-docker-images