KAFKA_MESSAGE_MAX_BYTES does not works

[black-lotus]

Hello, I have issue to increase message.max.bytes I read the documention about add KAFKA_MESSAGE_MAX_BYTES to environment

kafka:
    image: wurstmeister/kafka:latest
    ports:
      - 9092:9092
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_MESSAGE_MAX_BYTES: 2000000000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

i tried to produce some big message on kafka console

and when i listner via consomer console, it get truncted

Thank you

[sscaling]

Have you configured the consumer's fetch size appropriately (https://kafka.apache.org/documentation/) ?

p.s. it helps to include the text output so this can easily be copy/pasted and parsed when investigating an issue. For example instead of images, I could have just counted the characters in the the response to see if that matched an existing default image.

Also, is the original message actually larger than the standard max message size? Are you sure this isn't just a kafka-console-consumer formatting issue rather than a data storage issue?

[black-lotus]

@sscaling I didn't configure consumer, but in my app (spring app) i've configure the consumer with max data for about maxRequestSizeConfig=987654300 and still truncated

If you want to reproduce the message, here you go

{"mandatoryRequest":{"storeId":"AAAA","channelId":"WEB","requestId":"123123","serviceId":"gateway","username":"username"},"integratorBookRequest":{"distributionType":"supplier","adult":1,"child":1,"infant":1,"promoCode":"","viaSelect":"","flightSelect":"","fareSellKey":"","journeySellKey":"","sessionGroup":"","sessionGroupType":"","currency":"IDR","countryId":"ID","account":{"name":"aaa.com","code":"tix"},"contact":{"profileId":"123","firstName":"dondon","lastName":"dev","dob":"2000-01-01","phoneCode":"62","phoneNumber":"123123123","title":"mr","gender":"male","email":"dondon.dev@mail.com"},"paxes":[{"profileId":"123","type":"adult","title":"mr","gender":"male","firstName":"dondon","lastName":"dev","nationality":"ID","dob":"2000-01-01","passport":null,"baggage":null,"frequentFlyer":""},{"profileId":"123","type":"child","title":"mr","gender":"male","firstName":"doni","lastName":"dev","nationality":"ID","dob":"2010-01-01","passport":null,"baggage":null,"frequentFlyer":""},{"profileId":"123","type":"infant","title":"mr","gender":"male","firstName":"purba","lastName":"dev","nationality":"ID","dob":"2018-01-01","passport":null,"baggage":null,"frequentFlyer":""}],"itineraries":[{"airline":"Garuda Indonesia","departure":"CGK","arrival":"JOG","date":"2018-11-11","connecting":"","international":0,"cabinClass":"ECONOMY","schedules":[{"arrivalDate":"2018-11-11","departureDate":"2018-11-11","operatingAirline":{"code":"GA","number":"111"},"fareClass":"","origin":"CGK","destination":"JOG","marketingAirline":"GA","flightNumber":"111","marriageGroup":"Garuda Indonesia"}]}]},"bookResponse":{"mandatoryRequest":{"storeId":"AAA","channelId":"WEB","requestId":"123123","serviceId":"gateway","username":"username"},"integratorBookRequest":{"distributionType":"supplier","adult":1,"child":1,"infant":1,"promoCode":"","viaSelect":"","flightSelect":"","fareSellKey":"","journeySellKey":"","sessionGroup":"","sessionGroupType":"","currency":"IDR","countryId":"ID","account":{"name":"aaa.com","code":"tix"},"contact":{"profileId":"123","firstName":"dondon","lastName":"dev","dob":"2000-01-01","phoneCode":"62","phoneNumber":"123123123","title":"mr","gender":"male","email":"dondon.dev@mail.com"},"paxes":[{"profileId":"123","type":"adult","title":"mr","gender":"male","firstName":"dondon","lastName":"dev","nationality":"ID","dob":"2000-01-01","passport":null,"baggage":null,"frequentFlyer":""},{"profileId":"123","type":"child","title":"mr","gender":"male","firstName":"doni","lastName":"dev","nationality":"ID","dob":"2010-01-01","passport":null,"baggage":null,"frequentFlyer":""},{"profileId":"123","type":"infant","title":"mr","gender":"male","firstName":"purba","lastName":"dev","nationality":"ID","dob":"2018-01-01","passport":null,"baggage":null,"frequentFlyer":""}],"itineraries":[{"airline":"Garuda Indonesia","departure":"CGK","arrival":"JOG","date":"2018-11-11","connecting":"","international":0,"cabinClass":"ECONOMY","schedules":[{"arrivalDate":"2018-11-11","departureDate":"2018-11-11","operatingAirline":{"code":"GA","number":"111"},"fareClass":"","origin":"CGK","destination":"JOG","marketingAirline":"GA","flightNumber":"111","marriageGroup":"Garuda Indonesia"}]}]},"bookDetail":{"distributionType":"","bookingCode":"XXXX","bookingExpired":"2018-12-12","currency":"IDR","account":{"name":"aaa.com","code":"TIX"},"bookingFare":{"balanceDue":0,"bookedFare":0,"fare":0,"adultFare":0,"childFare":0,"infantFare":0,"commission":0,"nta":0,"agentFee":0,"ssrFee":{"baggageFare":0},"incentive":0,"upselling":0},"itineraries":[{"international":0,"currency":"IDR","fare":0,"adultFare":0,"childFare":0,"infantFare":0,"adult":1,"child":1,"infant":1,"cabinClass":"ECONOMY","schedules":[{"departureDate":"2018-11-11","arrivalDate":"2018-11-11","departureTerminal":"3","arrivalTerminal":"","isPromo":1,"fareClass":"ECONOMY","origin":"CGK","destination":"JOG","marketingAirline":"GA","stopQuantity":"","aircraftType":"","marriageGroup":"","supplierBookingCode":"XXX","operatingAirline":{"code":"GA","number":"111"},"baggage":{"cabin":{"unit":"","measurement":"","qty":1},"checkIn":{"unit":"","measurement":"","qty":1}},"connectings":[]}],"markupInfo":{"type":"","value":"","class":""},"breakdownInfo":{"adult":{"psc":0,"iwjr":0,"discount":0,"baseFare":0,"originalBaseFare":0,"nta":0,"commission":0,"tax":0},"child":{"psc":0,"iwjr":0,"discount":0,"baseFare":0,"originalBaseFare":0,"nta":0,"commission":0,"tax":0},"infant":{"psc":0,"iwjr":0,"discount":0,"baseFare":0,"originalBaseFare":0,"nta":0,"commission":0,"tax":0},"baggage":1}}],"passengerFareInfo":{"adult":{"psc":0,"iwjr":0,"discount":0,"baseFare":0,"originalBaseFare":0,"nta":0,"commission":0,"tax":0},"child":{"psc":0,"iwjr":0,"discount":0,"baseFare":0,"originalBaseFare":0,"nta":0,"commission":0,"tax":0},"infant":{"psc":0,"iwjr":0,"discount":0,"baseFare":0,"originalBaseFare":0,"nta":0,"commission":0,"tax":0},"baggage":1}}},"bookResultCode":"200","bookResultMessage":"success"}

Thank you

[sscaling]

Your provided message is only 4939 bytes long (the default message size is 1MB so this is way under).

I can run the following without issue

$ for i in {1..2000}; do printf "%s," "$i"; done > data.txt
$ wc -c data.txt 
    8893 data.txt
$ cat data.txt | kafkacat -b 10.10.0.5:9092 -P -t test
$ kafkacat -b 10.10.0.5:9092 -C -t test -e | wc -c
% Reached end of topic test [0] at offset 1: exiting
    8894

I can also view all data using kafka-console-consumer.

So this would isolate the issue to the producer, kafkacat appears to be working fine.

A quick try of the console-producer using the stdin reader, rather than copy and pasting into the interactive prompt also works fine.

$ docker exec -it abcd1234 /bin/bash
bash-4.4#  for i in {1..2000}; do printf "%s," "$i"; done > data.txt
bash-4.4#  cat data.txt | ./kafka-console-producer.sh --topic test --broker-list 10.10.0.5:9092

So I cannot reproduce the issue with your example, so it's likely your application configuration.

It'd be worth printing the size of each message before you actually send it to Kafka (the default message size of 1MB is pretty generous. if 2GB is actually your theoretical maximum message size, then Kafka may not be the best option for your needs. Based on your example, you're only using a few thousand Kbs.

If this is failing in your Java application, then I'd recommend posting in the Kafka / Spring mailing list as there will be a broader audience to be able to help.

If you find your answer, it would be great if you could update this ticket for future readers. Thanks.

[black-lotus]

Hoo, i see.. it might be about terminal console issue

I have fixes the issue by copy local file to container $ docker cp /Users/dondon/Desktop/book_request_result.txt CONTAINER_NAME:/