search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
175 stars 3.14k forks source link

rdkafka_example with mode producer is not working with Druid #856

Closed parulagrawal14 closed 7 years ago

parulagrawal14 commented 7 years ago

Description

I am trying to use librdkafka as a producer and would like to send the data to Druid as mentioned in https://imply.io/docs/latest/tutorial-kafka-indexing-service.html.

But kakfa-console-consumer of kafka_2.11-0.9.0.0 is not receiving any data.

How to reproduce

Following steps will reproduce the issue: 1) Run Druid as mentioned in site which will in turn starts the zookeeper (https://imply.io/docs/latest/tutorial-kafka-indexing-service.html)

2) Run Broker as mentioned in site (https://imply.io/docs/latest/tutorial-kafka-indexing-service.html)

Run rdkafka_example as producer 3) ./rdkafka_example -P -t application -d broker -b localhost:9092

4) Run kafka-console-consumer from the downloaded kafka of site (https://imply.io/docs/latest/tutorial-kafka-indexing-service.html) cd kafka_2.11-0.9.0.0 bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic application --from-beginning

Here consumer will not be able to receive any data from rdkafka_example.

Checklist

Please provide the following information:

edenhill commented 7 years ago

run rdkafka_example with -d topic,broker,msg,protocol and post some logs following you typing a message to produce

parulagrawal14 commented 7 years ago

Kindly find the logs below:

Script started on Mon 24 Oct 2016 09:13:37 AM CDT administrator@aeverie14:~/libkafka/librdkafka/examples$ exitps -eaf | grep rdkafkakill %./rdkafka_example -P -t application -b 10.1.0.225:9092 -d topic,broker,msg,protocol 1477318421.007 RDKAFKA-7-BROKER: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Added new broker with NodeId -1 1477318421.007 RDKAFKA-7-TOPIC: rdkafka#producer-1: New local topic: application 1477318421.007 RDKAFKA-7-TOPPARNEW: rdkafka#producer-1: NEW application [-1] 0xe90d70 (at rd_kafka_topic_new0:261) 1477318421.007 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Enter main broker thread 1477318421.007 RDKAFKA-7-CONNECT: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: broker in state INIT connecting 1477318421.007 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: :0/internal: Enter main broker thread 1477318421.007 RDKAFKA-7-STATE: rdkafka#producer-1: :0/internal: Broker changed state INIT -> UP 1477318421.007 RDKAFKA-7-CONNECT: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Connecting to ipv4#10.1.0.225:9092 (plaintext) with socket 4 1477318421.007 RDKAFKA-7-STATE: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker changed state INIT -> CONNECT 1477318421.017 RDKAFKA-7-CONNECT: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Connected to ipv4#10.1.0.225:9092 1477318421.017 RDKAFKA-7-CONNECTED: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Connected (#1) 1477318421.017 RDKAFKA-7-APIVERSION: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Using (configuration fallback) 0.9.0 protocol features 1477318421.017 RDKAFKA-7-FEATURE: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4 1477318421.017 RDKAFKA-7-STATE: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker changed state CONNECT -> UP 1477318421.028 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 1) 1477318421.038 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 1, rtt 10.25ms) 1477318421.038 RDKAFKA-7-BROKER: rdkafka#producer-1: 0.0.0.0:9092/0: Added new broker with NodeId 0 1477318421.038 RDKAFKA-7-STATE: rdkafka#producer-1: Topic application changed state unknown -> exists 1477318421.038 RDKAFKA-7-PARTCNT: rdkafka#producer-1: Topic application partition count changed from 0 to 1 1477318421.038 RDKAFKA-7-TOPPARNEW: rdkafka#producer-1: NEW application [0] 0x7f0bb40012e0 (at rd_kafka_topic_partition_cnt_update:473) 1477318421.038 RDKAFKA-7-BRKDELGT: rdkafka#producer-1: application [0]: delegate to broker 0.0.0.0:9092/0 (rktp 0x7f0bb40012e0, term 0, ref 2, remove 0) 1477318421.038 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: 0.0.0.0:9092/0: Enter main broker thread 1477318421.038 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: broker in state INIT connecting 1477318421.038 RDKAFKA-7-BRKDELGT: rdkafka#producer-1: application [0]: broker 0.0.0.0:9092/0 is now leader for partition with 0 messages (0 bytes) queued 1477318421.038 RDKAFKA-7-BRKMIGR: rdkafka#producer-1: Migrating topic application [0] 0x7f0bb40012e0 from (none) to 0.0.0.0:9092/0 (sending PARTITION_JOIN to 0.0.0.0:9092/0) 1477318421.038 RDKAFKA-7-PARTCNT: rdkafka#producer-1: Partitioning 0 unassigned messages in topic application to 1 partitions 1477318421.038 RDKAFKA-7-UAS: rdkafka#producer-1: 0/0 messages were partitioned in topic application 1477318421.038 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connecting to ipv4#0.0.0.0:9092 (plaintext) with socket 5 1477318421.039 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state INIT -> CONNECT 1477318421.039 RDKAFKA-7-TOPBRK: rdkafka#producer-1: 0.0.0.0:9092/0: Topic application [0]: joining broker (rktp 0x7f0bb40012e0) 1477318421.049 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connected to ipv4#0.0.0.0:9092 1477318421.049 RDKAFKA-7-CONNECTED: rdkafka#producer-1: 0.0.0.0:9092/0: Connected (#1) 1477318421.049 RDKAFKA-7-APIVERSION: rdkafka#producer-1: 0.0.0.0:9092/0: Using (configuration fallback) 0.9.0 protocol features 1477318421.049 RDKAFKA-7-FEATURE: rdkafka#producer-1: 0.0.0.0:9092/0: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4 1477318421.049 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state CONNECT -> UP 1477318421.049 RDKAFKA-7-SEND: rdkafka#producer-1: 0.0.0.0:9092/0: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 1) 1477318421.050 RDKAFKA-7-RECV: rdkafka#producer-1: 0.0.0.0:9092/0: Received MetadataResponse (v0, 70 bytes, CorrId 1, rtt 0.52ms) {"timestamp": "Sep 04 2016 07:06:59 CDT","pid": "1","tgid": "2"}

After waiting for few minutes, following logs were displayed at the console.

{"timestamp": "Sep 04 2016 07:06:59 CDT","pid": "1","tgid": "2"} ===> Message which is sent

1477318421.0491477318721.066 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 2) 1477318721.076 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 2, rtt 10.22ms) 1477319021.105 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 3) 1477319021.115 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 3, rtt 10.17ms) 1477319021.173 RDKAFKA-7-BROKERFAIL: rdkafka#producer-1: 0.0.0.0:9092/0: failed: err: Local: Broker transport failure: (errno: Resource temporarily unavailable) 1477319021.173 RDKAFKA-3-FAIL: rdkafka#producer-1: 0.0.0.0:9092/0: Receive failed: Disconnected 1477319021.173 RDKAFKA-3-ERROR: rdkafka#producer-1: 0.0.0.0:9092/0: Receive failed: Disconnected 1477319021.173 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state UP -> DOWN 1477319021.236 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 4) 1477319021.246 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 4, rtt 10.20ms) 1477319021.273 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: broker in state DOWN connecting 1477319021.273 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connecting to ipv4#0.0.0.0:9092 (plaintext) with socket 5 1477319021.273 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state DOWN -> CONNECT 1477319021.283 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connected to ipv4#0.0.0.0:9092 1477319021.283 RDKAFKA-7-CONNECTED: rdkafka#producer-1: 0.0.0.0:9092/0: Connected (#2) 1477319021.283 RDKAFKA-7-APIVERSION: rdkafka#producer-1: 0.0.0.0:9092/0: Using (configuration fallback) 0.9.0 protocol features 1477319021.283 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state CONNECT -> UP 1477319021.284 RDKAFKA-7-SEND: rdkafka#producer-1: 0.0.0.0:9092/0: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 2) 1477319021.284 RDKAFKA-7-RECV: rdkafka#producer-1: 0.0.0.0:9092/0: Received MetadataResponse (v0, 70 bytes, CorrId 2, rtt 0.63ms)

Request you guidance in this.

parulagrawal14 commented 7 years ago

Hi,

Please find the attached logs. After sending the message from producer there was no log at the console.

Thanks and Regards, Parul

On Mon, Oct 24, 2016 at 6:43 PM, Magnus Edenhill notifications@github.com wrote:

run rdkafka_example with -d topic,broker,msg,protocol and post some logs following you typing a message to produce

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/edenhill/librdkafka/issues/856#issuecomment-255736600, or mute the thread https://github.com/notifications/unsubscribe-auth/AL780wYZF_uRnw0oEL1rTgC8TJ6KYDhXks5q3K8LgaJpZM4KesuQ .

edenhill commented 7 years ago

I dont think you can send attachments through email.

parulagrawal14 commented 7 years ago

Hi,

After waiting for few minutes, following logs were displayed at the console.

{"timestamp": "Sep 04 2016 07:06:59 CDT","pid": "1","tgid": "2"} ===> Message which is sent

1477318421.0491477318721.066 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 2) 1477318721.076 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 2, rtt 10.22ms) 1477319021.105 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 3) 1477319021.115 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 3, rtt 10.17ms) 1477319021.173 RDKAFKA-7-BROKERFAIL: rdkafka#producer-1: 0.0.0.0:9092/0: failed: err: Local: Broker transport failure: (errno: Resource temporarily unavailable) 1477319021.173 RDKAFKA-3-FAIL: rdkafka#producer-1: 0.0.0.0:9092/0: Receive failed: Disconnected 1477319021.173 RDKAFKA-3-ERROR: rdkafka#producer-1: 0.0.0.0:9092/0: Receive failed: Disconnected 1477319021.173 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state UP -> DOWN 1477319021.236 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 4) 1477319021.246 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 4, rtt 10.20ms) 1477319021.273 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: broker in state DOWN connecting 1477319021.273 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connecting to ipv4#0.0.0.0:9092 (plaintext) with socket 5 1477319021.273 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state DOWN -> CONNECT 1477319021.283 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connected to ipv4#0.0.0.0:9092 1477319021.283 RDKAFKA-7-CONNECTED: rdkafka#producer-1: 0.0.0.0:9092/0: Connected (#2) 1477319021.283 RDKAFKA-7-APIVERSION: rdkafka#producer-1: 0.0.0.0:9092/0: Using (configuration fallback) 0.9.0 protocol features 1477319021.283 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state CONNECT -> UP 1477319021.284 RDKAFKA-7-SEND: rdkafka#producer-1: 0.0.0.0:9092/0: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 2) 1477319021.284 RDKAFKA-7-RECV: rdkafka#producer-1: 0.0.0.0:9092/0: Received MetadataResponse (v0, 70 bytes, CorrId 2, rtt 0.63ms)

Request you guidance in this.

Thanks and Regards, Parul

On Mon, Oct 24, 2016 at 7:49 PM, Parul Agrawal parulagrawal14@gmail.com wrote:

Hi,

Please find the attached logs. After sending the message from producer there was no log at the console.

Thanks and Regards, Parul

On Mon, Oct 24, 2016 at 6:43 PM, Magnus Edenhill <notifications@github.com

wrote:

run rdkafka_example with -d topic,broker,msg,protocol and post some logs following you typing a message to produce

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/edenhill/librdkafka/issues/856#issuecomment-255736600, or mute the thread https://github.com/notifications/unsubscribe-auth/AL780wYZF_uRnw0oEL1rTgC8TJ6KYDhXks5q3K8LgaJpZM4KesuQ .

edenhill commented 7 years ago

Sorry, can you reproduce this with -d topic,broker,msg,protocol,metadata

parulagrawal14 commented 7 years ago

Hi,

Kindly find the logs as requested. Please let me know if you need any other information.

administrator@kafka:~/libkafka/librdkafka/examples$ ./rdkafka_example -P -t application -b 10.1.0.225:9092 -d topic,broker,msg,protocol,metadata 1477446304.725 RDKAFKA-7-BROKER: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Added new broker with NodeId -1 1477446304.725 RDKAFKA-7-TOPIC: rdkafka#producer-1: New local topic: application 1477446304.725 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: :0/internal: Enter main broker thread 1477446304.725 RDKAFKA-7-STATE: rdkafka#producer-1: :0/internal: Broker changed state INIT -> UP 1477446304.725 RDKAFKA-7-TOPPARNEW: rdkafka#producer-1: NEW application [-1] 0x2041d70 (at rd_kafka_topic_new0:261) 1477446304.725 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Enter main broker thread 1477446304.725 RDKAFKA-7-CONNECT: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: broker in state INIT connecting 1477446304.725 RDKAFKA-7-CONNECT: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Connecting to ipv4#10.1.0.225:9092 (plaintext) with socket 4 1477446304.725 RDKAFKA-7-STATE: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker changed state INIT -> CONNECT 1477446304.735 RDKAFKA-7-CONNECT: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Connected to ipv4#10.1.0.225:9092 1477446304.736 RDKAFKA-7-CONNECTED: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Connected (#1) 1477446304.736 RDKAFKA-7-APIVERSION: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Using (configuration fallback) 0.9.0 protocol features 1477446304.736 RDKAFKA-7-FEATURE: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4 1477446304.736 RDKAFKA-7-STATE: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker changed state CONNECT -> UP 1477446304.736 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: connected 1477446304.736 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: connected 1477446304.746 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 1) 1477446304.756 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 1, rtt 10.21ms) 1477446304.756 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: ===== Received metadata ===== 1477446304.756 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: 1 brokers, 1 topics 1477446304.756 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker #0/1: 0.0.0.0:9092 NodeId 0 1477446304.756 RDKAFKA-7-BROKER: rdkafka#producer-1: 0.0.0.0:9092/0: Added new broker with NodeId 0 1477446304.756 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic #0/1: application with 1 partitions 1477446304.756 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: 0.0.0.0:9092/0: Enter main broker thread 1477446304.756 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: broker in state INIT connecting 1477446304.756 RDKAFKA-7-STATE: rdkafka#producer-1: Topic application changed state unknown -> exists 1477446304.757 RDKAFKA-7-PARTCNT: rdkafka#producer-1: Topic application partition count changed from 0 to 1 1477446304.757 RDKAFKA-7-TOPPARNEW: rdkafka#producer-1: NEW application [0] 0x7fe40c0012c0 (at rd_kafka_topic_partition_cnt_update:473) 1477446304.757 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic application partition 0 Leader 0 1477446304.757 RDKAFKA-7-BRKDELGT: rdkafka#producer-1: application [0]: delegate to broker 0.0.0.0:9092/0 (rktp 0x7fe40c0012c0, term 0, ref 2, remove 0) 1477446304.757 RDKAFKA-7-BRKDELGT: rdkafka#producer-1: application [0]: broker 0.0.0.0:9092/0 is now leader for partition with 0 messages (0 bytes) queued 1477446304.757 RDKAFKA-7-BRKMIGR: rdkafka#producer-1: Migrating topic application [0] 0x7fe40c0012c0 from (none) to 0.0.0.0:9092/0 (sending PARTITION_JOIN to 0.0.0.0:9092/0) 1477446304.757 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connecting to ipv4#0.0.0.0:9092 (plaintext) with socket 5 1477446304.757 RDKAFKA-7-PARTCNT: rdkafka#producer-1: Partitioning 0 unassigned messages in topic application to 1 partitions 1477446304.757 RDKAFKA-7-UAS: rdkafka#producer-1: 0/0 messages were partitioned in topic application 1477446304.757 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state INIT -> CONNECT 1477446304.757 RDKAFKA-7-TOPBRK: rdkafka#producer-1: 0.0.0.0:9092/0: Topic application [0]: joining broker (rktp 0x7fe40c0012c0) 1477446304.767 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connected to ipv4#0.0.0.0:9092 1477446304.767 RDKAFKA-7-CONNECTED: rdkafka#producer-1: 0.0.0.0:9092/0: Connected (#1) 1477446304.767 RDKAFKA-7-APIVERSION: rdkafka#producer-1: 0.0.0.0:9092/0: Using (configuration fallback) 0.9.0 protocol features 1477446304.767 RDKAFKA-7-FEATURE: rdkafka#producer-1: 0.0.0.0:9092/0: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4 1477446304.767 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state CONNECT -> UP 1477446304.767 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Request metadata for locally known topics: connected 1477446304.767 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Request metadata for locally known topics: connected 1477446304.767 RDKAFKA-7-SEND: rdkafka#producer-1: 0.0.0.0:9092/0: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 1) 1477446304.768 RDKAFKA-7-RECV: rdkafka#producer-1: 0.0.0.0:9092/0: Received MetadataResponse (v0, 70 bytes, CorrId 1, rtt 1.00ms) 1477446304.768 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: ===== Received metadata ===== 1477446304.768 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: 1 brokers, 1 topics 1477446304.768 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Broker #0/1: 0.0.0.0:9092 NodeId 0 1477446304.768 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Topic #0/1: application with 1 partitions 1477446304.768 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Topic application partition 0 Leader 0 {"timestamp": "Sep 04 2016 07:06:59 CDT","pid": "1","tgid": "2"}

(Immediately after sending the message no logs are displayed, but after waiting for sometime below logs are displayed)

1477446604.725 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: sparse periodic refresh: scheduled: not in broker thread 1477446604.773 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: sparse periodic refresh 1477446604.773 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: sparse periodic refresh 1477446604.783 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 2) 1477446604.794 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 2, rtt 10.17ms) 1477446604.794 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: ===== Received metadata ===== 1477446604.794 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: 1 brokers, 1 topics 1477446604.794 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker #0/1: 0.0.0.0:9092 NodeId 0 1477446604.794 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic #0/1: application with 1 partitions 1477446604.794 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic application partition 0 Leader 0 1477446904.725 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: sparse periodic refresh: scheduled: not in broker thread 1477446904.776 RDKAFKA-7-BROKERFAIL: rdkafka#producer-1: 0.0.0.0:9092/0: failed: err: Local: Broker transport failure: (errno: Resource temporarily unavailable) 1477446904.776 RDKAFKA-3-FAIL: rdkafka#producer-1: 0.0.0.0:9092/0: Receive failed: Disconnected 1477446904.776 RDKAFKA-3-ERROR: rdkafka#producer-1: 0.0.0.0:9092/0: Receive failed: Disconnected 1477446904.776 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state UP -> DOWN 1477446904.776 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: leader query: scheduled: not in broker thread 1477446904.807 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: sparse periodic refresh 1477446904.807 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: sparse periodic refresh 1477446904.807 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: leader query 1477446904.807 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Request metadata for locally known topics: leader query 1477446904.817 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 3) 1477446904.817 RDKAFKA-7-SEND: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 4) 1477446904.827 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 3, rtt 10.24ms) 1477446904.827 RDKAFKA-7-RECV: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Received MetadataResponse (v0, 70 bytes, CorrId 4, rtt 10.28ms) 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: ===== Received metadata ===== 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: 1 brokers, 1 topics 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker #0/1: 0.0.0.0:9092 NodeId 0 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic #0/1: application with 1 partitions 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic application partition 0 Leader 0 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: ===== Received metadata ===== 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: 1 brokers, 1 topics 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Broker #0/1: 0.0.0.0:9092 NodeId 0 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic #0/1: application with 1 partitions 1477446904.828 RDKAFKA-7-METADATA: rdkafka#producer-1: 10.1.0.225:9092/bootstrap: Topic application partition 0 Leader 0 1477446904.876 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: broker in state DOWN connecting 1477446904.876 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connecting to ipv4#0.0.0.0:9092 (plaintext) with socket 5 1477446904.876 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state DOWN -> CONNECT 1477446904.886 RDKAFKA-7-CONNECT: rdkafka#producer-1: 0.0.0.0:9092/0: Connected to ipv4#0.0.0.0:9092 1477446904.886 RDKAFKA-7-CONNECTED: rdkafka#producer-1: 0.0.0.0:9092/0: Connected (#2) 1477446904.886 RDKAFKA-7-APIVERSION: rdkafka#producer-1: 0.0.0.0:9092/0: Using (configuration fallback) 0.9.0 protocol features 1477446904.886 RDKAFKA-7-STATE: rdkafka#producer-1: 0.0.0.0:9092/0: Broker changed state CONNECT -> UP 1477446904.886 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Request metadata for locally known topics: connected 1477446904.886 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Request metadata for locally known topics: connected 1477446904.886 RDKAFKA-7-SEND: rdkafka#producer-1: 0.0.0.0:9092/0: Sent MetadataRequest (v0, 38 bytes @ 0, CorrId 2) 1477446904.887 RDKAFKA-7-RECV: rdkafka#producer-1: 0.0.0.0:9092/0: Received MetadataResponse (v0, 70 bytes, CorrId 2, rtt 0.80ms) 1477446904.887 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: ===== Received metadata ===== 1477446904.887 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: 1 brokers, 1 topics 1477446904.887 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Broker #0/1: 0.0.0.0:9092 NodeId 0 1477446904.887 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Topic #0/1: application with 1 partitions 1477446904.887 RDKAFKA-7-METADATA: rdkafka#producer-1: 0.0.0.0:9092/0: Topic application partition 0 Leader 0

Please let me know if i have missed anything. This is quite critical for my project and would request you help in resolving this issue.

edenhill commented 7 years ago

Stupid question but are you actually hitting enter after pasting that JSON line? It looks like the message never makes it into librdkafka.

Does this work? echo hi there | ./rdkafka_example -b ..broker.. -P -t application -d topic,broker,msg,metadata,protocol

parulagrawal14 commented 7 years ago

Hi,

Thank you very much for your prompt reply. I updated the code with recent master branch and the issue is not seen. You can close this issue.

edenhill commented 7 years ago

Which librdkafka version were you previously on?