search

italia / daf

Data & Analytics Framework (DAF)
BSD 3-Clause "New" or "Revised" License
50 stars 32 forks source link

[IoT_ingestion_manager] Kafka consumer offsets out of range with no configured reset policy for partitions #178

Open fabiana001 opened 6 years ago

fabiana001 commented 6 years ago

Running the iot_ingestion manager, I have the following exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, slave1.platform.daf.gov.it, executor 4): org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {daf-iot-events-10=46123606}

fabiana001 commented 6 years ago

Get the earliest offset still in a topic

kafka-run-class kafka.tools.GetOffsetShell --broker-list edge1:9092,edge2:9092,edge3:9092 --topic daf-iot-events --time -2

daf-iot-events:8:50471174
daf-iot-events:2:50469614
daf-iot-events:11:50471442
daf-iot-events:5:50471146
daf-iot-events:4:50471226
daf-iot-events:7:50470598
daf-iot-events:10:50471296
daf-iot-events:1:50471551
daf-iot-events:9:50470366
daf-iot-events:3:50471328
daf-iot-events:6:50470914
daf-iot-events:0:50471592

So it seems that the message with offset 46123606 is no longer available. We should change the code in the way that in case of this kind of error we should return the smallest offset available by kafka