search

mozilla-services / hindsight

Hindsight - light weight data processing skeleton
Mozilla Public License 2.0
668 stars 55 forks source link

Hindsight segfaults when unable to send data to kafka #203

Open deric opened 4 years ago

deric commented 4 years ago

After reaching certain number of retries on sending data to kafka

1602157587455342258 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out
1602157771477482273 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out
1602157827484986170 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out
1602158033512463221 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out
1602158049514770134 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out
1602158101521738537 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out
1602158327551397487 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out
1602158337552731474 [error] rdkafka#producer-1 delivery error   -192    Local: Message timed out

the process segfaults, instead of writing error message to log

hindsight[22643]: segfault at 0 ip           (null) sp 00007fdcf1ffa5e8 error 14 in hindsight[55bfadbb8000+14000]
Oct 08 09:46:14 w40 hindsight[3718]: 1602150374945348677 [info] hindsight starting
Oct 08 09:46:14 w40 hindsight[3718]: 1602150374951200615 [info] input.tcp starting
Oct 08 09:46:14 w40 hindsight[3718]: 1602150374956800279 [info] input.udp starting
Oct 08 09:46:14 w40 hindsight[3718]: 1602150374957511367 [info] input.prune starting
Oct 08 09:46:14 w40 hindsight[3718]: 1602150374962313639 [info] analysis_plugins starting thread: 0
Oct 08 09:46:14 w40 hindsight[3718]: 1602150374970768206 [info] output.kafka starting
Oct 08 09:51:16 w40 systemd[1]: hindsight.service: Main process exited, code=killed, status=11/SEGV
Oct 08 09:51:16 w40 systemd[1]: hindsight.service: Unit entered failed state.
Oct 08 09:51:16 w40 systemd[1]: hindsight.service: Failed with result 'signal'.

hindsight version: 0.16.0