Sporadic SIGSEGV with SEGV_MAPERR code in mpack_parse_tag() at mpack.c:2630

[asdavidov]

Bug Report

Describe the bug Fluent-bit sporadically SIGSEGV with SEGV_MAPPER code in mpack.c

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

  n/a

• Steps to reproduce the problem: Since it is a sporadic issue, the steps to reproduce it are unknown at the moment

Expected behavior No SIGSEGV during run time. Screenshots none

Your Environment

• Version used: buld from master branch at commit b7c70817ff986d81a04f4bfa002c8dac38c2acc3 on 17 Oct 2020
• Configuration:

  [SERVICE]
  Flush        10
  Daemon       Off
  Log_Level    debug
  Parsers_File parsers.conf
  Plugins_File plugins.conf
  storage.path /tmp/flb-storage
  storage.backlog.mem_limit 128MB
  HTTP_Server  Off
  [INPUT]
  Name forward
  Listen 0.0.0.0
  Port 24224
  Buffer_Max_Size 8MB
  Mem_Buf_Limit 64MB
  storage.type filesystem
  [OUTPUT]
  Name forward
  Host XX.XXX.XX.XX
  Port 24224
  Match *
  Require_ack_response True
  net.keepalive off

• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version: FreeBSD 11.4-RELEASE-p4 amd64
• Filters and plugins:

  
  Inputs
  tail                  Tail files
  dummy                 Generate dummy data
  head                  Head Input
  health                Check TCP server health
  collectd              collectd input plugin
  statsd                StatsD input plugin
  serial                Serial input
  stdin                 Standard Input
  syslog                Syslog
  exec                  Exec Input
  tcp                   TCP
  mqtt                  MQTT, listen for Publish messages
  forward               Fluentd in-forward
  random                Random

Filters alter_size Alter incoming chunk size aws Add AWS Metadata record_modifier modify record throttle Throttle messages using sliding window algorithm kubernetes Filter to append Kubernetes metadata modify modify records by applying rules nest nest events by specified field values parser Parse events expect Validate expected keys and values grep grep events by specified field values rewrite_tag Rewrite records tags lua Lua Scripting Filter stdout Filter events to STDOUT

Outputs azure Send events to Azure HTTP Event Collector azure_blob Azure Blob Storage bigquery Send events to BigQuery via streaming insert counter Records counter datadog Send events to DataDog HTTP Event Collector es Elasticsearch exit Exit after a number of flushes (test purposes) file Generate log file forward Forward (Fluentd protocol) http HTTP Output influxdb InfluxDB Time Series logdna LogDNA loki Loki kafka-rest Kafka REST Proxy nats NATS Server nrlogs New Relic null Throws away events plot Generate data file for GNU Plot slack Send events to a Slack channel splunk Send events to Splunk HTTP Event Collector stackdriver Send events to Google Stackdriver Logging stdout Prints events to STDOUT syslog Syslog tcp TCP Output td Treasure Data flowcounter FlowCounter gelf GELF Output cloudwatch_logs Send logs to Amazon CloudWatch kinesis_firehose Send logs to Amazon Kinesis Firehose s3 Send to S3

Internal Event Loop = select Build Flags = FLB_HAVE_PARSER FLB_HAVE_RECORD_ACCESSOR FLB_HAVE_STREAM_PROCESSOR JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_AWS FLB_HAVE_SIGNV4 FLB_HAVE_SQLDB FLB_HAVE_FORK FLB_HAVE_GMTOFF FLB_HAVE_UNIX_SOCKET FLB_HAVE_PROXY_GO FLB_HAVE_LIBBACKTRACE FLB_HAVE_REGEX FLB_HAVE_UTF8_ENCODER FLB_HAVE_LUAJIT FLB_HAVE_C_TLS FLB_HAVE_ACCEPT4

**Additional context**
I've got several core files and their content is almost identical.

The backtrace from the version built with debug symbols shows the following output:

(lldb) bt

• thread #1, name = 'fluent-bit', stop reason = signal SIGABRT
  • frame #0: 0x00000008013d50fa libc.so.7__sys_thr_kill + 10 frame #1: 0x00000008013d50c4 libc.so.7raise + 52 frame #2: 0x00000008013d5039 libc.so.7abort + 73 frame #3: 0x000000000043537c fluent-bitflb_signal_handler(signal=11) at fluent-bit.c:418:9 frame #4: 0x00000008010f5c80 libthr.so.3___lldb_unnamed_symbol101$$libthr.so.3 + 208 frame #5: 0x00000008010f524f libthr.so.3_lldb_unnamed_symbol82$$libthr.so.3 + 319 frame #6: 0x00007ffffffff003 frame #7: 0x0000000000530906 fluent-bitmpack_parse_tag(reader=0x0000000801c82f38, tag=0x0000000801c82eb8) at mpack.c:2630:20 frame #8: 0x0000000000530809 fluent-bitmpack_read_tag(reader=0x0000000801c82f38) at mpack.c:2979:20 frame #9: 0x0000000000531d55 fluent-bitmpack_discard(reader=0x0000000801c82f38) at mpack.c:3028:23 frame #10: 0x000000000047468f fluent-bitflb_mp_count(data=0x0000000800aeb028, bytes=144581) at flb_mp.c:40:9 frame #11: 0x00000000004b64f0 fluent-bitflb_forward_format_forward_mode(ctx=0x0000000801c421c0, fc=0x0000000801c8f000, ff=0x0000000801c731b0, tag="edok-ext.backend", tag_len=16, data=0x0000000800aeb028, bytes=144581, out_buf=0x0000000801c83208, out_size=0x0000000801c83200) at forward_format.c:271:19 frame #12: 0x00000000004b5ff7 fluent-bitflb_forward_format(config=0x0000000801a1dc80, ins=0x0000000801aa1180, ins_ctx=0x0000000801c421c0, flush_ctx=0x0000000801c731b0, tag="edok-ext.backend", tag_len=16, data=0x0000000800aeb028, bytes=144581, out_buf=0x0000000801c83208, out_size=0x0000000801c83200) at forward_format.c:433:15 frame #13: 0x00000000004b1877 fluent-bitcb_forward_flush(data=0x0000000800aeb028, bytes=144581, tag="edok-ext.backend", tag_len=16, i_ins=0x0000000801aa1180, out_context=0x0000000801c421c0, config=0x0000000801a1dc80) at forward.c:1108:12 frame #14: 0x0000000000450499 fluent-bitoutput_pre_cb_flush at flb_output.h:469:5 frame #15: 0x00000000005f7150 fluent-bit at amd64.c:117:3 frame #16: 0x00000000004b1d35 fluent-bit`flb_calloc(n=0, size=0) at flb_mem.h:78:11

Frame #11 variables were:

(flb_forward *) ctx = 0x0000000801c421c0
(flb_forward_config *) fc = 0x0000000801c8f000
(flb_forward_flush *) ff = 0x0000000801c731b0
(const char *) tag = 0x0000000801c34500 "edok-ext.backend"
(int) tag_len = 16
(const void *) data = 0x0000000800aeb028
(size_t) bytes = 144581
(void **) out_buf = 0x0000000801c83208
(size_t *) out_size = 0x0000000801c83200
(int) entries = 0
(char *) chunk = 0x0000000801c731b8 ""
(char [33]) chunk_buf = ""
(msgpack_packer) mp_pck = {
  data = 0x0000000801c83008
  callback = 0x00000000004b6940 (fluent-bit`msgpack_sbuffer_write at sbuffer.h:60)
}
(msgpack_sbuffer) mp_sbuf = (size = 0, data = 0x0000000000000000, alloc = 0)

Frame #10 variables:

(const void *) data = 0x0000000800aeb028
(size_t) bytes = 144581
(int) count = 1
(mpack_reader_t) reader = {
  context = 0x0000000000000000
  fill = 0x0000000000000000
  error_fn = 0x0000000000000000
  teardown = 0x0000000000000000
  skip = 0x0000000000000000
  buffer = 0x0000000000000000
  size = 0
  data = 0x0000000800aeb028 ""
  end = 0x0000000800b0e4ed ""
  error = mpack_ok
}

Frame #9 variables:

(mpack_reader_t *) reader = 0x0000000801c82f38
(mpack_tag_t) var = {
  type = mpack_type_missing
  exttype = '\0'
  v = (u = 34389634872, i = 34389634872, d = 1.6990737163279334E-313, f = 7.35361525E-38, b = true, l = 29896504, n = 29896504)
}

Frame #8 variables:

(mpack_reader_t *) reader = 0x0000000801c82f38
(mpack_tag_t) tag = {
  type = mpack_type_missing
  exttype = '\0'
  v = (u = 0, i = 0, d = 0, f = 0, b = false, l = 0, n = 0)
}
(size_t) count = 0

Frame #7 variables:

(mpack_reader_t *) reader = 0x0000000801c82f38
(mpack_tag_t *) tag = 0x0000000801c82eb8
(uint8_t) type = '\0'

Disassembled line of code at frame #7:

fluent-bit`mpack_parse_tag:
    0x530901 <+113>: callq  0x5291b0                  ; mpack_load_u8 at mpack.h:2028

Hope this will help to find the root cause of this problem. I can provide an additional output from debugger if it is needed.

[asdavidov]

FYI - disabling filesystem buffering (i.e. commenting out "storage.type filesystem" option in [INPUT] section) helps to work-around this issue.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[JeffLuoo]

bump, saw this with Loki plugin:

[2021/05/05 01:35:14] [ info] [output:loki:loki.0] loki.logging.svc:3100, HTTP status=204
[2021/05/05 01:35:15] [ info] [output:loki:loki.0] loki.logging.svc:3100, HTTP status=204
[2021/05/05 01:35:16] [engine] caught signal (SIGSEGV)
#0  0x561e046a1727      in  mpack_load_u8() at lib/mpack-amalgamation-1.0/src/mpack/mpack.h:2029
#1  0x561e046a7574      in  mpack_parse_tag() at lib/mpack-amalgamation-1.0/src/mpack/mpack.c:2630
#2  0x561e046a80a5      in  mpack_read_tag() at lib/mpack-amalgamation-1.0/src/mpack/mpack.c:2979
#3  0x561e046a8166      in  mpack_discard() at lib/mpack-amalgamation-1.0/src/mpack/mpack.c:3028
#4  0x561e045d6f3d      in  flb_mp_count() at src/flb_mp.c:44
#5  0x561e04636632      in  loki_compose_payload() at plugins/out_loki/loki.c:954
#6  0x561e046369d9      in  cb_loki_flush() at plugins/out_loki/loki.c:1062
#7  0x561e045af4da      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:466
#8  0x561e04a4e526      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#9  0xffffffffffffffff  in  ???() at ???:0

Version v1.7.3

[chetanv-oi]

Another one:

{"log":"[2022/02/17 00:42:50] [engine] caught signal (SIGSEGV)\n","stream":"stderr","time":"2022-02-17T00:42:50.500981706Z"}
{"log":"#0  0x562629754281      in  mpack_load_u8() at lib/mpack-amalgamation-1.0/src/mpack/mpack.h:2029\n","stream":"stderr","time":"2022-02-17T00:42:50.526985167Z"}
{"log":"#1  0x56262975a0ce      in  mpack_parse_tag() at lib/mpack-amalgamation-1.0/src/mpack/mpack.c:2630\n","stream":"stderr","time":"2022-02-17T00:42:50.527018029Z"}
{"log":"#2  0x56262975abff      in  mpack_read_tag() at lib/mpack-amalgamation-1.0/src/mpack/mpack.c:2979\n","stream":"stderr","time":"2022-02-17T00:42:50.527027388Z"}
{"log":"#3  0x56262975acc0      in  mpack_discard() at lib/mpack-amalgamation-1.0/src/mpack/mpack.c:3028\n","stream":"stderr","time":"2022-02-17T00:42:50.527035311Z"}
{"log":"#4  0x562629693615      in  flb_mp_count() at src/flb_mp.c:40\n","stream":"stderr","time":"2022-02-17T00:42:50.527044261Z"}
{"log":"#5  0x5626296e7a62      in  flush_forward_mode() at plugins/out_forward/forward.c:999\n","stream":"stderr","time":"2022-02-17T00:42:50.527249043Z"}
{"log":"#6  0x5626296e840f      in  cb_forward_flush() at plugins/out_forward/forward.c:1216\n","stream":"stderr","time":"2022-02-17T00:42:50.527258158Z"}
{"log":"#7  0x56262967e64c      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:472\n","stream":"stderr","time":"2022-02-17T00:42:50.527373361Z"}
{"log":"#8  0x562629ad2ec6      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117\n","stream":"stderr","time":"2022-02-17T00:42:50.527392048Z"}

Version 1.6.6