filelog receiver: the memory hit the boundary and cause massive event loss

[polarG]

Describe the bug A clear and concise description of what the bug is. component with this issue: filelog receiver. When use 3 filelog receivers in the use case, the otel will show error message "the memory hit the boundary and cause massive event loss". But if only 1 filelog receiver is used, no error come up.

Steps to reproduce If possible, provide a recipe for reproducing the error. Data ingestion:

• File number: 648
• Quantity of event: 20,000,000

Test setup: otel agent ---> otel agent (forwarder) ----> Splunk

• otel agent to otel agent (forwarder) via OTLP
• otel agent (forwarder) to Splunk via Splunk HEC

What did you expect to see? A clear and concise description of what you expected to see. No error messages.

What did you see instead? A clear and concise description of what you saw instead. Otel log error message "the memory hit the boundary and cause massive event loss".

What version did you use? Version: (e.g., v0.4.0, 1eb551b, etc)

What config did you use? Config: (e.g. the yaml config file) receivers: filelog: include:

• /home/ansible/data/otel_mixed_data_20M/generate/Memory.perfmon*
• /home/ansible/data/otel_mixed_data_20M/generate/CPUTime.perfmon*
• /home/ansible/data/otel_mixed_data_20M/generate/change.security.628.windows*
• /home/ansible/data/otel_mixed_data_20M/generate/change.security.1102.windows*
• /home/ansible/data/otel_mixed_data_20M/generate/auth_win_Security.4776.windows*
• /home/ansible/data/otel_mixed_data_20M/generate/network_samplelog_tcp_connection_failure.cisco.asa*
• /home/ansible/data/otel_mixed_data_20M/generate/web_noncorpsite.websense*

• /home/ansible/data/otel_mixed_data_20M/generate/instrusion.cisco.asa* multiline: line_start_pattern: ^\d{2}\/\d{2}\/\d{2,4}\s+\d{2}:\d{2}:\d{2} start_at: beginning include_file_name: false

  prometheus: config: scrape_configs:

• job_name: 'otel-collector' scrape_interval: 5s static_configs:
  • targets: ['0.0.0.0:8888'] metric_relabel_configs:
    • source_labels: [ name ] regex: '.grpc_io.' action: drop

processors: batch:

memory_limiter: ballast_size_mib: ${SPLUNK_BALLAST_SIZE_MIB} check_interval: 2s limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}

exporters: otlphttp: endpoint: "http://:55681" insecure: true compression: none/gzip

splunk_hec: token: "${SPLUNK_HEC_TOKEN}" endpoint: "${SPLUNK_HEC_URL}" source: "metrics" sourcetype: "otel_hostmetrics"

service: pipelines: logs: receivers:

• filelog processors: [memory_limiter, batch] exporters: [otlphttp]

  metrics: receivers:

• prometheus

  - hostmetrics

  processors: [memory_limiter, batch] exporters: [splunk_hec]

Environment OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

Additional context Add any other context about the problem here. @tigrannajaryan

[tigrannajaryan]

@polarG can you please attach the full Collector log?

[tigrannajaryan]

@djaglowski any ideas on what may be causing this? I can't find the error message anywhere in our code base.

[djaglowski]

I'm not aware of any obvious cause off the top of my head. I'm OOO until Thursday, but will look into it then.

[djaglowski]

@polarG Can you attach the collector log? I also cannot find this error message, or one like it, anywhere in the code base. I'll need more context to look into this further.

[djaglowski]

I have not observed this issue or heard of it again from anyone else. Closing this but please open a new issue if necessary.