auto_partial_merge doesn't work in kubernetes source

abanaev commented 8 months ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We use the vector configuration attached below

According to the vector documentation :

Partial message merging By default, Vector merges partial messages that are split due to the Docker size limit. For everything else, we recommend that you use the reduce\transform, which enables you to handle custom merging of things like stacktraces.

Our applications in k8s can have logs larger than 1MB. They are split by container runtime(docker) into partial. In kafka we already see a split message.

We also tried to use a configuration without transformations

Configuration

sources:
  k8sLogs_source:
    type: kubernetes_logs
    auto_partial_merge: true
    data_dir: /vector-data-dir
    delay_deletion_ms: 60000
    fingerprint_lines: 1
    glob_minimum_cooldown_ms: 60000
    ignore_older_secs: 600
    max_line_bytes: 1048576
    oldest_first: true
    read_from: beginning

transforms:
  k8sLogs_transform_RAW_log:
    inputs:
      -k8sLogs_source
    type: remap
    source: |
        parse .........
        .....
        ....

sinks:
  i16nodes_k8sLogs_sink:
    type: "kafka"

Version

0.34.2-alpine

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

Alexander-177 commented 8 months ago

same problem

cyberjihadxyz commented 7 months ago

Same problem please up

vectordotdev / vector