[http_client] cannot increase buffer: current=4096, requested=36864 max=4096

[LFrankJun]

Bug Report

Describe the bug [SERVICE] Parsers_File /opt/fluent-bit/etc/fluent-bit/fluent-bit-parsers.conf flush 1 daemon Off log_level info log_file /var/log/fluent-bit/fluent-bit.log hot_reload on [INPUT] name tail parser suricataInput buffer_chunk_size 50M buffer_max_size 50M path /fluent-bit/data/suricata_event.json db /opt/fluent-bit/etc/fluent-bit/record_pos.db [FILTER] name lua match script /opt/fluent-bit/etc/fluent-bit/preprocess_suricata_event.lua call preprocess_suricata_event [FILTER] name nest match operation lift nested_under alert [FILTER] name record_modifier match record sensor_id xxxxxxyyyyyyddddd allowlist_key timestamp .................. [FILTER] name modify match rename timestamp time_at rename src_ip source_ip .................. [FILTER] name nest match operation nest wildcard evi. nest_under evidence remove_prefix evi. [OUTPUT] name http match * host 10.11.78.25 port 3000 tls on tls.verify off uri /tenants/f3ct9myv9p/security-events header Content-Type application/json header Authorization Bearer rfasrqe324324 format json json_date_key false workers 0

I set the fluent-bit.conf as above, but I got a error "[http_client] cannot increase buffer: current=4096, requested=36864 max=4096"

Expected behavior no bug occured

Your Environment

• Version used: Fluent-bit 2.1.3
• Filters and plugins: tail , modify, record_modifier, nest, http

Additional context I find many solutions about the same error, most of solution is that add "Buffer_Size" property in output plugin(eg: elasticsearch) for solve this issue, but my output plugin is "HTTP", it doesn't have this property, so I don't know how to solve this issue？

[agusl88]

Hi, did you find any solution? I'm having the same problem with the parser plugin, it doesn't include a "Buffer_Size"

[LFrankJun]

Hi, did you find any solution? I'm having the same problem with the parser plugin, it doesn't include a "Buffer_Size"

No, I don't know how to solve this problem.

[rayer4u]

have the same problem.

[2023/09/11 01:00:20] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2023/09/11 01:00:21] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2023/09/11 01:00:21] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000
[2023/09/11 01:00:22] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000

[ptr1120]

Same problem when using Loki Output

 [fluent bit] version=2.1.10

[OUTPUT]
    name                   loki
    match                  kube.*
    host                     loki-gateway.logging.svc.cluster.local
    port                     80
    tenant_id             1
    auto_kubernetes_labels on
    label_map_path         /fluent-bit/etc/conf/k8s-label-mapping.json

[2023/12/07 17:05:28] [ warn] [http_client] cannot increase buffer: current=4096 requested=36864 max=4096

[legeech]

Same problem different versions of td-agent-bit 1.8., 1.9. [2024/01/16 07:34:11] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 debugging does not give any clues about the reason and there are no correlating parameters to output about size equal 512000

[FabioMaas]

Also got the same problem with the Loki Output, using custom multilineparser.

[OUTPUT]
    Name       loki
    Match      custom.production
    Host       10.220.42.20
    Port       3100
    Labels     job=fluentbit

Is it possible to expose the underlying values as a parameter buffer_size for output plugins in the future, that are using the client? This seems to be already done internally:

https://github.com/fluent/fluent-bit/blob/228b85715f27993b41fa012b736d5a8b4d5f139c/src/flb_http_client.c#L1365

https://github.com/fluent/fluent-bit/blob/228b85715f27993b41fa012b736d5a8b4d5f139c/src/flb_http_client.c#L795

[chaddewitt]

Same problem using the influxdb output plugin.

[2024/04/03 15:36:52] [ info] [fluent bit] version=2.2.2, commit=eeea396e88, pid=1

[OUTPUT]
    Name                            influxdb
    Match                           k8s
    Host                            <redacted>
    Port                            8086
    Database                        metrics

[2024/04/03 15:36:58] [ warn] [http_client] cannot increase buffer: current=4096 requested=36864 max=4096

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[jadiym]

Having the same issue in file output plugin {"log":"[2024/07/09 01:24:10] [ info] [fluent bit] version=2.1.10, commit=b19e9ce674, pid=1\n","stream":"stderr","time":"2024-07-09T01:24:10.774595759Z"}

{"log":"[2024/07/09 01:24:23] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000\n","stream":"stderr","time":"2024-07-09T01:24:23.378850095Z"}

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.