Nats output plugin does not work

[asvetliy]

Nats output plugin does not work

Description Hello, I wanted to use a bunch of Nats + fluent-bit for my application, but it did not work. The connection is terminated as soon as it has been established.

To Reproduce

1. I use simple docker-compose.yml for testing:

   
   version: '3'

networks: app-net: driver: bridge

services: fluent-bit: image: fluent/fluent-bit:latest command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf -i cpu -o nats -p host=nats volumes:

• ./etc/fluentbit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf depends_on:
• nats ports:
• 24224:24224
• 24224:24224/udp networks:

• app-net

  nats: image: nats:latest ports:

• 4222:4222
• 8222:8222 networks:

• app-net

  2. My `fluent-bit.conf`:

  [SERVICE] Flush 1 Daemon Off Log_Level debug

  Parsers_File /fluent-bit/etc/parsers.conf

  HTTP_Server Off HTTP_Listen 0.0.0.0 HTTP_Port 2020

  3. fluent-bit logs:

  fluent-bit_1 | Fluent Bit v1.5.7 fluent-bit_1 | Copyright (C) 2019-2020 The Fluent Bit Authors fluent-bit_1 | Copyright (C) 2015-2018 Treasure Data fluent-bit_1 | Fluent Bit is a CNCF sub-project under the umbrella of Fluentd fluent-bit_1 | https://fluentbit.io fluent-bit_1 | fluent-bit_1 | [2020/10/05 11:42:15] [ info] Configuration: fluent-bit_1 | [2020/10/05 11:42:15] [ info] flush time | 1.000000 seconds fluent-bit_1 | [2020/10/05 11:42:15] [ info] grace | 5 seconds fluent-bit_1 | [2020/10/05 11:42:15] [ info] daemon | 0 fluent-bit_1 | [2020/10/05 11:42:15] [ info] _ fluent-bit_1 | [2020/10/05 11:42:15] [ info] inputs: fluent-bit_1 | [2020/10/05 11:42:15] [ info] cpu fluent-bit_1 | [2020/10/05 11:42:15] [ info] ___ fluent-bit_1 | [2020/10/05 11:42:15] [ info] filters: fluent-bit1 | [2020/10/05 11:42:15] [ info] fluent-bit_1 | [2020/10/05 11:42:15] [ info] outputs: fluent-bit_1 | [2020/10/05 11:42:15] [ info] nats.0 fluent-bit_1 | [2020/10/05 11:42:15] [ info] ___ fluent-bit_1 | [2020/10/05 11:42:15] [ info] collectors: fluent-bit_1 | [2020/10/05 11:42:15] [ info] [engine] started (pid=1) fluent-bit_1 | [2020/10/05 11:42:15] [debug] [engine] coroutine stack size: 24576 bytes (24.0K) fluent-bit_1 | [2020/10/05 11:42:15] [debug] [storage] [cio stream] new stream registered: cpu.0 fluent-bit_1 | [2020/10/05 11:42:15] [ info] [storage] version=1.0.5, initializing... fluent-bit_1 | [2020/10/05 11:42:15] [ info] [storage] in-memory fluent-bit_1 | [2020/10/05 11:42:15] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128 fluent-bit_1 | [2020/10/05 11:42:15] [debug] [router] default match rule cpu.0:nats.0 fluent-bit_1 | [2020/10/05 11:42:15] [ info] [sp] stream processor started fluent-bit_1 | [2020/10/05 11:42:16] [trace] [input:cpu:cpu.0 at plugins/in_cpu/cpu.c:414] CPU 0.25% fluent-bit_1 | [2020/10/05 11:42:17] [debug] [task] created task=0x7f89b042f3a0 id=0 OK fluent-bit_1 | [2020/10/05 11:42:17] [debug] [upstream] KA connection #20 to nats:4222 is now available fluent-bit_1 | [2020/10/05 11:42:17] [trace] [input:cpu:cpu.0 at plugins/in_cpu/cpu.c:414] CPU 0.25% fluent-bit_1 | [2020/10/05 11:42:17] [debug] [task] destroy task=0x7f89b042f3a0 (task_id=0) fluent-bit_1 | [2020/10/05 11:42:18] [debug] [task] created task=0x7f89b042f3a0 id=0 OK fluent-bit_1 | [2020/10/05 11:42:18] [debug] [upstream] KA connection #20 is in a failed state to: nats:4222, cleaning up fluent-bit_1 | [2020/10/05 11:42:18] [debug] [upstream] KA connection #20 to nats:4222 is now available ..................

  Then its write that debug message all the time:

  [debug] [upstream] KA connection #20 is in a failed state to: nats:4222, cleaning up [debug] [upstream] KA connection #20 to nats:4222 is now available

  4. In Nats I get this connection:

  { "server_id": "NAGWUWIFRJH3N5W3RMEAKXQIGKQZHTG2YLZ637FXB5SI2YWX6DKD5T4Y", "now": "2020-10-05T11:46:21.7989072Z", "num_connections": 1, "total": 1, "offset": 0, "limit": 1024, "connections": [ { "cid": 336, "ip": "172.26.0.3", "port": 59826, "start": "2020-10-05T11:46:21.0619663Z", "last_activity": "2020-10-05T11:46:21.062688Z", "rtt": "552µs", "uptime": "0s", "idle": "0s", "pending_bytes": 0, "in_msgs": 1, "out_msgs": 0, "in_bytes": 703, "out_bytes": 0, "subscriptions": 0, "name": "fluent-bit", "lang": "c", "version": "1.5.7" } ] }

  
  But it always reconnect with 0s uptime and with new port, etc.

Additional context As described in the documentation of NATS:

Fundamentally, NATS is about publishing and listening for messages. 
Both of these depend heavily on Subjects which scope messages into streams or topics. 
At its simplest, a subject is just a string of characters that form a name the publisher and 
subscriber can use to find each other.

So, for publishing some messages you need to set subject and it is also need to set queue group. But in the fluent-bit NATS plugin documentation no fields to do this. Thanks!

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[lionello]

It works, but you have to set a tag to whichever input you're using. Note that the FORWARD plugin doesn't set tags (but can optionally add a tag_prefix to the tag of the events it's forwarding.)

[SamyElHamass]

@lionello Thanks for your comment! This is kind of confusingly documented.

Is there also a possibility for authentication?