search

fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.9k stars 1.59k forks source link

Please add support for Azure Event Hub #1925

Open igorgatis opened 4 years ago

igorgatis commented 4 years ago

Is your feature request related to a problem? Please describe. My company would like to use Azure Event Hub to collect system logs so they can be processed by Elastic Search and Databricks pipeline.

Describe the solution you'd like Simple output plugin for Azure Event Hub.

Describe alternatives you've considered I consider writing my own c plugin using https://github.com/Azure/azure-event-hubs-c.

Additional context My company uses AKS and Azure Analytics is not the best tool for logging management.

ahacking commented 4 years ago

FYI: I am using the kafka output plugin with Azure Event Hubs. The output plugin uses librdkafka so you just have to configure the required connection string and messages arrive as expected. Just don't try and use message level compression as that is not supported in Azure Event Hubs.

lukasmrtvy commented 4 years ago

+1 Logstash and Filebeat support it:

edsiper commented 4 years ago

We just implemented Azure Blob support (will be part of 1.6 release in ~2 weeks), I am wondering if it solves the needs since the data can be streamed internally:

https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview

lukasmrtvy commented 4 years ago

Superb !

One thing, if I remember correctly, there is slightly delay in capturing from event hub to azure blob ( it takes up to 10 minutes, realtime alerts might be a real problem ) and Event Hubs Capture is charged, it means that one have to pay for event hub and event hub capture...

There is another option for getting Metrics and Logs from Azure via Azure Monitor REST. For example:

edsiper commented 4 years ago

FYI: here you can find the doc for the new plugin for 1.6 (GIT master):

https://docs.fluentbit.io/manual/v/master/pipeline/outputs/azure_blob

note: you can choose between blockblob and appendblob

lukasmrtvy commented 4 years ago

OP ( and me ) asked for input plugin, not output..

Anyway, Azure Storage ( blob ) output plugin is also good thing for storing logs to something more scalable/cheaper for archiving purposes.

arunp-motorq commented 3 years ago

Will be good to have an input plugin for Eventhub. Most of the azure services has support to stream logs to eventhubs

MagicJohnJang commented 3 years ago

Worked for me using kafka output to Azure EventHub

td-agent-bit.conf

[INPUT]
  Name                tail
  Path                xxx.log
  Refresh_Interval    10

[OUTPUT]
  Name        kafka
  Match       *
  brokers     xxx.xxx.windows.net:9093
  topics      xxx
  rdkafka.security.protocol   SASL_SSL
  rdkafka.sasl.username       $ConnectionString
  rdkafka.sasl.password       Endpoint=sb://xxx.xxx.windows.net/;SharedAccessKeyName=xxx;SharedAccessKey=xxx
  rdkafka.sasl.mechanism      PLAIN

[OUTPUT]
  name  stdout
  match *

Inside docker container (MUST HAVE or broker down/ssl fail)

docker-compose.yml

version: "3.7"
services:
  fluent-bit:
    image: fluent/fluent-bit:1.6.2
    container_name: fluentbit
    restart: always
    volumes:
      - ./td-agent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./xxx.log:/fluent-bit/etc/xxx.log:ro
GazRagzz commented 2 years ago

Works for me ! Thanks a lot !

techny210 commented 1 year ago

FYI: I am using the kafka output plugin with Azure Event Hubs. The output plugin uses librdkafka so you just have to configure the required connection string and messages arrive as expected. Just don't try and use message level compression as that is not supported in Azure Event Hubs.

care to share a sample config?

deepsandeepme commented 1 year ago

@MagicJohnJang Hi John. I understand that the event hub configurations worked for you. I am trying to implement a similar configuration with event hubs. However, Event hubs have a limitation of processing 1MB of data. Have you come across a situation where it may have created a blocker during implementation? Any idea how to ensure that the log file sent to event hub is less than 1MB?

Thanks in advance.

syedsuhaib commented 1 year ago

I am in the process of implementing a custom framework for handling Audit logs from various internal systems. These are not system or application logs, so we will be saving structured data which will include previous and changed states of arbitrary resources.

One of the key specs is for this framework to send these audit logs to multiple destinations for storage (we are considering Elasticsearch to begin with). So I thought fluentbit would be perfect for the processing and dispatching part. What I am missing is an input plugin to get logs from Azure Service Bus. As you can guess, we are getting the individual applications to push their audit logs to SB and then a listener application in my framework (or a fluentbit plugin!) picks it up form there and does the magic.

I would really appreciate any ideas on this from everyone here, if there's a better way to achieve what I'm trying to do and whether there already is an plugin that can help me get data from Azure Service Bus. I'm open to replacing SB with a different message broker/queue if that will be supported by fleuntbit. Thanks for reading.

debashisgho commented 4 months ago

Worked for me using kafka output to Azure EventHub

td-agent-bit.conf

[INPUT]
  Name                tail
  Path                xxx.log
  Refresh_Interval    10

[OUTPUT]
  Name        kafka
  Match       *
  brokers     xxx.xxx.windows.net:9093
  topics      xxx
  rdkafka.security.protocol   SASL_SSL
  rdkafka.sasl.username       $ConnectionString
  rdkafka.sasl.password       Endpoint=sb://xxx.xxx.windows.net/;SharedAccessKeyName=xxx;SharedAccessKey=xxx
  rdkafka.sasl.mechanism      PLAIN

[OUTPUT]
  name  stdout
  match *

Inside docker container (MUST HAVE or broker down/ssl fail)

docker-compose.yml

version: "3.7"
services:
  fluent-bit:
    image: fluent/fluent-bit:1.6.2
    container_name: fluentbit
    restart: always
    volumes:
      - ./td-agent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./xxx.log:/fluent-bit/etc/xxx.log:ro

what is $ConnectionString ?