search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.66k stars 3.42k forks source link

line 14: field azure_event_hubs not found in type scrapeconfig.plain #9562

Open nimblenitin opened 1 year ago

nimblenitin commented 1 year ago

Describe the bug Getting below error while trying to deploy prom tail helm chart to integrate promtail with azure eventhub

To Reproduce Steps to reproduce the behavior:

  1. helm install azure-promtail grafana/promtail --version 6.11.2 -f values.yaml -n observability

Expected behavior Promtail helm chart should deploy without error

Environment:

Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.

promtail config:

config:
  snippets:
    scrapeConfigs: |
      - job_name: azure_event_hubs
        azure_event_hubs:
          group_id: "mygroup"
          fully_qualified_namespace: xxxx.servicebus.windows.net:9093
          connection_string: "xxxx"
          event_hubs:
            - xxx-logs-dev
          labels:
            job: azure_event_hub
        relabel_configs:
          - action: replace
            source_labels:
              - __azure_event_hubs_category
            target_label: category

error:

  line 14: field azure_event_hubs not found in type scrapeconfig.plain
  line 25: cannot unmarshal !!str `__azure...` into model.LabelNames. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file
nimblenitin commented 1 year ago

@andriikushch can you please help?

andriikushch commented 1 year ago

Hi @nimblenitin,

Thank you very much for reporting this.

helm install azure-promtail grafana/promtail --version 6.11.2 -f values.yaml -n observability

I think this setup uses promtail (loki) version 2.8.x.

The azure_event_hubs target was not included in the latest releases. Adding it to the 2.8.x changelog was a mistake.

This target is already in the main branch, and I expect it will be included in the 2.9 release.

nimblenitin commented 1 year ago

thanks for letting me know! meanwhile is there any way I can still implement eventhub loki integration through some other way? Please do share any document if you have if any. thanks again. @andriikushch

andriikushch commented 1 year ago

Hi @nimblenitin,

There is an option to get the data from the Azure Event Hubs using the Kafka API. Important: This API is unavailable for the "Basic" pricing plan. Here is the description for pricing plans: https://azure.microsoft.com/en-us/pricing/details/event-hubs/.

To do so, please try the following:

Config example:

scrape_configs:
 - job_name: azure
   kafka:
     group_id: <MY-GROUP>
     version: "1.0.0"
     brokers:
       - <MY-NAMESPACE>.servicebus.windows.net:9093
     topics:
       - <MY-TOPIC>
     labels:
       job: azure
     authentication:
       type: "sasl"
       tls_config:
         insecure_skip_verify: true
       sasl_config:
         user: "$ConnectionString"
         password: <MY-CONNECTION-STRING-VALUE>
         use_tls: true
         insecure_skip_verify: true
         mechanism: "PLAIN"

This config will configure the Promtail Kafka target to get data from the Azure Event Hubs. Here is the link to the documentation https://grafana.com/docs/loki/latest/clients/promtail/scraping/#kafka.

The disadvantage of this approach is the following: When I used Azure Event Hubs to get the application logs from the infrastructure components and Kafka API to get the logs to Loki, then multiple logs/events from the Event Hubs were delivered in a single Kafka message and appeared as a single log line in the Loki. It might be acceptable or not for you depending on your usecase.

nimblenitin commented 1 year ago

thanks this helps. Does the eventhub target which is yet to be released fixes the issue of bunching of logs?

andriikushch commented 1 year ago

Hi @nimblenitin,

Yes, it will deliver these logs as separate lines. There are a few tests that demonstrate this behavior:

https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/azureeventhubs/parser_test.go

nimblenitin commented 1 year ago

appreciate the help. We will wait for the release.

TaylorMutch commented 9 months ago

This is still broken in promtail 2.8.7, tested this today locally and the syntax is correct based on the docs, but fails otherwise.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/loki-positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: azure_event_hubs
  azure_event_hubs:
    group_id: "promtaileventhubconsumergroup"
    fully_qualified_namespace: my-fqdn-test.servicebus.windows.net:9093
    connection_string: "$myconnstring"

Using

/promtail-darwin-amd64 --version
promtail, version 2.8.7 (branch: HEAD, revision: 1dfdc432c)
  build user:       root@5a462433bd8a
  build date:       2023-12-12T21:49:15Z
  go version:       go1.20.10
  platform:         darwin/amd64
yanniand commented 7 months ago

It works for me with promtail 2.9 (standard pricing plan event hub)

My promtail configuration is:

job_name: azure_event_hubs
azure_event_hubs:
  fully_qualified_namespace: ehns.servicebus.windows.net:9093
  connection_string: connection-string
  event_hubs: eh-name
  labels:
    job: azure_event_hub
relabel_configs:
  - action: replace
    source_labels:
      - __azure_event_hubs_category
    target_label: category

@andriikushch Is it possible to use the config you provided with a normal Kafka instance (not the one backed by Azure Event Hub) so that the costs are not very high?

andriikushch commented 6 months ago

Hi @yanniand,

Thank you for your question.

There is an option to use promtail with Kafka:

There are some more examples at https://github.com/grafana/loki/tree/main/clients/cmd/promtail: