search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.2k stars 1.6k forks source link

azure_monitor_logs canot recreate table once deleted in Azure #21832

Open saikumarch7548 opened 2 days ago

saikumarch7548 commented 2 days ago

A note for the community

Problem

I tried deleting the Azure custom table created by below configuration and redeployed it since I filtered only the data I need. Vector canot create the Azure log analytics table again Vector didnt show any logs/errors related to the table creation Vector didnt published the results Vector works okay if we provide a new table name

Configuration

sources:
      rabbitmq_api:
        type: http_client
        endpoint: "http://rabbitmq-epd.medicoretest-epd-rabbitmq.svc.cluster.local:15672/api/queues"  # dynamic FQDN
        decoding:
          codec: json
        method: GET
        auth:
          strategy: basic
          user: 'masked'
          password: 'masked'
        headers:
          Accept:
            - "application/json"
        scrape_interval_secs: 60 # Set scraping interval in seconds
        scrape_timeout_secs: 5
    transforms:
      extract_fields:
        type: remap
        inputs:
          - rabbitmq_api
        drop_on_abort: true
        source: |
          # Access and extract necessary fields from the parsed event
          .vhost = .vhost          # Extract the 'vhost' field
          .queue = .name           # Extract the 'name' field (queue name)
          .messages_ready = .messages_ready   # Extract the 'messages_ready' field
          # Return a structured event with these fields
          . = {
            "vhost": .vhost, 
            "queue": .queue, 
            "messages_ready": .messages_ready 
          }
    # sinks:
    #   console:
    #     type: console
    #     inputs:
    #       - extract_fields
    #     encoding:
    #       codec: json  # Add the encoding for the console sink (format the output as JSON)
    sinks:
      azure_monitor_logs:
        # batch:
        #   max_events: 500
        #   timeout_secs: 30
        customer_id: "masked"
        inputs:
        - extract_fields
        log_type: "masked"
        shared_key: "masked"
        tls:
          ca_file: /etc/ssl/certs/ca-certificates.crt
          server_name: ods.opinsights.azure.com
          verify_certificate: true
          verify_hostname: true
        type: azure_monitor_logs

### Version

timberio/vector:0.42.0-distroless-libc

### Debug Output

```text
2024-11-19T18:19:20.212173Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-11-19T18:19:20.212218Z  INFO vector::app: Log level is enabled. level="debug"
2024-11-19T18:19:20.212275Z DEBUG vector::app: messaged="Building runtime." worker_threads=1
2024-11-19T18:19:20.212613Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2024-11-19T18:19:20.213861Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2024-11-19T18:19:20.214894Z DEBUG vector::topology::builder: Building new source. component=rabbitmq_api
2024-11-19T18:19:20.215026Z DEBUG vector::topology::builder: Building new transform. component=extract_fields
2024-11-19T18:19:20.215315Z DEBUG vector::topology::builder: Building new sink. component=azure_monitor_logs
2024-11-19T18:19:20.240634Z  INFO vector::topology::running: Running healthchecks.
2024-11-19T18:19:20.240665Z DEBUG vector::topology::running: Connecting changed/added component(s).
2024-11-19T18:19:20.240675Z DEBUG vector::topology::running: Configuring outputs for source. component=rabbitmq_api
2024-11-19T18:19:20.240689Z DEBUG vector::topology::running: Configuring output for component. component=rabbitmq_api output_id=None
2024-11-19T18:19:20.240695Z DEBUG vector::topology::running: Configuring outputs for transform. component=extract_fields
2024-11-19T18:19:20.240701Z DEBUG vector::topology::running: Configuring output for component. component=extract_fields output_id=None
2024-11-19T18:19:20.240707Z DEBUG vector::topology::running: Connecting inputs for transform. component=extract_fields
2024-11-19T18:19:20.240716Z DEBUG vector::topology::running: Adding component input to fanout. component=extract_fields fanout_id=rabbitmq_api
2024-11-19T18:19:20.240750Z DEBUG vector::topology::running: Connecting inputs for sink. component=azure_monitor_logs
2024-11-19T18:19:20.240757Z DEBUG vector::topology::running: Adding component input to fanout. component=azure_monitor_logs fanout_id=extract_fields
2024-11-19T18:19:20.240770Z DEBUG vector::topology::running: Spawning new source. key=rabbitmq_api
2024-11-19T18:19:20.240805Z DEBUG vector::topology::running: Spawning new transform. key=extract_fields
2024-11-19T18:19:20.240853Z  INFO vector: Vector has started. debug="false" version="0.42.0" arch="x86_64" revision="3d16e34 2024-10-21 14:10:14.375255220"
2024-11-19T18:19:20.240868Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-11-19T18:19:20.240965Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=https://id.ods.opinsights.azure.com/api/logs?api-version=2016-04-01 method=POST version=HTTP/1.1 headers={"log-type": "epdRabbitMQMetrics", "time-generated-field": "timestamp", "content-type": "application/json", "authorization": Sensitive, "x-ms-date": "Tue, 19 Nov 2024 18:19:20 GMT", "user-agent": "Vector/0.42.0 (x86_64-unknown-linux-gnu 3d16e34 2024-10-21 14:10:14.375255220)", "accept-encoding": "identity"} body=[2 bytes]
2024-11-19T18:19:20.241204Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}: vector::topology::builder: Source pump supervisor starting.
2024-11-19T18:19:20.241224Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}: vector::topology::builder: Source pump starting.
2024-11-19T18:19:20.241232Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}: vector::topology::builder: Source starting.
2024-11-19T18:19:20.241616Z DEBUG hyper::client::connect::dns: resolving host="id.ods.opinsights.azure.com"
2024-11-19T18:19:20.247345Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}: vector_core::tls::settings: Fetching system root certs.
2024-11-19T18:19:20.252968Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}: vector_core::tls::settings: Fetching system root certs.
2024-11-19T18:19:20.311193Z DEBUG transform{component_kind="transform" component_id=extract_fields component_type=remap}: vector::topology::builder: Synchronous transform starting.
2024-11-19T18:19:20.311219Z DEBUG sink{component_kind="sink" component_id=azure_monitor_logs component_type=azure_monitor_logs}: vector::topology::builder: Sink starting.
2024-11-19T18:19:20.311279Z DEBUG http: hyper::client::connect::http: connecting to 52.236.186.221:443
2024-11-19T18:19:20.311457Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://url:15672/api/queues method=GET version=HTTP/1.1 headers={"accept": "application/json", "accept": "text/plain", "authorization": Sensitive, "user-agent": "Vector/0.42.0 (x86_64-unknown-linux-gnu 3d16e34 2024-10-21 14:10:14.375255220)", "accept-encoding": "identity"} body=[empty]
2024-11-19T18:19:20.311517Z DEBUG sink{component_kind="sink" component_id=azure_monitor_logs component_type=azure_monitor_logs}: vector::utilization: utilization=0.1066887289904428
2024-11-19T18:19:20.311534Z DEBUG hyper::client::connect::dns: resolving host="url"
2024-11-19T18:19:20.314757Z DEBUG http: hyper::client::connect::http: connected to 52.236.186.221:443
2024-11-19T18:19:20.327349Z DEBUG hyper::proto::h1::io: flushed 504 bytes
2024-11-19T18:19:20.329745Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}:http: hyper::client::connect::http: connecting to 10.0.63.245:15672
2024-11-19T18:19:20.331536Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}:http: hyper::client::connect::http: connected to 10.0.63.245:15672
2024-11-19T18:19:20.331612Z DEBUG hyper::proto::h1::io: flushed 374 bytes
2024-11-19T18:19:20.341898Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-11-19T18:19:20.341909Z DEBUG hyper::proto::h1::conn: incoming body is content-length (44621 bytes)
2024-11-19T18:19:20.341942Z DEBUG source{component_kind="source" component_id=rabbitmq_api component_type=http_client}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"cache-control": "no-cache", "content-length": "44621", "content-security-policy": "script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src 'self'", "content-type": "application/json", "date": "Tue, 19 Nov 2024 18:19:20 GMT", "server": "Cowboy", "vary": "accept, accept-encoding, origin"} body=[44621 bytes]
2024-11-19T18:19:20.343385Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-11-19T18:19:20.343401Z DEBUG hyper::client::pool: pooling idle connection for ("http", url:15672)
2024-11-19T18:19:20.469458Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-11-19T18:19:20.469477Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-11-19T18:19:20.469508Z DEBUG http: hyper::client::pool: pooling idle connection for ("https", id.ods.opinsights.azure.com)
2024-11-19T18:19:20.469524Z DEBUG http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"cache-control": "private", "server": "Microsoft-IIS/10.0", "x-aspnet-version": "4.0.30319", "x-powered-by": "ASP.NET", "strict-transport-security": "max-age=31536000; includeSubDomains", "date": "Tue, 19 Nov 2024 18:19:20 GMT", "content-length": "0"} body=[empty]
2024-11-19T18:19:20.469564Z  INFO vector::topology::builder: Healthcheck passed.
2024-11-19T18:19:21.344864Z DEBUG sink{component_kind="sink" component_id=azure_monitor_logs component_type=azure_monitor_logs}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://id.ods.opinsights.azure.com/api/logs?api-version=2016-04-01 method=POST version=HTTP/1.1 headers={"log-type": "epdRabbitMQMetrics", "time-generated-field": "timestamp", "content-type": "application/json", "authorization": Sensitive, "x-ms-date": "Tue, 19 Nov 2024 18:19:21 GMT", "user-agent": "Vector/0.42.0 (x86_64-unknown-linux-gnu 3d16e34 2024-10-21 14:10:14.375255220)", "accept-encoding": "identity"} body=[2870 bytes]
2024-11-19T18:19:21.344903Z DEBUG sink{component_kind="sink" component_id=azure_monitor_logs component_type=azure_monitor_logs}:request{request_id=1}:http: hyper::client::pool: reuse idle connection for ("https", id.ods.opinsights.azure.com)
2024-11-19T18:19:21.345016Z DEBUG hyper::proto::h1::io: flushed 3375 bytes
2024-11-19T18:19:21.441689Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-11-19T18:19:21.441703Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-11-19T18:19:21.441741Z DEBUG sink{component_kind="sink" component_id=azure_monitor_logs component_type=azure_monitor_logs}:request{request_id=1}:http: hyper::client::pool: pooling idle connection for ("https", id.ods.opinsights.azure.com)
2024-11-19T18:19:21.441761Z DEBUG sink{component_kind="sink" component_id=azure_monitor_logs component_type=azure_monitor_logs}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"cache-control": "private", "server": "Microsoft-IIS/10.0", "x-aspnet-version": "4.0.30319", "x-powered-by": "ASP.NET", "strict-transport-security": "max-age=31536000; includeSubDomains", "date": "Tue, 19 Nov 2024 18:19:20 GMT", "content-length": "0"} body=[empty]
2024-11-19T18:19:25.312174Z DEBUG sink{component_kind="sink" component_id=azure_monitor_logs component_type=azure_monitor_logs}: vector::utilization: utilization=0.010669637618939385


### Example Data

_No response_

### Additional Context

_No response_

### References

_No response_
jszwedko commented 12 hours ago

Hi @saikumarch7548 ,

I don't believe that the azure_monitor_logs sink creates the table (at least not explicitly; maybe Azure automatically creates it?); it expects the table to already exist. You can see in the debug logs that Vector is sending logs to Azure and is receiving a 200 OK back so, from Vector's perspective, it seems like everything is working. The issue may like on the Azure side.

saikumarch7548 commented 10 hours ago

@jszwedko when I pass a new tablename that doesnt exist in Azure yet, It works properly. which API is vector using

jszwedko commented 8 hours ago

@jszwedko when I pass a new tablename that doesnt exist in Azure yet, It works properly. which API is vector using

Interesting, maybe Azure is automatically creating the table? The API Vector uses is this one: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api