search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.69k stars 1.56k forks source link

Azure blob sink don't work with azure-workload-identity #17376

Closed alexey-ban closed 1 year ago

alexey-ban commented 1 year ago

A note for the community

Problem

Azure blob sink don't work with azure-workload-identity

Configuration

sinks:
  azure:
    type: azure_blob
    encoding:
      codec: text
    inputs:
      - nginx_logs
    container_name: proxy-logs
    blob_prefix: "test-proxy/%F/"
    storage_account: my-storage-account
    blob_append_uuid: true

sources:
  nginx_logs:
    include:
    - /var/log/nginx/proxy*.log
    type: file

Version

0.29.1

Debug Output

2023-05-12T10:14:04.652936Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-05-12T10:14:04.652984Z  INFO vector::app: Log level is enabled. level="debug"
2023-05-12T10:14:04.654219Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2023-05-12T10:14:04.654906Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-05-12T10:14:04.656146Z DEBUG vector::topology::builder: Building new source. component=nginx_logs
2023-05-12T10:14:04.656506Z DEBUG vector::topology::builder: Building new sink. component=azure
2023-05-12T10:14:04.656849Z DEBUG azure_core::http_client::reqwest: instantiating an http client using the reqwest backend    
2023-05-12T10:14:04.670518Z DEBUG azure_core::http_client::reqwest: instantiating an http client using the reqwest backend    
2023-05-12T10:14:04.682310Z DEBUG azure_core::http_client::reqwest: instantiating an http client using the reqwest backend    
2023-05-12T10:14:04.693310Z  INFO vector::topology::running: Running healthchecks.
2023-05-12T10:14:04.693469Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-05-12T10:14:04.693540Z DEBUG vector::topology::running: Configuring outputs for source. component=nginx_logs
2023-05-12T10:14:04.693546Z DEBUG azure_core::http_client::reqwest: performing request GET 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com%2F' with `reqwest`    
2023-05-12T10:14:04.693618Z DEBUG vector::topology::running: Configuring output for component. component=nginx_logs output_id=None
2023-05-12T10:14:04.693652Z DEBUG reqwest::connect: starting new connection: http://169.254.169.254/    
2023-05-12T10:14:04.693707Z DEBUG hyper::client::connect::http: connecting to 169.254.169.254:80
2023-05-12T10:14:04.693710Z DEBUG vector::topology::running: Connecting inputs for sink. component=azure
2023-05-12T10:14:04.693853Z DEBUG vector::topology::running: Adding component input to fanout. component=azure fanout_id=nginx_logs
2023-05-12T10:14:04.693962Z DEBUG vector::topology::running: Spawning new source. key=nginx_logs
2023-05-12T10:14:04.694073Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="file" component_id="nginx_logs" group_id="2"
2023-05-12T10:14:04.694183Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="azure_blob" component_id="azure" group_id="3"
2023-05-12T10:14:04.694357Z  INFO vector: Vector has started. debug="false" version="0.29.1" arch="x86_64" revision="74ae15e 2023-04-20 14:50:42.739094536"
2023-05-12T10:14:04.694454Z DEBUG hyper::client::connect::http: connected to 169.254.169.254:80
2023-05-12T10:14:04.694458Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-05-12T10:14:04.694669Z DEBUG hyper::proto::h1::io: flushed 167 bytes
2023-05-12T10:14:04.694714Z DEBUG source{component_kind="source" component_id=nginx_logs component_type=file component_name=nginx_logs}: vector::topology::builder: Source pump supervisor starting.
2023-05-12T10:14:04.694737Z DEBUG source{component_kind="source" component_id=nginx_logs component_type=file component_name=nginx_logs}: vector::topology::builder: Source pump starting.
2023-05-12T10:14:04.694751Z DEBUG source{component_kind="source" component_id=nginx_logs component_type=file component_name=nginx_logs}: vector::topology::builder: Source starting.
2023-05-12T10:14:04.694761Z  INFO source{component_kind="source" component_id=nginx_logs component_type=file component_name=nginx_logs}: vector::sources::file: Starting file server. include=["/var/log/nginx/proxy*.log"] exclude=[]
2023-05-12T10:14:04.695365Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}: vector::topology::builder: Sink starting.
2023-05-12T10:14:04.696577Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}: vector::utilization: utilization=0.03114042320238586
2023-05-12T10:14:04.696725Z  INFO source{component_kind="source" component_id=nginx_logs component_type=file component_name=nginx_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-05-12T10:14:04.696992Z  WARN source{component_kind="source" component_id=nginx_logs component_type=file component_name=nginx_logs}:file_server: vector::internal_events::file::source: Currently ignoring file too small to fingerprint. file=/var/log/nginx/proxy.log
2023-05-12T10:14:04.703988Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-05-12T10:14:04.704020Z DEBUG hyper::proto::h1::conn: incoming body is content-length (2022 bytes)
2023-05-12T10:14:04.704059Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-05-12T10:14:04.704238Z DEBUG hyper::client::pool: pooling idle connection for ("http", 169.254.169.254)
2023-05-12T10:14:04.704779Z DEBUG azure_core::policies::transport: the following request will be passed to the transport policy: Request {
    url: Url {
        scheme: "https",
        cannot_be_a_base: false,
        username: "",
        password: None,
        host: Some(
            Domain(
                "my-storage-account.blob.core.windows.net",
            ),
        ),
        port: None,
        path: "/proxy-logs",
        query: Some(
            "restype=container",
        ),
        fragment: None,
    },
    method: Head,
    headers: Headers(
        {
            HeaderName(
                "x-ms-date",
            ): HeaderValue(
                "Fri, 12 May 2023 10:14:04 GMT",
            ),
            HeaderName(
                "content-length",
            ): HeaderValue(
                "0",
            ),
            HeaderName(
                "x-ms-version",
            ): HeaderValue(
                "2019-12-12",
            ),
            HeaderName(
                "user-agent",
            ): HeaderValue(
                "azsdk-rust-storage/0.6.0 (1.66.1; linux; x86_64)",
            ),
            HeaderName(
                "authorization",
            ): HeaderValue(
                "Bearer ###Hide_Bearer###",
            ),
        },
    ),
    body: Bytes(
        b"",
    ),
}    
2023-05-12T10:14:04.707161Z DEBUG azure_core::http_client::reqwest: performing request HEAD 'https://my-storage-account.blob.core.windows.net/proxy-logs?restype=container' with `reqwest`    
2023-05-12T10:14:04.707191Z DEBUG reqwest::connect: starting new connection: https://my-storage-account.blob.core.windows.net/    
2023-05-12T10:14:04.709214Z DEBUG hyper::client::connect::dns: resolving host="my-storage-account.blob.core.windows.net"
2023-05-12T10:14:04.727814Z DEBUG hyper::client::connect::http: connecting to 52.239.171.228:443
2023-05-12T10:14:04.730306Z DEBUG hyper::client::connect::http: connected to 52.239.171.228:443
2023-05-12T10:14:04.738220Z DEBUG hyper::proto::h1::io: flushed 2067 bytes
2023-05-12T10:14:04.762847Z DEBUG hyper::proto::h1::io: parsed 6 headers
2023-05-12T10:14:04.762871Z DEBUG hyper::proto::h1::conn: incoming body is empty
2023-05-12T10:14:04.762911Z DEBUG hyper::client::pool: pooling idle connection for ("https", my-storage-account.blob.core.windows.net)
2023-05-12T10:14:04.762971Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Invalid connection string specified component_kind="sink" component_type="azure_blob" component_id=azure component_name=azure

....

2023-05-12T10:23:29.082666Z DEBUG vector::sinks::azure_blob::request_builder: Sending events. bytes=964 events_len=1 blob="test-proxy/2023-05-12/1683887009-2cf3ddee-1575-4539-a63a-5b4683446435.log.gz" container="proxy-logs"
2023-05-12T10:23:29.082796Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}:request: azure_core::policies::transport: the following request will be passed to the transport policy: Request {
    url: Url {
        scheme: "https",
        cannot_be_a_base: false,
        username: "",
        password: None,
        host: Some(
            Domain(
                "my-storage-account.blob.core.windows.net",
            ),
        ),
        port: None,
        path: "/proxy-logs/test-proxy/2023-05-12/1683887009-2cf3ddee-1575-4539-a63a-5b4683446435.log.gz",
        query: None,
        fragment: None,
    },
    method: Put,
    headers: Headers(
        {
            HeaderName(
                "x-ms-blob-type",
            ): HeaderValue(
                "BlockBlob",
            ),
            HeaderName(
                "x-ms-version",
            ): HeaderValue(
                "2019-12-12",
            ),
            HeaderName(
                "authorization",
            ): HeaderValue(
                "Bearer ###Hide_Bearer###",
            ),

            ),
            HeaderName(
                "x-ms-blob-content-type",
            ): HeaderValue(
                "application/gzip",
            ),
            HeaderName(
                "x-ms-date",
            ): HeaderValue(
                "Fri, 12 May 2023 10:23:29 GMT",
            ),
            HeaderName(
                "content-length",
            ): HeaderValue(
                "964",
            ),
            HeaderName(
                "x-ms-blob-content-encoding",
            ): HeaderValue(
                "gzip",
            ),
            HeaderName(
                "user-agent",
            ): HeaderValue(
                "azsdk-rust-storage/0.6.0 (1.66.1; linux; x86_64)",
            ),
        },
    ),
    body: Bytes(
        b"###Hide_body###",
    ),
}   
2023-05-12T10:23:29.083026Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}:request: azure_core::http_client::reqwest: performing request PUT 'https://my-storage-account.blob.core.windows.net/proxy-logs/test-proxy/2023-05-12/1683887009-2cf3ddee-1575-4539-a63a-5b4683446435.log.gz' with `reqwest`    
2023-05-12T10:23:29.083067Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}:request: reqwest::connect: starting new connection: https://my-storage-account.blob.core.windows.net/    
2023-05-12T10:23:29.083129Z DEBUG hyper::client::connect::dns: resolving host="my-storage-account.blob.core.windows.net"
2023-05-12T10:23:29.102927Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}:request: hyper::client::connect::http: connecting to 52.239.171.228:443
2023-05-12T10:23:29.104412Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}:request: hyper::client::connect::http: connected to 52.239.171.228:443
2023-05-12T10:23:29.113439Z DEBUG hyper::proto::h1::io: flushed 3194 bytes
2023-05-12T10:23:29.206212Z DEBUG hyper::proto::h1::io: parsed 7 headers
2023-05-12T10:23:29.206250Z DEBUG hyper::proto::h1::conn: incoming body is content-length (279 bytes)
2023-05-12T10:23:29.206265Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-05-12T10:23:29.206306Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}:request: hyper::client::pool: pooling idle connection for ("https", my-storage-account.blob.core.windows.net)
2023-05-12T10:23:29.206381Z ERROR sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}: vector::sinks::util::retries: Unexpected error type; dropping the request. error=server returned error status which will not be retried: 403 internal_log_rate_limit=true
2023-05-12T10:23:29.206418Z  WARN sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}: vector::sinks::util::adaptive_concurrency::controller: Unhandled error response. error=server returned error status which will not be retried: 403 internal_log_rate_limit=true
2023-05-12T10:23:29.206495Z ERROR sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Error { context: Full(Custom { kind: HttpResponse { status: Forbidden, error_code: Some("AuthorizationPermissionMismatch") }, error: HttpError { status: Forbidden, details: ErrorDetails { code: Some("AuthorizationPermissionMismatch"), message: None }, headers: {"content-type": "application/xml", "x-ms-request-id": "4eb09363-701e-0044-6dbb-84e499000000", "content-length": "279", "x-ms-version": "2019-12-12", "server": "Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "date": "Fri, 12 May 2023 10:23:28 GMT", "x-ms-error-code": "AuthorizationPermissionMismatch"}, body: b"\xef\xbb\xbf<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.\nRequestId:4eb09363-701e-0044-6dbb-84e499000000\nTime:2023-05-12T10:23:29.2171425Z</Message></Error>" } }, "server returned error status which will not be retried: 403") }) request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-05-12T10:23:29.206569Z ERROR sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2023-05-12T10:23:29.696310Z DEBUG sink{component_kind="sink" component_id=azure component_type=azure_blob component_name=azure}: vector::utilization: utilization=0.000000037991283816715964

Example Data

No response

Additional Context

Vector agent run as sidecar and try use azure workload identity auth model

Some env in pod for auth. ~ # env | grep AZURE_ AZURE_AUTHORITY_HOST=https://login.microsoftonline.com/ AZURE_CLIENT_ID=###DEL_CRED### AZURE_FEDERATED_TOKEN_FILE=/var/run/secrets/azure/tokens/azure-identity-token AZURE_TENANT_ID=###DEL_CRED###

P.S. This auth have "Contributor" role and tested via python library

References

No response

zamazan4ik commented 1 year ago

Not related to your problem but:

url: Url {
        scheme: "https",
        cannot_be_a_base: false,
        username: "",
        password: None,
        host: Some(

I do not like logging sensitive information like passwords even in debug mode. I am curious - could we do something about that?

neuronull commented 1 year ago

I do not like logging sensitive information like passwords even in debug mode. I am curious - could we do something about that?

Hmm, I think that might be coming from the upstream azure crate:

2023-05-12T10:14:04.704779Z DEBUG azure_core::policies::transport: the following request will be passed to the transport policy: Request {
fernandowiek commented 1 year ago

Hey @alexey-ban. I'm facing the same problem here. If I use connection_string, it works. Did you manage to resolve this or Vector really didn't work with azure-workload-identity?

alexey-ban commented 1 year ago

Hi @fernandowiek Unfortunately no. Looks like vector didn't work with azure-workload-identity

fernandowiek commented 1 year ago

@alexey-ban I tried to use workload-identity sidecar (https://learn.microsoft.com/en-us/azure/aks/workload-identity-migrate-from-pod-identity#deploy-the-workload-with-migration-sidecar) along with vector, but still not working. I received the message: "Healthcheck failed." error=failed to get bearer token component_kind="sink" component_type="azure_blob"

2023-07-12T22:13:54.617697Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info" 2023-07-12T22:13:54.618435Z INFO vector::app: Loading configs. paths=["/etc/vector"] 2023-07-12T22:13:54.624925Z INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs component_name=kube_log}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="aks-default-" 2023-07-12T22:13:54.632052Z INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs component_name=kube_log}: vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"] 2023-07-12T22:13:54.656506Z INFO vector::topology::running: Running healthchecks. 2023-07-12T22:13:54.656776Z INFO vector: Vector has started. debug="false" version="0.29.1" arch="x86_64" revision="74ae15e 2023-04-20 14:50:42.739094536" 2023-07-12T22:13:54.656857Z INFO vector::app: API is disabled, enable by settingapi.enabledtotrueand use commands likevector top. 2023-07-12T22:13:54.659534Z INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs component_name=kube_log}:file_server: file_source::checkpointer: Loaded checkpoint data. 2023-07-12T22:13:54.659550Z ERROR vector::topology::builder: msg="Healthcheck failed." error=failed to get bearer token component_kind="sink" component_type="azure_blob" component_id=azure_blob component_name=azure_blob

@jszwedko It would be great if this functionality were in Vector's backlog.

fernandowiek commented 1 year ago

Good news @alexey-ban! Using the workload-identity sidecar and disabling the healthcheck on the sink configuration, data is being written to the azure storage container. However, I'm getting a lot of WARN messages that, at least for now, are not causing issues:

2023-07-13T21:35:48.784252Z WARN source{component_kind="source" component_id=kube_log component_type=kubernetes_logs component_name=kube_log}:file_server: vector::internal_events::file::source: Currently ignoring file too small to fingerprint. file=/var/log/pods/0.log 4 2023-07-13T21:44:32.327262Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(ReadEvents(Custom { kind: Other, error: hyper::Error(Body, hyper::Error(Body, Kind(TimedOut))) })) 3 2023-07-13T21:59:07.661201Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(ReadEvents(Custom { kind: Other, error: hyper::Error(Body, hyper::Error(Body, Kind(TimedOut))) })) 2 2023-07-13T22:08:24.987531Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(ReadEvents(Custom { kind: Other, error: hyper::Error(Body, hyper::Error(Body, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) })) 1 2023-07-13T22:13:20.990182Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=WatchFailed(ReadEvents(Custom { kind: Other, error: hyper::Error(Body, hyper::Error(Body, Kind(TimedOut))) }))

sudhirpandey commented 1 year ago

Can azure workload identity be support be implemented natively, because sider car approach is recommended to be not used in production setip 

 Note

The migration sidecar is not supported for production use. This feature is meant to give you time to migrate your application SDK's to a supported version, and not meant or intended to be a long-term solution.

https://learn.microsoft.com/en-us/azure/aks/workload-identity-migrate-from-pod-identity

Sofar sidecar approach has worked for us but we would very much prefer to use the native capability with in vector to be able to use workload idenity.

dsmith3197 commented 1 year ago

Support for azure workload identity was only recently added to the upstream crate (PR). This will be included in the next version of Vector and should work then.

Please re-open this ticket if it still does not work in the next Vector release (v0.33.0).