New component: Azure Blob Exporter

hgaol commented 4 months ago

The purpose and use-cases of the new component

Azure Blob Storage is Microsoft's object storage solution for the cloud. Blob Storage is optimized for storing massive amounts of unstructured data. It's also used to batch analysis via ADLA, Synapse and other services to analyze big data. link

Example configuration for the component

The following settings are required:

url: Must be specified if auth type is not connection_string. If auth type is connection_string, it's optional or will be override by the auth.connection_string. Azure storage account endpoint. e.g. https://<account-name>.blob.core.windows.net/
auth (no default): Authentication method for exporter to ingest data.
- type (no default): Authentication type for expoter. supported values are: connection_string, service_principal, system_managed_identity, user_managed_identity and etc.
- tenand_id: Tenand Id for the client, only needed when type is service_principal.
- client_id: Client Id for the auth, only needed when type is service_principal and user_managed_identity.
- client_secret: Secret for the client, only needed when type is service_principal.
- connection_string: Connection string to the endpoint. Only needed for connection_string auth type. Once provided, it'll override the url parameter to the storage account.

The following settings can be optionally configured and have default values:

container: container for metrics, logs and traces.
- metrics (default metrics): container to store metrics. default value is metrics.
- logs (default logs): container to store logs. default value is logs.
- traces (default traces): container to store traces. default value is traces.
blob_name_format:
- format (default {{.Year}}/{{.Month}}/{{.Day}}/{{.BlobName}}_{{.Hour}}_{{.Minute}}_{{.Second}}_{{.SerialNum}}.{{.FileExtension}}): blob name format.
- blob_name: value of BlobName.
- metrics (default metrics): default is metrics.
- traces (default is traces): default is traces.
- logs (default is logs): default is logs.
- year (default 2006): The date format follows constants in Golang, refer here.
- month (default 01): similar as year.
- day (default 02): similar as year.
- hour (default 15): similar as year.
- minute (default 04): similar as year.
- second (default 05): similar as year.
format (default json): json or proto. which present otel json or otel protobuf format, the file extension will be json or pb.
encoding (default none): if specified, uses an encoding extension to encode telemetry data. Overrides format.

An example configuration is provided as follows:

exporters:
  azureblob:
    # storage account url
    url: "https://<account-name>.blob.core.windows.net/"
    # Authentication method for exporter to ingest data.
    auth:
      #  Authentication type for expoter. supported values are: connection_string, service_principal, system_managed_identity, user_managed_identity and etc.
      type: "system_managed_identity"
    format: "json"
    container:
      metrics: "test"
      logs: "test"
      traces: "test"
  azureblob/2:
    url: "https://<account-name>.blob.core.windows.net/"
    auth:
      type: "service_principal"
      # Tenand Id for the client, only needed when type is service_principal.
      tenant_id: "xxxxxxxx-32c4-4a70-949c-c7bd7ff7af89"
      #  Client Id for the auth, only needed when type is service_principal and user_managed_identity.
      client_id: "xxxxxxx-46e6-456a-9ee6-cf3b9cb23608"
      # Secret for the client, only needed when type is service_principal.
      client_secret: "<client secret>"
    # `json` or `proto`. which present otel json or otel protobuf format, the file extension will be `json` or `pb`.
    format: "proto"
    # container name for metrics, logs and traces
    container:
      metrics: "test"
      logs: "test"
      traces: "test"
  azureblob/3:
    auth:
      type: "connection_string"
      connection_string: "DefaultEndpointsProtocol=https;AccountName=<account-name>;AccountKey=<account key>;EndpointSuffix=core.windows.net"
    container:
      metrics: "test"
      logs: "test"
      traces: "test"

Telemetry data types supported

traces, metrics, logs

Is this a vendor-specific component?

[X] This is a vendor-specific component
[ ] If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
[ ] If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@hgaol

Sponsor (optional)

No response

Additional context

I found there's an issue about Azure Blob exporter 2 years ago and the issue is marked as staled #8823 . Thus, I'd like to contribute this component with some more functionalities like more authentication methods, signals and others.

hgaol commented 2 months ago

The implementation could be found here. https://github.com/hgaol/opentelemetry-collector-contrib/tree/azureblob-exporter-dev

example ocb config.

dist:
  module: go.opentelemetry.io/collector/cmd/otelcorecol
  name: otelcorecol
  description: Local OpenTelemetry Collector binary, testing only.
  version: 0.103.0-dev
  otelcol_version: 0.103.0
  output_path: .

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.103.0
exporters:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/azureblobexporter v0.1.0

providers:
  - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v0.103.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v0.103.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/httpprovider v0.103.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/httpsprovider v0.103.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v0.103.0

replaces:
  - github.com/open-telemetry/opentelemetry-collector-contrib/exporter/azureblobexporter =>  github.com/hgaol/opentelemetry-collector-contrib/exporter/azureblobexporter v0.1.0

example otel-config.yaml.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318

exporters:
  azureblob/3:
    auth:
      type: "connection_string"
      connection_string: "DefaultEndpointsProtocol=https;AccountName=<accout name>;AccountKey=<key>;EndpointSuffix=core.windows.net"
    container:
      metrics: "test"
      logs: "test"
      traces: "test"

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      exporters: [ azureblob/3 ]
    logs:
      receivers: [ otlp ]
      exporters: [ azureblob/3 ]
    metrics:
      receivers: [ otlp ]
      exporters: [ azureblob/3 ]

example output.

2024-09-02T23:48:07.132+0800    info    service@v0.103.0/service.go:208 Everything is ready. Begin running and processing data.
2024-09-02T23:48:07.132+0800    warn    localhostgate/featuregate.go:63 The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.   {"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-09-02T23:48:11.318+0800    info    azureblobexporter@v0.1.0/exporter.go:225        Successfully exported traces to Azure Blob Storage      {"kind": "exporter", "data_type": "traces", "name": "azureblob/3", "container": "test", "blob": "2024/09/02/traces_23_48_11_518409.json", "size": 2476}

demo. azureblobexporter

atoulme commented 1 month ago

Please come to a SIG meeting to discuss this exporter.

hgaol commented 1 month ago

Please come to a SIG meeting to discuss this exporter.

Thanks for response @atoulme ! I've joined SIG meeting at Aug 28, 2024 for this new component. I got suggestions that it'll not be assigned a sponsor automatically and I need to find one.

open-telemetry / opentelemetry-collector-contrib