Telegraf output plugin for Azure Data Explorer

[minwal]

Feature Request

Opening a feature request kicks off a discussion.

Proposal: My team wants to contribute by building an output plugin for Azure Data Explorer(ADX)

Current behavior: No such output plugin available for Telegraf

Desired behavior:

Use case: Lot of large enterprise customers requested for an ADX output plugin so they can ingest data collected using various input plugins directly to ADX.

Azure Data Explorer is a big data analytics platform, purpose built for logs and telemetry analytics scenarios.

[tomconte]

Hi Minnie, just doing some research on this proposal, and I had a couple questions you might be able to help me with:

• Could you share (maybe privately) some throughput and/or performance requirements for the plugin?
• Which ingestion type should this plugin use: batching or streaming?
• Does the Go SDK support both ingestion options?

cc/ @minwal @yannickbrombach

[minwal]

@ssoroka @helenosheaa @Jagularr - Hi All, Can you please help me on the next steps for this request or connect me with the relevant folks? Looking for your feedback on following -

1. Influx requires to sign Contributor License Agreement (CLA). There is a segment where it mentions a Corporate CLA rather than Individual CLA - https://www.influxdata.com/legal/cla/
   Whom to contact to start this process?
2. I will be creating a PR for this plugin soon, is there a way to schedule sometime with this repo's maintainers/experts to review it together and brainstorm few scenarios?

[tomconte]

Hi @minwal, regarding 1. I have initiated the internal review process for the CLA. I will forward you the details in DM.

[minwal]

Thanks Thomas, please let me know if any action required from my side. Is your team responsible for all Telegraf contributions integration from process perspective?

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Thomas Conté @.> Sent: Friday, June 4, 2021 5:35:29 PM To: influxdata/telegraf @.> Cc: Minni Walia @.>; Mention @.> Subject: Re: [influxdata/telegraf] Telegraf output plugin for Azure Data Explorer (#9230)

Hi @minwalhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fminwal&data=04%7C01%7Cminni.walia%40microsoft.com%7C697b91073bd847d0fee508d9272b5420%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637583889331618918%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SptHmQAnh1APPMjULTaTV5e8%2B5tCH2V2KwIKuamC1sQ%3D&reserved=0, regarding 1. I have initiated the internal review process for the CLA. I will forward you the details in DM.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Finfluxdata%2Ftelegraf%2Fissues%2F9230%23issuecomment-854443253&data=04%7C01%7Cminni.walia%40microsoft.com%7C697b91073bd847d0fee508d9272b5420%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637583889331628881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TPym2X%2F4DCV4mRivWiANBbzcX4SEGx4amiaiTBbDkx8%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKKPAGU5LXNNWPTR4WZXJIDTRB64DANCNFSM44A5B6SQ&data=04%7C01%7Cminni.walia%40microsoft.com%7C697b91073bd847d0fee508d9272b5420%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637583889331628881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zOurVq9tGHTgrSkd3m%2FLRsK%2FR2i%2BAVahjBYHeeNRXoU%3D&reserved=0.

[helenosheaa]

Hi @minwal,

Our typical process is if you are looking for feedback on ideas/implementation details you can post them on the InfluxDB Community slack channels #telegraf or #telegraf-dev. You can join this slack channel here.

Then once you're ready with your PR, submit it and you'll get feedback/reviews from the community and engineers on the Telegraf team. There's also some info on the review process here.

Thanks!

[minwal]

Hi @minwal,

Our typical process is if you are looking for feedback on ideas/implementation details you can post them on the InfluxDB Community slack channels #telegraf or #telegraf-dev. You can join this slack channel here.

Then once you're ready with your PR, submit it and you'll get feedback/reviews from the community and engineers on the Telegraf team. There's also some info on the review process here.

Thanks!

Thanks @helenosheaa! Posted question on slack channel, hoping to get a quick response from the community.

[sjwang90]

Questions from @minwal about writing the Telegraf ADX output

In the Write method, array of telegraf.Metric object is received so is there any implicit convention to group metrics into buckets/namespaces within the same inputter? e.g. I am testing ADX outputter with SQL inputter – there are ~900 metrics collected by SQL inputter that are bucketed into following namespaces depending on database_type

  ## database_type =  AzureSQLDB  by default collects the following queries

  ## - AzureSQLDBWaitStats

  ## - AzureSQLDBResourceStats

  ## - AzureSQLDBResourceGovernance

  ## - AzureSQLDBDatabaseIO

  ## - AzureSQLDBServerProperties

  ## - AzureSQLDBOsWaitstats

  ## - AzureSQLDBMemoryClerks

  ## - AzureSQLDBPerformanceCounters

  ## - AzureSQLDBRequests

  ## - AzureSQLDBSchedulers

from: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver#configuration

This bucketing logic is very specific to SQL inputter, it is already implemented by Azure monitor outputter like this –

https://github.com/influxdata/telegraf/blob/da7f2c7a934b5dc763ab10c7db02fe65b201686e/plugins/outputs/azure_monitor/azure_monitor.go#L420

Looking for suggestions if there is a better and generic way that can work for any input plugin:

1. To group relevant metrics into specific buckets/namespaces so data can be persisted as single table for each namespace rather than a table for each metric
2. Also is there a way to identify data is being generated by which inputter?