Add support of Processors in all the Observability Integrations

[ishleenk17]

Processors are used to provide the ability to the user to enrich their data. Currently, there is inconsistency in the Integrations wrt availability of Processors.

• [x] Identify the Packages/datastreams missing the Processors
• [x] Do bulk addition of Processors to the Integrations
• [x] Test the changes

Expectation: Processors yaml box like below should be present in all Integrations

[Linu-Elias]

The list of packages with missing processor:

• [ ] Activemq https://github.com/elastic/integrations/blob/main/packages/activemq/data_stream/broker https://github.com/elastic/integrations/blob/main/packages/activemq/data_stream/queue https://github.com/elastic/integrations/blob/main/packages/activemq/data_stream/topic

• [ ] Apache https://github.com/elastic/integrations/blob/main/packages/apache/data_stream/status

• [ ] Apache Spark https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/application https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/driver https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/executor https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/node

• [ ] AWSFargate https://github.com/elastic/integrations/blob/main/packages/awsfargate/data_stream/task_stats

• [ ] Azure Application Insights https://github.com/elastic/integrations/blob/main/packages/azure_application_insights/data_stream/app_insights https://github.com/elastic/integrations/blob/main/packages/azure_application_insights/data_stream/app_state

• [ ] Azure Billing https://github.com/elastic/integrations/blob/main/packages/azure_billing/data_stream/billing

• [ ] Azure Functions https://github.com/elastic/integrations/blob/main/packages/azure_functions/data_stream/metrics

• [ ] Azure OpenAI https://github.com/elastic/integrations/blob/main/packages/azure_openai/data_stream/metrics

• [ ] Cassandra https://github.com/elastic/integrations/blob/main/packages/cassandra/data_stream/metrics

• [ ] CockroachDB https://github.com/elastic/integrations/blob/main/packages/cockroachdb/data_stream/status

• [ ] Etcd https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/leader https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/metrics https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/self https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/store

• [ ] HAProxy https://github.com/elastic/integrations/blob/main/packages/haproxy/data_stream/info https://github.com/elastic/integrations/blob/main/packages/haproxy/data_stream/stat

• [ ] InfluxDB https://github.com/elastic/integrations/blob/main/packages/influxdb/data_stream/advstatus https://github.com/elastic/integrations/blob/main/packages/influxdb/data_stream/status

• [ ] Jolokia Input https://github.com/elastic/integrations/blob/main/packages/jolokia_input/agent/

• [ ] Kafka https://github.com/elastic/integrations/blob/main/packages/kafka/data_stream/broker https://github.com/elastic/integrations/blob/main/packages/kafka/data_stream/consumergroup https://github.com/elastic/integrations/blob/main/packages/kafka/data_stream/partition

• [ ] Memcached https://github.com/elastic/integrations/blob/main/packages/memcached/data_stream/stats

• [ ] Microsoft SQLServer https://github.com/elastic/integrations/blob/main/packages/microsoft_sqlserver/data_stream/performance https://github.com/elastic/integrations/blob/main/packages/microsoft_sqlserver/data_stream/transaction_log

• [ ] MongoDB https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/collstats https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/dbstats https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/metrics https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/replstatus https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/status

• [ ] MYSQL https://github.com/elastic/integrations/blob/main/packages/mysql/data_stream/galera_status https://github.com/elastic/integrations/blob/main/packages/mysql/data_stream/performance https://github.com/elastic/integrations/blob/main/packages/mysql/data_stream/status

• [ ] Nginx https://github.com/elastic/integrations/blob/main/packages/nginx/data_stream/stubstatus

• [ ] Oracle WebLogic https://github.com/elastic/integrations/blob/main/packages/oracle_weblogic/data_stream/deployed_application https://github.com/elastic/integrations/blob/main/packages/oracle_weblogic/data_stream/threadpool

• [ ] PostgreSQL https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/activity https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/bgwriter https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/database https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/statement

• [ ] Prometheus Input https://github.com/elastic/integrations/blob/main/packages/prometheus_input/agent/

• [ ] RabbitMQ https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/connection https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/exchange https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/node https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/queue

• [ ] Redis https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/info https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/key https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/keyspace https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/slowlog

• [ ] Redis Enterprise https://github.com/elastic/integrations/blob/main/packages/redisenterprise/data_stream/node https://github.com/elastic/integrations/blob/main/packages/redisenterprise/data_stream/proxy

• [ ] SpringBoot https://github.com/elastic/integrations/blob/main/packages/spring_boot/data_stream/gc https://github.com/elastic/integrations/blob/main/packages/spring_boot/data_stream/memory https://github.com/elastic/integrations/blob/main/packages/spring_boot/data_stream/threading

• [ ] Traefik https://github.com/elastic/integrations/blob/main/packages/traefik/data_stream/health

• [ ] VSphere https://github.com/elastic/integrations/blob/main/packages/vsphere/data_stream/datastore https://github.com/elastic/integrations/blob/main/packages/vsphere/data_stream/host https://github.com/elastic/integrations/blob/main/packages/vsphere/data_stream/virtualmachine

• [ ] Zookeeper https://github.com/elastic/integrations/blob/main/packages/zookeeper/data_stream/connection https://github.com/elastic/integrations/blob/main/packages/zookeeper/data_stream/mntr https://github.com/elastic/integrations/blob/main/packages/zookeeper/data_stream/server

[ishleenk17]

@Linu-Elias :Lets organize it at package level. I have done that for some in above comment

[ali786XI]

As I was reviewing some of the PRs related to processors raised by third party contributors, I just want to list them so that we can possibly reduce some of them in our ones.

• https://github.com/elastic/integrations/pull/10068
• https://github.com/elastic/integrations/pull/10123

[shmsr]

@Linu-Elias Just curious. Are the changes done manually or you used any existing/new automation to do this change?

[Linu-Elias]

@shmsr I wrote a script to first fetch those packages that had missing processors and adding the processors by the same code, but updating changelog and manifest was done manually.

[shmsr]

@shmsr I wrote a script to first fetch those packages that had missing processors and adding the processors by the same code, but updating changelog and manifest was done manually.

Great! If the code is sufficient to perform the task on all packages safely and you believe it would be beneficial for others, you can open a pull request in this repository: https://github.com/elastic/obs-integrations-robots.

If not, that's okay too. However, it's great to have reliable automation tools that are generally useful, like this one: https://github.com/andrewkroh/go-examples/tree/main. We sometimes use tools created primarily by Andrew to execute extensive automation changes.

[ali786XI]

As we have reached the last PR on this should we include the vsphere package changes in the final PR or try to push the PR from the collaborator?

[ishleenk17]

As we have reached the last PR on this should we include the vsphere package changes in the final PR or try to push the PR from the collaborator?

The community PR has been blocked on CLA agreement, so I think it's ok to have vsphere as part of this meta issue. You can link this issue in community PR.