[APM][User Experience] APM telemetry triggers deprecation logs

[dominiqueclarke]

Kibana version: 7.16.1

Elasticsearch version: 7.16.1

Original install method (e.g. download page, yum, from source, etc.): apm-integration-testing script, plus Kibana source

Description of the problem including expected versus actual behavior:

The following deprecation logs are reported from APM telemetry

server    log   [12:45:55.957] [info][deprecation][elasticsearch] Elasticsearch deprecation: 299 Elasticsearch-7.16.1-5b38441b16b1ebb16a27c107a4c3865776e20c53 "this request accesses system indices: [.async-search, .security-7, .tasks, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default"
Origin:kibana
Query:
200
POST /_search
{"size":0,"timeout":"5m","query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"now-1d"}}}]}},"aggs":{"transaction.name":{"cardinality":{"field":"transaction.name"}},"user_agent.original":{"cardinality":{"field":"user_agent.original"}}}}
server    log   [12:45:55.957] [info][deprecation][elasticsearch] Elasticsearch deprecation: 299 Elasticsearch-7.16.1-5b38441b16b1ebb16a27c107a4c3865776e20c53 "this request accesses system indices: [.async-search, .security-7, .tasks, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default"

server    log   [12:45:55.838] [info][deprecation][elasticsearch] Elasticsearch deprecation: 299 Elasticsearch-7.16.1-5b38441b16b1ebb16a27c107a4c3865776e20c53 "this request accesses system indices: [.async-search, .security-7, .tasks, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default"
Origin:kibana
Query:
200
POST /_search
{"size":0,"timeout":"5m","query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"now-1d"}}},{"terms":{"agent.name":["js-base","rum-js","opentelemetry/webjs"]}}]}},"aggs":{"client.geo.country_iso_code":{"cardinality":{"field":"client.geo.country_iso_code"}},"transaction.name":{"cardinality":{"field":"transaction.name"}},"user_agent.original":{"cardinality":{"field":"user_agent.original"}}}}

It appears to be triggered from the query found here: https://github.com/elastic/kibana/blob/main/x-pack/plugins/apm/server/lib/apm_telemetry/collect_data_telemetry/tasks.ts

Steps to reproduce:

1. Using apm-integration-testing repo, run the below command (I run Kibana separately to better monitor Kibana logs)

   ./scripts/compose.py start \
   --release \
   --with-opbeans-node \
   7.16.1 --no-kibana

2. Update kibana.dev.yml with the below config

   elasticsearch:
   hosts: ["http://localhost:9200"]
   username: "admin"
   password: "changeme"

3. Update apm.dev.js with the following

   module.exports = {
   active: true,
   serverUrl: 'http://127.0.0.1:8200',
   centralConfig: false,
   breakdownMetrics: false,
   transactionSampleRate: 0.1,
   metricsInterval: '120s',
   contextPropagationOnly: false
   };

   4. Monitor Kibana logs while navigating to APM or User Experience. Notice the deprecation log.

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)