Investigate additional testcases for APM on ECK

[simitt]

The ECK team added some basic automated tests for the APM integration. Follow up with the ECK team on additional testcases that would make sense for APM.

cc @pebrc

[simitt]

Sharing some ideas for additional testcases:

• configuration tests focus on overwriting output.elasticsearch settings. Adding a config option for Kibana, e.g. apm-server.kibana.host and one for general APM Server configuration e.g. apm-server.rum.enabled should be relatively straight forward and seems a good idea, as the ES output is treated special afaik.
• Could you test some actual ingestion? A sample payload can be downloaded from testdata/intake-v2/events.ndjson and could be sent via curl cmd (curl -H "Content-type: application/x-ndjson" --data-binary @testdata/intake-v2/events.ndjson <apm-server-address:port>/intake/v2/events). Ingested events should then be available in apm* indices. This would fit into the sample_test.
• not sure running setup commands is supported, I think we discussed this could be done via init container. If supported a test could run apm-server setup and check apm templates, ILM aliases and policy are there.

Some questions came up when checking current behavior:

• Why is apm-server.ilm.enabled: false in some of the tests? Is that some arbitrarily chosen config option to test config changes or are there any issues with ILM?

• TLS is enabled by default for APM Server as previously agreed on. Is there any documentation around for how to extract the cert information so the agents can pin them?

[pebrc]

Why is apm-server.ilm.enabled: false in some of the tests?

I think we ran into some test failures because ILM was enabled. Unfortunately the issue is a bit light on details https://github.com/elastic/cloud-on-k8s/issues/1550

TLS is enabled by default for APM Server as previously agreed on. Is there any documentation around for how to extract the cert information so the agents can pin them?

https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-apm-server.html#k8s-apm-tls

and

https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-accessing-elastic-services.html#k8s-tls-certificates

are relevant here I think. Especially the latter as it is the same principle for all stack products on ECK at the moment.

[graphaelli]

• actual ingested is now tested in CheckEventsAPI.

• 2298 is up to test another config setting, apm-server.rum.enabled

I haven't looked into running one off commands or enabling ILM, will leave those for a follow up.