[ci.jenkins.io] collect datadog metrics for ephemeral VMs

[dduportal]

Service(s)

ci.jenkins.io

Summary

Part of high level topic https://github.com/jenkins-infra/helpdesk/issues/2769.

This issue is scoped to only ephemeral VM agents.

It requires the following tasks to be accomplished:

• Install datadog tooling in the VM templates - https://github.com/jenkins-infra/packer-images
• Configure ci.jenkins.io to enable datadog agent collection when booting the VMs - https://github.com/jenkins-infra/jenkins-infra/blob/9111da0d831730f7b7ac2bac146d1d7214f189e2/dist/profile/templates/jenkinscontroller/casc/clouds.yaml.erb
  • Need a datadog API key, stored as a system-level credential (not available for pipelines), injected into the agent startup process (cloud-init)?
• Validate that data is collected in datadog - https://jenkins.datadoghq.com/apm/home
• Make the datadog data available for developers - TBD

[dduportal]

Installation of the datadog agent on templates: https://github.com/jenkins-infra/packer-images/pull/318

[timja]

Need a datadog API key, stored as a system-level credential (not available for pipelines), injected into the agent startup process (cloud-init)?

Might be easier to inject at build time with packer? (needs to be in something like opt as home directories get recreated with new users and tmp isn't safe from tmp sweepers)

[dduportal]

Need a datadog API key, stored as a system-level credential (not available for pipelines), injected into the agent startup process (cloud-init)?

Might be easier to inject at build time with packer? (needs to be in something like opt as home directories get recreated with new users and tmp isn't safe from tmp sweepers)

We thought about this, but with the upcoming "packerization" for the Docker images, that mean the packer templates would be available publicly with the API key stored within (today, the VM templates are private in both AWS or Azure so less an issue).

Hence the "add the key as late as posible" in the process.

[timja]

I don't think there's a way with the Azure vm agents plugin to add the key secretly currently.

[dduportal]

I don't think there's a way with the Azure vm agents plugin to add the key secretly currently.

I was thinking about using the "VM First Startup Configuration"'s Init script. But I assume the challenge is to allow passing the credential as a variable in this script?

[timja]

Yes, there’s an open issue for it on the plugin to allow that, apart from that you can put it in plain text in the script or somewhere the agent can retrieve it from

[dduportal]

Yes, there’s an open issue for it on the plugin to allow that, apart from that you can put it in plain text in the script or somewhere the agent can retrieve it from

For both ec2 and azure vm, it looks ok to put the api key in the script:

• source is jcasc (so the credential is encrypted until jcasc persists configuration in the jenkins home)
• The script would delete itself and is owned by root

Wdyt?

[timja]

Should be fine for datadog, although not ideal

[smerle33]

WIP on azure for now : https://github.com/jenkins-infra/jenkins-infra/pull/2347

[smerle33]

ec2 VM fail for now with :

Cloud-init v. 22.2-0ubuntu1~20.04.3 running 'modules:config' at Mon, 05 Sep 2022 09:21:26 +0000. Up 40.47 seconds.
Job for datadog-agent.service failed because the control process exited with error code.
See "systemctl status datadog-agent.service" and "journalctl -xe" for details.
Cloud-init v. 22.2-0ubuntu1~20.04.3 running 'modules:final' at Mon, 05 Sep 2022 09:21:28 +0000. Up 42.66 seconds.
Cloud-init v. 22.2-0ubuntu1~20.04.3 finished at Mon, 05 Sep 2022 09:21:29 +0000. Datasource DataSourceEc2Local.  Up 43.16 seconds

investigating ...

[smerle33]

while starting manually the datadog agent on an ephemeral ec2 VM, it triggered an alert on diskspace. We will have to specify new limit for those agent not to get spammed with false alert.

[dduportal]

No action required for the alert about the disk: it make sense to alert if the hard drive is full. The default agents have ~100 Gb free so all is good :)

[dduportal]

Status:

• Linux Ok (all cloud providers)

ToDo:

• Dashboard (first private, then public)
• Windows VMs

[dduportal]

Wip on https://github.com/jenkins-infra/helpdesk/issues/3149

[smerle33]

The datadog dashboard may not be the solution, a public dashboard cannot provide selectors on top (template variable) they are grayed in public view (thanks Michael Pailloncy) and the straight solution to "just" get the jenkins agent would be to tag them within the datadog.yaml file not really ideal ... I will get in touch with @jtnord to see if he could deal with just a call for info from us or if he can help by any ways to solve this ....

[dduportal]

Closing as we now have metrics from all ephemeral agents of ci.jenkins.io flowing in datadog.