[Fleet]: Ubuntu 22 agent goes unhealthy on installing Elastic Security integration to Agent Policy.

[harshitgupta-qasource]

Kibana Build details:

VERSION: 7.17.22
BUILD 47585
COMMIT 43696930d77d3bb567e445624874eab9cf363872

Host OS and Browser version: [Ubuntu 22] , All

Preconditions:

1. 7.17.22 BC1 Kibana Cloud environment should be available.
2. 7.17.22 Ubuntu agent should be installed with Elastic Security integration.

Steps to reproduce:

1. Navigate to the Agents Tab
2. Wait for a while till the agent becomes unhealthy.
3. Go to the Endpoint Tab
4. Observe that the Ubuntu agent goes to unhealthy.

Expected:

• Ubuntu Agent should remain Healthy throughout when installed with Endpoint security.

Screenshot:

Note: Reproducible on Ubuntu agents only.

Agents Logs: elastic-agent-diagnostics-2024-06-07T04-56-45Z-00.zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[amolnater-qasource]

Secondary Review for this ticket is Done.

[cmacknz]

{"log.level":"warn","@timestamp":"2024-06-07T04:49:10.601Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'degraded'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-07T04:49:10.601Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2024-06-07T04:49:10Z - message: Application: endpoint-security--7.17.22[07f90b01-8921-47e5-9c49-48140492df28]: State changed to DEGRADED: Protecting with policy {55ab3dc2-1356-4eb0-8341-cad5745a39bd} - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-07T04:53:43.041Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-07T04:53:43.041Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2024-06-07T04:53:43Z - message: Application: endpoint-security--7.17.22[07f90b01-8921-47e5-9c49-48140492df28]: State changed to CONFIG: Protecting with policy {55ab3dc2-1356-4eb0-8341-cad5745a39bd} - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-06-07T04:54:03.042Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'degraded'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-07T04:54:03.043Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2024-06-07T04:54:03Z - message: Application: endpoint-security--7.17.22[07f90b01-8921-47e5-9c49-48140492df28]: State changed to DEGRADED: Protecting with policy {55ab3dc2-1356-4eb0-8341-cad5745a39bd} - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}

These are the only logs in the diagnostics, there are no endpoint logs.

Are you able to get the endpoint logs from the affected machine? They should be in /opt/Elastic/Endpoint.

[nicholasberlin]

What's the kernel version?

[nicholasberlin]

@nick-alayil According to the support matrix, Elastic Defend v7.17 is not supported on Ubuntu 22.04. Assuming this problem is related to a new Ubuntu 22.04 kernel, should we backport fixes to support it?

[nick-alayil]

should we backport fixes to support it?

No, we don't need to backport fixes for this. Additionally, agent also does not support Ubuntu 22.04 as per support matrix

[harshitgupta-qasource]

Hi Team,

We have tested this issue on Ubuntu 20 and found it working fine.

Observations:

• Ubuntu 20 agent remains Healthy with Endpoint Security.

Build details:

VERSION: 7.17.22
BUILD 47585
COMMIT 43696930d77d3bb567e445624874eab9cf363872

Further, we have reproduced this issue on Ubuntu 22, please find the attached enpoint logs file: endpoint-000000.log

Kindly let us know if anything is else required from our end. Thanks

[nicholasberlin]

These are the pertinent log lines

"2024-06-10T05:58:10.289401227Z PerfWatcher.cpp:93 Failed to write: (r:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe tcp_sendpage rv=$retval)"
"2024-06-10T05:58:10.28946858Z PerfWatcher.cpp:93 Failed to write: (-:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe)"

The tcp_sendpage symbol does not exist in the newer kernel, which is causing Endpoint to fail to install network event sources. This is known and fixed in the 8.* series. But, since we do not support v7.17 series on Ubuntu 22.04, there will not be any backports.

I think the only requirement on your end would be to stop formal testing of 7.17 on Ubuntu 22.04.