[Horde]: 10k agents version 8.5.0 fail to upgrade/schedule upgrade on 8.6.

[amolnater-qasource]

Kibana version: 8.6 BC5 kibana cloud environment

Host OS and Browser version: Horde

Build details:

VERSION: 8.6.0 BC5 Kibana cloud environment
BUILD: 58693
COMMIT: ed40c16ce9999cc47ad55c11bb097d2e443b31a6

Preconditions:

1. 8.6 BC5 kibana cloud environment should be available.

Steps to reproduce:

1. Navigate to Fleet>Settings tab.
2. Add Output: https://staging.elastic.co/8.6.0-8cf9e954/downloads/
3. Navigate to Agents tab and select 10k agents.
4. Set schedule upgrade time and observe on schedule time, agents failed to upgrade and remained v8.5.0.

Expected Result: 10k agents version 8.5.0 should upgrade successfully when upgraded/scheduled upgrade on 8.6.

Screen Recording:

https://user-images.githubusercontent.com/77374876/206126406-639df708-7717-4bc1-a659-af0c4ccebacc.mp4

https://user-images.githubusercontent.com/77374876/206126508-909f2752-e799-4d0c-8408-437dfa33e5c4.mp4

[amolnater-qasource]

@manishgupta-qasource Please review.

[jlind23]

@amolnater-qasource would you please provide the logs?

[amolnater-qasource]

Hi @jlind23 Thank you for looking into this. We have run this test on Horde based 10k agents, for which we don't have any logs for agents.

However, logs from Horde CLI are attached below: Horde.txt

ehswarm_1    | {"level":"info","action_type":"UPGRADE","action_id":"1d5c916b-eb9e-428b-b777-0dcb1f595b27","agent_id":"fa593fe7-4984-4aa6-a186-73ade411b2a2","cur_version":"8.5.0","tgt_version":"8.6.0","Gen":0,"Id":"QiDFs2zEvWDPdP96XUvzMj","Idx":18,"droneId":"imb6E3YTAGbwiKW8YhrDHS","droneName":"eh-Dancergossamer-imb6","error":"HTTP Fail","rtt":88.534245,"time":1670406671772,"message":"Upgrade download done"}
ehswarm_1    | {"level":"error","action_type":"UPGRADE","action_id":"1d5c916b-eb9e-428b-b777-0dcb1f595b27","agent_id":"fa593fe7-4984-4aa6-a186-73ade411b2a2","cur_version":"8.5.0","error":"HTTP Fail","Gen":0,"Id":"QiDFs2zEvWDPdP96XUvzMj","Idx":18,"droneId":"imb6E3YTAGbwiKW8YhrDHS","droneName":"eh-Dancergossamer-imb6","time":1670406671773,"message":"could not upgrade the drone: HTTP Fail"}
ehswarm_1    | {"level":"debug","sz":259,"name":"drone.artifact.queue","time":1670406671774,"message":"Publish event"}

Further, we are successfully able to upgrade real fleet agents.

Please let us know if anything else is required from our end. Thanks

[manishgupta-qasource]

Secondary review for this ticket is Done

[jlind23]

@joshdover looking at "message":"could not upgrade the drone: HTTP Fail"}

Is this a drone behaviour you have already seen?

[joshdover]

This is expected behavior before 8.6 is released. Horde drones don't have the logic to try to pull from the staging or snapshot endpoints. When using horde, the version you upgrade from and to isn't really relevant because horde doesn't do anything real other than download the file. We can simply test upgrading horde drones from 8.5.0 to 8.5.1 to verify that Fleet Server 8.6 and the UI are behaving correctly.

[jlind23]

Thanks @joshdover for your answer, thus closing this issue as won't fix.

[amolnater-qasource]

Hi Team, Thank you for the update.

We will be considering above shared information, while testing agent upgrades using Horde.

Thanks!