upgrade using uri from fleet api isn't working

[pjbertels]

I can't seem to do an upgrade in Fleet using the Fleet API when I want to specify the URI. Moreover if this is not supported the logs don't indicate that adequately.

8.7 SNAPSHOT, linux VM running elastic-agent 8.5.3

Here is the link for the API: https://petstore.swagger.io/?url=https://raw.githubusercontent.com/elastic/kibana/master/x-pack/plugins/fleet/common/openapi/bundled.json#/default/bulk-upgrade-agents

This is what I am doing ...

[16:53:32] INFO     Upgrading all agents at 8.5.3 to 8.6.0 https://snapshots.elastic.co/8.6.0-08909b84/downloads/beats/elastic-agent            test_bulk_upgrade.py:31
{'version': '8.6.0', 'source_uri': 'https://snapshots.elastic.co/8.6.0-08909b84/downloads/beats/elastic-agent', 'rollout_duration_seconds': 600, 'agents': 'local_metadata.elastic.agent.version : 8.5.3 and local_metadata.elastic.agent.upgradeable : true'}
<Response [200 OK]>
FleetActionId(actionId='2d4cc156-0636-438c-88cd-db015742769f')

I have tried using 8.6 as the version and leaving it empty, setting the URI never seems to result in a log that says it will try that URI. I have turned on debug logs in elastic-agent. I have tried different flavors of URI. I have verified that I can wget the file I want from the URI.

https://snapshots.elastic.co/8.6.0-08909b84/downloads/beats/elastic-agent/elastic-agent-8.6.0-SNAPSHOT-linux-x86_64.tar.gz

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[cmacknz]

In 8.6 we added a new binary to the Elastic agent package called the elastic-agent-shipper that shows up in the artifact API response, and triggers a bug in the agent where it doesn't expect anything other than the elastic-agent .tar.gz package to be there. This is fixed in 8.6 and above: https://github.com/elastic/elastic-agent/pull/1791/.

This only affects upgrades from pre-8.6 versions to 8.6 snapshot versions, which is a different code path in the agent from upgrading to official releases. The work around would be do a non-snapshot upgrade first, either by waiting for the soon to be available official 8.6 release or trying to upgrade to one of the 8.6 build candidates (may have the same problem I haven't tested it recently).

[pjbertels]

I moved to 8.6.0 and I can upgrade to 8.7.0 but I can't get seem to get the URI to be used so I can upgrade to an arbitrary snapshot.

[cmacknz]

Do you have agent-diagnostics or logs from the failure we could look at for the snapshot failures?

[pjbertels]

We are working on updating OGC to do regular real agent testing we can revisit this issue and get debug info as part of testing that.