search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
123 stars 132 forks source link

Cache external binaries downloaded when building with `EXTERNAL=true` #2909

Open cmacknz opened 1 year ago

cmacknz commented 1 year ago

Today if you build the agent with EXTERNAL=true SNAPSHOT=true mage package the agent will download the latest snapshot version of the binaries in the following list each time you run the command:

https://github.com/elastic/elastic-agent/blob/7772cf2a42bd674a3347e04f9639d1b63ee9aabf/magefile.go#L795-L807

We should instead cache the snapshot artifacts for each artifact and snapshot build ID. Running the package command sequentially should not re-download the artifacts each time. We can use the snapshot versioning convention introduced in https://github.com/elastic/elastic-agent/pull/2752.

For example we should be able to build a cached directory tree that looks something like the following.

8.8.0-SNAPSHOT+49c28bdb/
  linux-x86_64/
  darwin-aarch64/
  windows-x86_64/

To prevent using an ever increasing amount of space we should expire versions based on either age or the total number of versions that are cached.

We should also be able to easily specify the snapshot version of the artifacts to use. It should be possible to build with something like EXTERNAL=true EXTERNAL_VERSION=8.8.0-SNAPSHOT+49c28bdb mage package and pin the dependencies to that particular version so that a new snapshot becoming available does not force you to redownload everything.

elasticmachine commented 1 year ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

gsantoro commented 1 year ago

Caching would definitively help but fetching in parallel is another optimization that could make things faster. Even better not even fetching beats that you don't need beats caching. 👍🏻 . more info at https://github.com/elastic/elastic-agent/issues/2426#issuecomment-1634566616