Open cmacknz opened 1 year ago
Pinging @elastic/elastic-agent (Team:Elastic-Agent)
Caching would definitively help but fetching in parallel is another optimization that could make things faster. Even better not even fetching beats that you don't need beats caching. 👍🏻 . more info at https://github.com/elastic/elastic-agent/issues/2426#issuecomment-1634566616
Today if you build the agent with
EXTERNAL=true SNAPSHOT=true mage package
the agent will download the latest snapshot version of the binaries in the following list each time you run the command:https://github.com/elastic/elastic-agent/blob/7772cf2a42bd674a3347e04f9639d1b63ee9aabf/magefile.go#L795-L807
We should instead cache the snapshot artifacts for each artifact and snapshot build ID. Running the package command sequentially should not re-download the artifacts each time. We can use the snapshot versioning convention introduced in https://github.com/elastic/elastic-agent/pull/2752.
For example we should be able to build a cached directory tree that looks something like the following.
To prevent using an ever increasing amount of space we should expire versions based on either age or the total number of versions that are cached.
We should also be able to easily specify the snapshot version of the artifacts to use. It should be possible to build with something like
EXTERNAL=true EXTERNAL_VERSION=8.8.0-SNAPSHOT+49c28bdb mage package
and pin the dependencies to that particular version so that a new snapshot becoming available does not force you to redownload everything.