Dump all the integrations ES assets for benchmark rally command

[aspacca]

Currently the benchmark rally command simulate the index template of the target data stream, change the time_series.start_date/end_date values and dump it inline in the rally tracks.

This approach requires that the integrations is already installed in the cluster in order to replay the rally track with the proper setup: we'd miss ingest pipelines on the contrary.

For this reason we would like to dump instead al the ES assets (including ingest pipelines) when executing the command, and embed them inline in the generated rally track,

[ruflin]

embed them inline in the generated rally track,

Can you share a bit more details on "inline" vs put them into a file and reference it? Pros and cons are not clear to me.

[aspacca]

Can you share a bit more details on "inline" vs put them into a file and reference it? Pros and cons are not clear to me.

in esrally, for templates, it is possible to add an element referencing a file with the template body, or, with a different element, add the template body inline directly in the rally track. the pros of having the content referenced is that the track is leaner, while at the same time the cons is that you have to carry over multiple files if you want to replay the track (at the moment this is true only for the corpus). moreover pipelines can be added only with an inline element, so we would need to use two different approaches for templates and pipelines, if we decide to reference the first, that could be confusing, and we won't end up with a lean track anyway.

if we put everything inline, but for the corpus, the track would be "self-sufficient"

[ruflin]

moreover pipelines can be added online with an inline element,

What does this exactly mean? pipelines can only be put inline and not referenced? It is odd that rally supports one for reference but not the other.

If pipeline only can be inline, agree lets do both inline. My assumption is the effort to potentially change this in the future would not fundamentally change things.

[aspacca]

What does this exactly mean? pipelines can only be put inline and not referenced? It is odd that rally supports one for reference but not the other.

it was a typo :) "moreover pipelines can be added only with an inline element"

It is odd that rally supports one for reference but not the other.

https://esrally.readthedocs.io/en/stable/track.html#composable-templates https://esrally.readthedocs.io/en/stable/track.html#create-composable-template

vs

https://esrally.readthedocs.io/en/stable/track.html#put-pipeline

I could not find the equivalent of composable-templates element for pipelines

[ruflin]

Lets stick to inline for now. I suggest to open an issue in rally for the future to track potentially support for "non inline pipelines".