Closed achuguy closed 1 week ago
@achuguy Under composable-logsdb
there are copies of some files that are specific for the logsdb
benchmark. Can we make sure the changes are applied to those files too? Thank you.
Question: does that logic generating the dataset needs adjustment?
Question: does that logic generating the dataset needs adjustment?
For generating a dataset I'll take endpoint data from a cluster with real endpoints. There is room to adjust fields in the dataset depending on cardinality needed.
@achuguy Under
composable-logsdb
there are copies of some files that are specific for thelogsdb
benchmark. Can we make sure the changes are applied to those files too? Thank you.
@salvatore-campagna When I'm trying to run the security-track with track-param "index_mode": "logsdb"
I get an error trying to create the index template:
Create composable index template
PUT _index_template/logs-endpoint.events-process-test
{
"index_patterns": [
"logs-endpoint.events.process-*"
],
"template": {
"settings": {},
"mappings": {
"_meta": {
"package": {
"name": "endpoint"
},
"managed_by": "fleet",
"managed": true
}
}
},
"composed_of": [
"logs@mappings",
"logs@settings",
"logs-endpoint.events.process@package",
"ecs@mappings",
".fleet_globals-1",
".fleet_agent_id_verification-1",
"track-shared-logsdb-mode"
],
"priority": 200,
"_meta": {
"package": {
"name": "endpoint"
},
"managed_by": "fleet",
"managed": true
},
"data_stream": {
"hidden": false,
"allow_custom_routing": false,
"failure_store": false
}
}
Error
{
"error": {
"root_cause": [
{
"type": "x_content_parse_exception",
"reason": "[37:7] [data_stream_template] unknown field [failure_store]"
}
],
"type": "x_content_parse_exception",
"reason": "[37:24] [index_template] failed to parse field [data_stream]",
"caused_by": {
"type": "x_content_parse_exception",
"reason": "[37:7] [data_stream_template] unknown field [failure_store]"
}
},
"status": 400
}
Is the failure_store
parameter correct under data_stream
?
@achuguy Under
composable-logsdb
there are copies of some files that are specific for thelogsdb
benchmark. Can we make sure the changes are applied to those files too? Thank you.@salvatore-campagna When I'm trying to run the security-track with track-param
"index_mode": "logsdb"
I get an error trying to create the index template:Create composable index template
PUT _index_template/logs-endpoint.events-process-test { "index_patterns": [ "logs-endpoint.events.process-*" ], "template": { "settings": {}, "mappings": { "_meta": { "package": { "name": "endpoint" }, "managed_by": "fleet", "managed": true } } }, "composed_of": [ "logs@mappings", "logs@settings", "logs-endpoint.events.process@package", "ecs@mappings", ".fleet_globals-1", ".fleet_agent_id_verification-1", "track-shared-logsdb-mode" ], "priority": 200, "_meta": { "package": { "name": "endpoint" }, "managed_by": "fleet", "managed": true }, "data_stream": { "hidden": false, "allow_custom_routing": false, "failure_store": false } }
Error
{ "error": { "root_cause": [ { "type": "x_content_parse_exception", "reason": "[37:7] [data_stream_template] unknown field [failure_store]" } ], "type": "x_content_parse_exception", "reason": "[37:24] [index_template] failed to parse field [data_stream]", "caused_by": { "type": "x_content_parse_exception", "reason": "[37:7] [data_stream_template] unknown field [failure_store]" } }, "status": 400 }
Is the
failure_store
parameter correct underdata_stream
?
If you look at the copies under composable-logsdb
there is a Jinja annotation that is selectively including the line
"data_stream": {
"hidden": false,
"allow_custom_routing": false
{% if build_flavor != "serverless" %},
"failure_store": false
{% endif %}
}
based on build_flavor
. So, it is correct depending on build_flavor
. When updating files please make sure that Jinja logic is preserved.
Looks like failure_store
is in the 8.16.0-SNAPSHOT and not in 8.15.x so I've adjusted my testing.
LGTM after applying changes commented by @gbanasiak
@achuguy I've added https://github.com/elastic/rally-tracks/pull/675/commits/a92162e6bf7a84b25dd2a1902009dd7ecf5b8b0c to avoid collision between packages required by Rally, and packages installed by the track.
@salvatore-campagna
Looks like
failure_store
is in the 8.16.0-SNAPSHOT and not in 8.15.x so I've adjusted my testing.
We can't backport it to 8.15
with failure_store
, this will need adjustment.
To be clear, this is not a 8.16 vs 8.15 thing. failure_store
is only enabled in SNAPSHOT builds and should go away completely at some point.
Now with logs-endpoint
composable templates being up-to-date we can simplify further by removing composable-logsdb
directory. I'm also decoupling index mode setting from exclusions for benchmark scenarios with Kibana by introducing additional logs_endpoint_from_kibana
track parameter (otherwise it's not possible to run in LogsDB mode standalone, without Kibana). See https://github.com/elastic/rally-tracks/pull/675/commits/9e1089b75e5f334aa5fe3184425851a9da10d7c0.
https://github.com/elastic/rally-tracks/pull/675/commits/2bafd52864eb86d6ba75539a55a24e9b143f82eb moves the track closer to being fully supported in serverless (both standard user and operator). What remains to be done on that front (outside of this PR) is similar adjustment of Beats composable templates and addition of serverless IT tests (here).
To be clear, this is not a 8.16 vs 8.15 thing.
failure_store
is only enabled in SNAPSHOT builds and should go away completely at some point.
Thi si because it is using a feature flag right? That at some point will be removed.
Thanks @gbanasiak and @achuguy for fixing this!
Status | Branch | Result |
---|---|---|
❌ | 8.15 | Backport failed because of merge conflicts |
To create the backport manually run:
backport --pr 675
Please refer to the Backport tool documentation and see the Github Action logs for details
@achuguy Can I ask for a manual backport to 8.15
?
@gbanasiak Yes, I'll work on the backport
Update the artifacts, composable templates, component templates, pipelines, ilm, used in the
elastic/security
track to Elasticsearch 8.15.1. Also deleting unused artifacts from the repo.