elasticmachine
commented
1 year ago Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
Open SpencerLN opened 1 year ago
elasticmachine
commented
1 year ago Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
andrewkroh
commented
1 year ago This package is using the dynamic ECS mappings (source). So the template has dynamic mappings for all ECS fields.
My assumption is that without static mappings in the template then Fleet is unaware of the ECS fields that it should add to the index.query.default_field. IIRC it was designed to take the first 1024 static fields in the template and add them to the default_field list.
So I think we need a way to populate a useful index.query.default_field list when using the dynamic ECS mappings.
For reference:
GET _component_template/logs-google_workspace.admin@package
P1llus
commented
1 year ago There is unfortunately no "quick fix" for this one. We are discussing the best approach here for the future, as in the end, all integrations will use the newer dynamic ECS template.
We can keep this issue open for now for tracking purposes, @felixbarny @ruflin, but the fix is not only for this integration, but rather for all that will be or are using the dynamic template currently.
ruflin
commented
1 year ago If I remember correctly, the reason that Fleet does not just set * for the default field is because otherwise users see errors on queries because foo query on ip field returns error. The list of fields added to the default_fields can be found here: https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.ts#L11
felixbarny
commented
1 year ago Seems like we're working around limitations in Elasticsearch here. Do we already have an Elasticserach issue to track that enhancement?
P1llus
commented
1 year ago I think that we should look into possibilities to make default_fields: * work properly @felixbarny, that would be an enhancement issue with elasticsearch.
felixbarny
commented
1 year ago There's quite a bit of history around default_field. This is what I've found out so far:
default_field to message, which is in line with what we still do for Elasticsearch's built-in logs-*-* index template.default_field from metric data streams (why would we want to offer full text search on metrics?). Instead, we've added the logic that still exists today which adds the first 1024 fields of a package to default_field in https://github.com/elastic/kibana/pull/91791.default_field to * "because it protects users from certain query exceptions" according to this comment). According to this comment this happens when there are more 1024 fields in an index. Not sure if there are other issues, too for example when there are non-text field types, such as ip.default_field: https://github.com/elastic/kibana/issues/128152 discussing whether to use * as a default now that the number of fields for integrations is lower due to the data stream naming scheme.Overall, I think there are two options to move forward
default_field to message for data streams where message is defined as a field.default_field to *.As 1) may be considered a breaking change, 2) seems more feasible.
@javanna could you chime in here? Would it be feasible to make default_field=* more lenient to avoid exceptions at search time? Or maybe we could add another way of expressing that the first 1024 fields should be searched by default if we're worried to change the semantics of *.
ruflin
commented
1 year ago What's a bit surprising to me is that we didn't just remove default_field from metric data streams (why would we want to offer full text search on metrics?).
Lets assume a users as tags: ["foo"] or host.name: bar in their metric document, if they put in foo or bar in the query bar, these docs will show up.
++ on Fix Elasticsearch query issues related to setting default_field to *. Only being able to query on message field by default without being more specific seems to be too limiting.
javanna
commented
1 year ago I am not entirely up to speed on the issues with setting default field to *. I would expect that to make ES expand to fields that can be queried only and not cause errors. That may slow things down though as it ends up querying a lot of fields, and potentially doc_value only fields too that are slower to query than fields that have an inverted index. Are we absolutely sure that we want to go down that route? Have we considered using a catch_all multi_field that includes all the fields that need querying and set that as a default field?
@jpountz do you have more history / opinions here?
ruflin
commented
1 year ago I did some testing and so far I have not found issues around IP addresses for setting default fields to *. This is good news, ideally we would have a test suite going through the different types to see if anything stands out. Being able to query all fields I consider to be one of the super powers of Elasticsearch. I think it is ok if things are slower in this scenario (if we should users progress). As usual, the challenge is that in advance, we don't know the fields that will come in, the shipper decides it and we most provide a good experience without requiring the user to do work up front.
@SpencerLN Could you try to use an @custom template to overwrite on your end the default_field setting with * to see if you get the expected result? You should also be able to overwrite it on query time. I'll play around with it on some of our clusters.
I tried the following for the IP address where I thought it had some issues in the past but now the results are as expected:
PUT _index_template/logs-foo
{
"index_patterns": [
"logs-foo-*"
],
"data_stream": {},
"priority": 500,
"template": {
"settings": {
"index.query.default_field": "*"
},
"mappings": {
"properties": {
"host.name": {
"type": "keyword"
},
"source.ip": {
"type": "ip"
}
}
}
}
}
POST logs-foo-bar/_doc
{
"@timestamp": "2023-09-07T15:04:05.000000001Z",
"host.name": "elastic.co",
"source.ip": "34.107.161.234"
}
GET logs-foo-bar/_search?q=elastic.co
GET logs-foo-bar/_search?q=34.107.161.234Thanks for the tests, this helps a lot. I believe that the 1024 limit no longer applies after https://github.com/elastic/elasticsearch/pull/81850 and there may have been other changes in the meantime that made support for default_field: * better. Please let us know what issues you encounter and we'll look deeper.
SpencerLN
commented
1 year ago @ruflin, I updated our existing google_workspace indices to use the wildcard, and it seems to work alright. After the change, I could search for an email address and IP successfully directly in the query bar in Discover without specifying the field name. I didn't notice any error messages, but I didn't do extensive testing.
PUT .ds-logs-google_workspace.*/_settings
{
"index": {
"query": {
"default_field": "*"
}
}
}This is great news @SpencerLN , did also some more tests on my on some larger clusters and could not find issues so far. I'll follow up soon with a bit more specific proposal and what we could do as next step.
ruflin
commented
1 year ago Here is a proposal on a potential path forward: https://docs.google.com/document/d/1EA6jeWM1VElGuQwEXzxDZ1hNmVYPCDIGLxU_zl0xeR4/edit @javanna @jpountz Would be great if you could a look at this from the Elasticsearch perspective. I think the combination of changes that happened over the past 12 months will allow us to have a simpler implementation that "just works". @andrewkroh @P1llus Please also have a look.
ruflin
commented
11 months ago After same discussion in the doc I know opened the following issue in the Elasticsearch repo with the proposal of changing the default to *: https://github.com/elastic/elasticsearch/issues/99872
The Google Workspace data streams are missing the ECS fields; this results in searches that a user would expect to work not returning the same results that they do for other integrations. Just a single example, the
source.user.emailfield is missing from theindex.query.default_fieldsetting in the Login stream, so if you search for a user's email in quotes, i.e."first.last@elastic.co"it will return no results, whilesource.user.email: "first.last@elastic.co"returns results.I didn't review all of the data streams in the integration, but at least several of them seem impacted.