[KQL] Add support for case-insensitive searches

[joswr1ght]

Describe the feature:

Requesting the implementation of an UPPER() function in KQL to be applied to fields, converting the record content to all uppercase.

UPPER(url.original):*SELECT*

KQL currently lacks a mechanism to perform case-insensitive searches. When using Kibana for system log analysis, attackers can evade detection by mixing case in data that would show up in web logs. For example, a search to identify a SQL injection attack utilizing the SQL UNION keyword might be:

url.original:*UNION*

An attacker who constructs an attack where the HTTP request uses the SQL keyword UnioN would evade detection with this KQL query.

Describe a specific use case for the feature:

• Ease of use: Analysts can construct KQL queries without knowing the case expectation by converting content to uppercase as desired.
• Log analysis: Analysts can search for keywords to identify evidence of an attack in system logs while mitigating the opportunity for an adversary to evade detection through mixed case use.

Thank you for your consideration.

-Josh

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[Bargs]

This depends more on how your field was analyzed at ingestion time. If your field is configured to use the Lowercase token filter then your KQL queries on that field will be case insensitive.

[joswr1ght]

The use case I’m looking for is following the ingestion of Linux auth.log or access.log files using the Apache or Nginx modules with Filebeat. I’m not aware of a Filebeat configure mechanism to change the status of case-sensitivity during import, though I could be mistaken here.

Thanks!

[joswr1ght]

Following up on this, I don't believe the Filebeat module for Apache/Nginx/IIS web log ingestion offers an option to use the Lowercast token filter. The ability to query data with a KQL UPPER() function would be a useful addition, allowing analysts some post-import flexibility in how the data is evaluated.

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[joswr1ght]

Just my semi-regular follow-up on this request. Thank you!

[joswr1ght]

Hey, just checking in again! You guys do great work, and this would be a fantastic feature to add to Kibana for log analysis. Thank you! 🙏

[joswr1ght]

Just checking in, any status on getting case-insensitive KQL match functionality?

[wylieconlon]

@joswr1ght There's been no direct work on this, but in the future release of 7.10, Elasticsearch will offer a case-insensitive option for wildcard searches. I'm proposing that we change KQL to use this by default in this issue, if you want to follow that discussion.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[willemdh]

When will we be able to query case insensitive in Kibana KQL? I also see https://github.com/elastic/kibana/issues/134143 This seems like very basic search functionality?

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.

This works with ES|QL btw: https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-to_upper