elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.64k stars 8.22k forks source link

[Security GenAI] AI Assistant lower quality of ES|QL generation results with Bedrock and Gemini #189771

Closed peluja1012 closed 1 month ago

peluja1012 commented 3 months ago

Describe the bug: When using the AI Assistant with Bedrock and Gemini models, and asking the Assistant to generate ES|QL queries from natural language, the Assistant results incorrect or results. Examining the traces, there are two main issues:

  1. Gemini is not calling the ES|QL tool at all (trace)

    ff_on_gemini

  2. Sonnet 3.5 ES|QL tool output appears to be ignored (trace)

    ff_on_sonnet_3_5

Kibana/Elasticsearch Stack version: 8.15 BC5

Steps to reproduce:

  1. Open the AI Assistant
  2. Enabled the Knowledge Base
  3. Select a Bedrock connector or Gemini connector
  4. Ask the following question:

    Generate an ES|QL query that will count the number of connections made to external IP addresses, broken down by user. If the count is greater than 100 for a specific user, add a new field called "follow_up" that contains a value of "true", otherwise, it should contain "false". The user names should also be enriched with their respective group names.

  5. Notice that the output contains incorrect ES|QL syntax.
elasticmachine commented 3 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

peluja1012 commented 3 months ago

Bedrock ES|QL regeneration was improved by this PR: https://github.com/elastic/kibana/pull/189810. Gemini ES|QL generation was partially improved. With Gemini, ES|QL generation works only when the system prompt is cleared.

stephmilovic commented 3 months ago

I'm not able to reproduce the Bedrock issue: https://smith.langchain.com/public/258c1ef6-25ec-4856-a043-07e41ea6368c/r

I took a look at the linked PR: https://github.com/elastic/kibana/pull/189810. However, it does not make sense that it would improve bedrock because all it does is change the Gemini prompt?

jamesspi commented 2 months ago

As discussed in our team meeting on Aug 29th 2024, Gemini ES|QL generation is still flaky if tasked with generating a query as part of a wider conversation.

Below are prompts in a conversation using GPT-4o vs Sonnet 3.5 vs Gemini 1.5 Pro:

GPT-4o

CONTEXT:
"""
## Possible Initial Compromise and Data Exfiltration

This incident involves a possible compromise of `srv-win-s1-rsa` by `james_spiteri`.

### Summary
User `james_spiteri` on `srv-win-s1-rsa` opened a malicious Excel document that led to the execution of malicious code, a port scan, and potential data exfiltration to a Slack channel. The attack progressed to compromise a Linux host, `srv-nix-es-rsa`, where suspicious scripts were downloaded and executed, culminating in the potential exfiltration of data via Slack.

### Details
- At `2024-05-03T06:12:51.723Z`, user `james_spiteri` opened a malicious Excel document: {{ file.name Upcoming Events February 2018.xls }} on `srv-win-s1-rsa`.
- This triggered a low severity alert for suspicious registry key creation (`Evasion`) and another for loading a VBE DLL, which suggests macro execution (`Evasion`).  
- Moments later, a new file, `K7Y4P6H1.exe`, was created via the `certutil.exe` process, indicating the Excel macro likely decoded and dropped this file. 
- This triggered a low severity alert for abnormal document behavior (`Exploitation`) on `srv-win-s1-rsa`.
- At `2024-05-03T06:19:44.992Z`, the file `K7Y4P6H1.exe` was detected as malicious by SentinelOne and triggered a critical alert on `srv-win-s1-rsa`.
- At `2024-05-03T06:35:53.486Z`, user `james_spiteri` initiated a port scan using `nmap.exe` targeting `10.128.0.104` from `srv-win-s1-rsa`. This triggered a high severity alert for a port scan from an internal host. 
- This activity aligns with the Tactic Reconnaissance (`TA0043`) and Techniques Active Scanning (`T1595`) and Gather Victim Network Information (`T1590`).  The specific sub-techniques used were Scanning IP Blocks (`T1595.001`) and IP Addresses (`T1590.005`). 
- At `2024-05-03T06:36:20.166Z`, on `srv-nix-es-rsa`, a suspicious process chain was initiated. The `java` process spawned a `sh` process. 
- This new process downloaded two files: `pwn.sh` and `headerfile.txt` from `https://images.swiftcrypto.com`, made `pwn.sh` executable, and then executed it. 
- This activity aligns with the Tactic Execution (`TA0002`) and the Technique Command and Scripting Interpreter (`T1059`). The sub-technique used was JavaScript (`T1059.007`). 
- Subsequently, at `2024-05-03T06:38:04.751Z`, the `curl` process was used to upload a file, `l00t.zip`, to a Slack channel using the uploaded `headerfile.txt` for authentication. This suggests potential data exfiltration. 
- This activity aligns with the Tactics Command and Control (`TA0011`) and Exfiltration (`TA0010`) and the Techniques Web Service (`T1102`), Exfiltration Over C2 Channel (`T1041`), and Exfiltration Over Web Service (`T1567`).

### Attack Chain
- Reconnaissance
- Initial Access
- Execution
- Persistence
- Privilege Escalation
- Discovery
- Lateral Movement
- Command & Control
- Exfiltration

"""

How would I remediate this incident according to our internal policies, and who should I contact?

Response:

To remediate this incident according to our internal policies, follow these steps:

1. **Containment:**
   - Isolate the compromised hosts `srv-win-s1-rsa` and `srv-nix-es-rsa` from the network to prevent further malicious activity.
   - Disable the user account `james_spiteri` to prevent further unauthorized access.

2. **Eradication:**
   - Remove the malicious files `K7Y4P6H1.exe`, `pwn.sh`, and `headerfile.txt` from the affected systems.
   - Uninstall any unauthorized software and terminate any suspicious processes.
   - Apply patches and updates to the affected systems to close any vulnerabilities.

3. **Recovery:**
   - Restore the affected systems from clean backups.
   - Re-enable the user account `james_spiteri` only after ensuring the account is secure and the user has been educated on safe practices.

4. **Post-Incident Activity:**
   - Conduct a thorough investigation to understand the full scope of the incident.
   - Review and update security policies and procedures to prevent future incidents.
   - Monitor the network and systems for any signs of residual malicious activity.

For more information, refer to our internal wiki at [https://ourwiki.com/security-response](https://ourwiki.com/security-response).

**Contact Information:**
- **Windows Hosts:** Jimmy Beans is responsible for all Windows hosts. You can slack him (@jimmyb), or open a GitHub issue here - [https://github.com/jamesspi/assistant-demos/issues](https://github.com/jamesspi/assistant-demos/issues).

Follow up prompt:

Help me generate an ES|QL query to look for this user's behaviour in logs-*. Count by event.category, and always use "" for field values.

Response:

Based on the provided context, you can use the following ES|QL query to count the number of events by category for the user "james_spiteri":

FROM logs-*
| WHERE user.name == "james_spiteri"
| STATS count() BY event.category

This query will filter the logs to only include those where the `user.name` is "james_spiteri" and then count the number of events for each `event.category`.

Additional Follow up prompt:

Can you update that query? If any of the data is network traffic, and the destination ip address is external, create a new field to reflect this.

Response:

Here is the updated ES|QL query to count the number of events by category for the user "james_spiteri" and categorize the destination IP addresses as either "external" or "internal":

FROM logs-*
| WHERE user.name == "james_spiteri"
| EVAL destination_type = CASE(
    network.direction == "outbound" AND NOT CIDR_MATCH(destination.ip, "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"), "external",
    true, "internal"
)
| STATS count() BY event.category, destination_type

This query will:
1. Filter logs to include only those where the `user.name` is "james_spiteri".
2. Create a new field `destination_type` to categorize the destination IP addresses as "external" or "internal".
3. Count the number of events grouped by `event.category` and `destination_type`.

Sonnet 3.5

CONTEXT:
"""
## Possible Initial Compromise and Data Exfiltration

This incident involves a possible compromise of `srv-win-s1-rsa` by `james_spiteri`.

### Summary
User `james_spiteri` on `srv-win-s1-rsa` opened a malicious Excel document that led to the execution of malicious code, a port scan, and potential data exfiltration to a Slack channel. The attack progressed to compromise a Linux host, `srv-nix-es-rsa`, where suspicious scripts were downloaded and executed, culminating in the potential exfiltration of data via Slack.

### Details
- At `2024-05-03T06:12:51.723Z`, user `james_spiteri` opened a malicious Excel document: {{ file.name Upcoming Events February 2018.xls }} on `srv-win-s1-rsa`.
- This triggered a low severity alert for suspicious registry key creation (`Evasion`) and another for loading a VBE DLL, which suggests macro execution (`Evasion`).  
- Moments later, a new file, `K7Y4P6H1.exe`, was created via the `certutil.exe` process, indicating the Excel macro likely decoded and dropped this file. 
- This triggered a low severity alert for abnormal document behavior (`Exploitation`) on `srv-win-s1-rsa`.
- At `2024-05-03T06:19:44.992Z`, the file `K7Y4P6H1.exe` was detected as malicious by SentinelOne and triggered a critical alert on `srv-win-s1-rsa`.
- At `2024-05-03T06:35:53.486Z`, user `james_spiteri` initiated a port scan using `nmap.exe` targeting `10.128.0.104` from `srv-win-s1-rsa`. This triggered a high severity alert for a port scan from an internal host. 
- This activity aligns with the Tactic Reconnaissance (`TA0043`) and Techniques Active Scanning (`T1595`) and Gather Victim Network Information (`T1590`).  The specific sub-techniques used were Scanning IP Blocks (`T1595.001`) and IP Addresses (`T1590.005`). 
- At `2024-05-03T06:36:20.166Z`, on `srv-nix-es-rsa`, a suspicious process chain was initiated. The `java` process spawned a `sh` process. 
- This new process downloaded two files: `pwn.sh` and `headerfile.txt` from `https://images.swiftcrypto.com`, made `pwn.sh` executable, and then executed it. 
- This activity aligns with the Tactic Execution (`TA0002`) and the Technique Command and Scripting Interpreter (`T1059`). The sub-technique used was JavaScript (`T1059.007`). 
- Subsequently, at `2024-05-03T06:38:04.751Z`, the `curl` process was used to upload a file, `l00t.zip`, to a Slack channel using the uploaded `headerfile.txt` for authentication. This suggests potential data exfiltration. 
- This activity aligns with the Tactics Command and Control (`TA0011`) and Exfiltration (`TA0010`) and the Techniques Web Service (`T1102`), Exfiltration Over C2 Channel (`T1041`), and Exfiltration Over Web Service (`T1567`).

### Attack Chain
- Reconnaissance
- Initial Access
- Execution
- Persistence
- Privilege Escalation
- Discovery
- Lateral Movement
- Command & Control
- Exfiltration

"""

How would I remediate this incident according to our internal policies, and who should I contact?

Response:

Based on the information provided and our internal policies, I can provide you with guidance on how to remediate this incident and who to contact. Here's a step-by-step approach:

1. Contact the responsible team:
   Jimmy Beans is responsible for the Windows hosts involved in this incident. You should contact him immediately via Slack (@jimmyb) or open a GitHub issue at https://github.com/jamesspi/assistant-demos/issues.

2. Isolate the affected systems:
   Immediately isolate both `srv-win-s1-rsa` and `srv-nix-es-rsa` from the network to prevent further spread of the malware or data exfiltration.

3. Preserve evidence:
   Before making any changes, ensure that all relevant logs and system states are preserved for forensic analysis.

4. Remove malicious files:
   Delete the identified malicious files, including `Upcoming Events February 2018.xls`, `K7Y4P6H1.exe`, `pwn.sh`, and `headerfile.txt`.

5. Scan for additional malware:
   Perform a thorough malware scan on both affected systems using up-to-date antivirus software.

6. Patch vulnerabilities:
   Ensure all systems are patched with the latest security updates, especially focusing on any vulnerabilities that may have been exploited.

7. Reset credentials:
   Change all passwords for the affected systems and any accounts that may have been compromised, especially for user `james_spiteri`.

8. Review and restrict permissions:
   Audit user permissions, especially for `james_spiteri`, and implement the principle of least privilege.

9. Investigate data exfiltration:
   Analyze the extent of data exfiltration to the Slack channel and take necessary steps to secure any compromised data.

10. Enhance network security:
    Implement or update firewalls and intrusion detection/prevention systems to detect and prevent similar attacks in the future.

11. Update security policies:
    Review and update security policies, especially regarding the handling of email attachments and macro execution in Office documents.

12. Conduct security awareness training:
    Provide additional security awareness training to all employees, with a focus on recognizing phishing attempts and handling suspicious attachments.

13. Monitor for ongoing threats:
    Implement enhanced monitoring for any signs of persistent threats or additional compromise attempts.

14. Document the incident:
    Create a detailed report of the incident, including the attack chain, affected systems, and remediation steps taken.

15. Create a GitHub issue:
    Open a GitHub issue using the following template:

    **Host(s) involved:** srv-win-s1-rsa, srv-nix-es-rsa
    **Link to Elastic Security Timeline and/or case:** [Insert link here]
    **Host criticality:** [Determine and insert criticality level]

16. Review and update incident response plan:
    Use lessons learned from this incident to improve your organization's incident response plan.

Remember to follow your organization's specific incident response procedures and involve your legal and compliance teams as necessary, especially if there's a possibility of data breach notification requirements.

For more information, refer to our internal wiki at https://ourwiki.com/security-response.

Follow up prompt:

Help me generate an ES|QL query to look for this user's behaviour in logs-*. Count by event.category, and always use "" for field values.

Response:

Based on the ES|QL query generated, here's an explanation of how to use it to look for james_spiteri's behavior in the logs:

FROM logs-*
| WHERE user.name == "james_spiteri"
| STATS count = COUNT(*) BY event.category
| SORT count DESC

This query will help you analyze james_spiteri's activities across various event categories. Here's what each part of the query does:

1. `FROM logs-*`: This searches all indices that start with "logs-", which should cover the relevant log data.

2. `WHERE user.name == "james_spiteri"`: This filters the results to only include events associated with the user "james_spiteri". Note that the username is in double quotes as requested.

3. `STATS count = COUNT(*) BY event.category`: This counts the number of events for each unique event category.

4. `SORT count DESC`: This sorts the results by the count in descending order, so you'll see the most frequent event categories first.

This query will give you a breakdown of james_spiteri's activities categorized by event types. This can be useful for:

1. Understanding the user's normal behavior patterns
2. Identifying any unusual or suspicious activities
3. Focusing your investigation on the most frequent or relevant event categories

When you run this query, you'll get a list of event categories and their corresponding counts, all related to james_spiteri's actions. This will help you pinpoint which areas of activity were most prominent during the incident and guide your further investigation.

For example, you might see categories like "process", "file", "network", or "authentication" with their respective counts. If you notice an unusually high count in a particular category (e.g., many file creation events or network connections), that could indicate areas that need closer examination.

Remember, this query provides a high-level overview. Once you identify interesting event categories, you may want to create more specific queries to drill down into the details of those events.

Additional Follow up:

Can you update that query? If any of the data is network traffic, and the destination ip address is external, create a new field to reflect this.

Response:

Thank you for providing that information. I'll update the query to include this new field for external IP addresses. Here's the modified ES|QL query:

FROM logs-*
| WHERE user.name == "james_spiteri"
| EVAL is_external = CASE(
    CIDR_MATCH(destination.ip, 
        "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", 
        "127.0.0.0/8", "169.254.0.0/16", 
        "224.0.0.0/4", "240.0.0.0/4"
    ), "false", "true"
)
| STATS count = COUNT(*) BY event.category, is_external
| SORT count DESC

This updated query does the following:

1. It still starts by filtering for events related to "james_spiteri".

2. It adds a new field `is_external` using the `EVAL` command and a `CASE` statement. This field will be "true" if the destination IP is external (not matching any of the specified private or special-use IP ranges) and "false" otherwise.

3. The `STATS` command now groups by both `event.category` and `is_external`. This will give you a breakdown of event categories, further split by whether the destination IP (if applicable) was internal or external.

4. The results are still sorted by count in descending order.

This query will help you identify:

1. The types of events (categories) associated with james_spiteri's activities.
2. For each event category, how many events involved external IP addresses (if applicable).

When you run this query, you'll get results that look something like this:

event.category | is_external | count
---------------|-------------|------
process        | false       | 100
network        | true        | 75
network        | false       | 50
file           | false       | 30
authentication | false       | 20

This output would tell you, for example, that there were 75 network events with external destinations and 50 with internal destinations associated with james_spiteri's activities.

Remember that not all event categories will have a meaningful `destination.ip` field. For categories like "process" or "file", the `is_external` field might not be relevant and will likely show as "false" or null.

This enhanced query will give you a more detailed view of james_spiteri's activities, highlighting any interactions with external IP addresses which could be particularly relevant in investigating potential data exfiltration or communication with command and control servers.

Gemini 1.5 Pro

Prompt:

CONTEXT:
"""
## Possible Initial Compromise and Data Exfiltration

This incident involves a possible compromise of `srv-win-s1-rsa` by `james_spiteri`.

### Summary
User `james_spiteri` on `srv-win-s1-rsa` opened a malicious Excel document that led to the execution of malicious code, a port scan, and potential data exfiltration to a Slack channel. The attack progressed to compromise a Linux host, `srv-nix-es-rsa`, where suspicious scripts were downloaded and executed, culminating in the potential exfiltration of data via Slack.

### Details
- At `2024-05-03T06:12:51.723Z`, user `james_spiteri` opened a malicious Excel document: {{ file.name Upcoming Events February 2018.xls }} on `srv-win-s1-rsa`.
- This triggered a low severity alert for suspicious registry key creation (`Evasion`) and another for loading a VBE DLL, which suggests macro execution (`Evasion`).  
- Moments later, a new file, `K7Y4P6H1.exe`, was created via the `certutil.exe` process, indicating the Excel macro likely decoded and dropped this file. 
- This triggered a low severity alert for abnormal document behavior (`Exploitation`) on `srv-win-s1-rsa`.
- At `2024-05-03T06:19:44.992Z`, the file `K7Y4P6H1.exe` was detected as malicious by SentinelOne and triggered a critical alert on `srv-win-s1-rsa`.
- At `2024-05-03T06:35:53.486Z`, user `james_spiteri` initiated a port scan using `nmap.exe` targeting `10.128.0.104` from `srv-win-s1-rsa`. This triggered a high severity alert for a port scan from an internal host. 
- This activity aligns with the Tactic Reconnaissance (`TA0043`) and Techniques Active Scanning (`T1595`) and Gather Victim Network Information (`T1590`).  The specific sub-techniques used were Scanning IP Blocks (`T1595.001`) and IP Addresses (`T1590.005`). 
- At `2024-05-03T06:36:20.166Z`, on `srv-nix-es-rsa`, a suspicious process chain was initiated. The `java` process spawned a `sh` process. 
- This new process downloaded two files: `pwn.sh` and `headerfile.txt` from `https://images.swiftcrypto.com`, made `pwn.sh` executable, and then executed it. 
- This activity aligns with the Tactic Execution (`TA0002`) and the Technique Command and Scripting Interpreter (`T1059`). The sub-technique used was JavaScript (`T1059.007`). 
- Subsequently, at `2024-05-03T06:38:04.751Z`, the `curl` process was used to upload a file, `l00t.zip`, to a Slack channel using the uploaded `headerfile.txt` for authentication. This suggests potential data exfiltration. 
- This activity aligns with the Tactics Command and Control (`TA0011`) and Exfiltration (`TA0010`) and the Techniques Web Service (`T1102`), Exfiltration Over C2 Channel (`T1041`), and Exfiltration Over Web Service (`T1567`).

### Attack Chain
- Reconnaissance
- Initial Access
- Execution
- Persistence
- Privilege Escalation
- Discovery
- Lateral Movement
- Command & Control
- Exfiltration

"""

How would I remediate this incident according to our internal policies, and who should I contact?

Response:

To remediate this incident, you should immediately isolate the affected hosts (`srv-win-s1-rsa` in the New York data center, and `srv-nix-es-rsa`) to prevent further data exfiltration.  You should then contact the Security Operations Center (SOC) and provide them with the following information:

* **Host(s) involved:** `srv-win-s1-rsa`, `srv-nix-es-rsa`
* **Link to Elastic Security Timeline and/or case:**  (Provide the link to the relevant timeline or case in Elastic Security)
* **Host criticality:** (Specify the criticality of the affected hosts)

For more information, refer to our internal wiki at https://ourwiki.com/security-response. 

Follow up prompt:

Help me generate an ES|QL query to look for this user's behaviour in logs-*. Count by event.category, and always use "" for field values.

Response:


x in logs-* | where x.user.name == "james_spiteri" | group by x.event.category | count as category_count

Leaving out the last prompt in this case, as the point of this issues comes across in the first ES|QL response.