Whitespace issues - Githubissues

acumen-kevinr commented 1 year ago

Hi,

There are some issues with whitespace being added to the rule pushes, so that it will detect changes on each rule, even if you keep them identical.

Is it possible to modify the code to eliminate this whitespace issue at all?

Thanks

acumen-kevinr commented 1 year ago

Seems to be related to rule_id being populated after creation and then constantly trying to change/strip it out?

tsigouris007 commented 1 year ago

Hi @silvercloud-kevinrobertson , I am aware of this issue a couple of months now yet I had no time fixing it. I'll fix it in the upcoming version. Still you can continue using the provider without having trouble apart from the annoying output. I hope to eliminate it by next month. Thank you for your patience.

acumen-kevinr commented 1 year ago

Hi there @tsigouris007 ,

Thanks for the reply - actually it does cause us an issue as we have a very large number of resources pushed out across multiple deployments, so we hit timeout issues unfortunately.

tsigouris007 commented 1 year ago

Hi @silvercloud-kevinrobertson , can you try using version v0.0.5? It seems that the issue is fixed for me.

acumen-kevinr commented 1 year ago

Hi @tsigouris007 , it seems like the same issue here.

Each time it pushes it attempts to change the id value.

tsigouris007 commented 1 year ago

Have you checked the examples here: https://github.com/tsigouris007/terraform-provider-elastic-siem-detection/tree/main/examples

The id does not seem to cause any changes while testing with 40 rules. Did you copy the id from ES SIEM and paste it into the rule or did you specify your own? That might be the case. Also keep in mind that you have to use rule_id field and not id (this is internally given by ES).

acumen-kevinr commented 1 year ago

Here is an example of one of my rules, not sure what's wrong with the format here, if anything?

author:
  - Silver Cloud
description: "Identifies the use of the Exchange PowerShell cmdlet, New-MailBoxExportRequest,
  to export the contents of a primary\nmailbox or archive to a .pst file. Adversaries
  may target user email to collect sensitive information.\n"
false_positives:
  - Legitimate exchange system administration activity.
from: now-9m
index:
  - logs-endpoint.events.*
  - logs-windows.*
language: eql
license: Silver Cloud
name: Exporting Exchange Mailbox via PowerShell
note: "## Triage and analysis\n\n### Investigating Exporting Exchange Mailbox via
  PowerShell\n\nEmail mailboxes and their information can be valuable assets for attackers.
  Company mailboxes often contain sensitive information such as login credentials,
  intellectual property, financial data, and personal information, making them high-value
  targets for malicious actors.\n\nThe `New-MailBoxExportRequest` cmdlet is used to
  begin the process of exporting contents of a primary mailbox or archive to a .pst
  file. Note that this is done on a per-mailbox basis and this cmdlet is available
  only in on-premises Exchange.\n\nAttackers can abuse this functionality in preparation
  for exfiltrating contents, which is likely to contain sensitive and strategic data.\n
  \n#### Possible investigation steps\n\n- Investigate the process execution chain
  (parent process tree) for unknown processes. Examine their executable files for
  prevalence, whether they are located in expected locations, and if they are signed
  with valid digital signatures.\n- Investigate other alerts associated with the user/host
  during the past 48 hours.\n- Investigate the export operation:\n  - Identify the
  user account that performed the action and whether it should perform this kind of
  action.\n  - Contact the account owner and confirm whether they are aware of this
  activity.\n  - Check if this operation was approved and performed according to the
  organization's change management policy.\n  - Retrieve the operation status and
  use the `Get-MailboxExportRequest` cmdlet to review previous requests.\n  - By default,
  no group in Exchange has the privilege to import or export mailboxes. Investigate
  administrators that assigned the \"Mailbox Import Export\" privilege for abnormal
  activity.\n- Investigate if there is a significant quantity of export requests in
  the alert timeframe. This operation is done on a per-mailbox basis and can be part
  of a mass export.\n- If the operation was completed successfully:\n  - Check if
  the file is on the path specified in the command.\n  - Investigate if the file was
  compressed, archived, or retrieved by the attacker for exfiltration.\n\n### False
  positive analysis\n\n- This mechanism can be used legitimately. Analysts can dismiss
  the alert if the administrator is aware of the activity and it is done with proper
  approval.\n\n### Response and remediation\n\n- Initiate the incident response process
  based on the outcome of the triage.\n- If the involved host is not the Exchange
  server, isolate the host to prevent further post-compromise behavior.\n- Use the
  `Remove-MailboxExportRequest` cmdlet to remove fully or partially completed export
  requests.\n- Prioritize cases that involve personally identifiable information (PII)
  or other classified data.\n- Investigate credential exposure on systems compromised
  or used by the attacker to ensure all compromised accounts are identified. Reset
  passwords for these accounts and other potentially compromised credentials, such
  as email, business systems, and web services.\n- Review the privileges of users
  with the \"Mailbox Import Export\" privilege to ensure that the least privilege
  principle is being followed.\n- Run a full antimalware scan. This may reveal additional
  artifacts left in the system, persistence mechanisms, and malware components.\n
  - Determine the initial vector abused by the attacker and take action to prevent
  reinfection through the same vector.\n- Using the incident response data, update
  logging and audit policies to improve the mean time to detect (MTTD) and the mean
  time to respond (MTTR).\n\n## Setup\n\nIf enabling an EQL rule on a non-elastic-agent
  index (such as beats) for versions <8.2, events will not define `event.ingested`
  and default fallback for EQL rules was not added until 8.2, so you will need to
  add a custom pipeline to populate `event.ingested` to @timestamp for this rule to
  work.\n"
references:
  - https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-solarwinds-compromise-to-breach-organizations/
  - https://docs.microsoft.com/en-us/powershell/module/exchange/new-mailboxexportrequest?view=exchange-ps
risk_score: 47
rule_id: c1f26dc7-f383-425e-bdf0-d05845e8bc84
severity: medium
tags:
  - 'Domain: Endpoint'
  - 'OS: Windows'
  - 'Use Case: Threat Detection'
  - 'Tactic: Collection'
  - 'Resources: Investigation Guide'
  - 'Data Source: Elastic Endgame'
timestamp_override: event.ingested
type: eql
query: 'process where host.os.type == "windows" and event.type == "start" and process.name:
  ("powershell.exe", "pwsh.exe", "powershell_ise.exe") and process.command_line :
  ("*MailboxExportRequest*", "*-Mailbox*-ContentFilter*")'
threat:
  - framework: MITRE ATT&CK
    technique:
      - id: T1005
        name: Data from Local System
        reference: https://attack.mitre.org/techniques/T1005/
      - id: T1114
        name: Email Collection
        reference: https://attack.mitre.org/techniques/T1114/
        subtechnique:
          - id: T1114.002
            name: Remote Email Collection
            reference: https://attack.mitre.org/techniques/T1114/002/
    tactic:
      id: TA0009
      name: Collection
      reference: https://attack.mitre.org/tactics/TA0009/
interval: 8m
max_signals: 100
enabled: true
actions:
  - group: default
    action_type_id: .webhook
    id: 685d0ba0-26d2-11ee-ab5b-39c27c3337ae
    params:
      body: "{\n        \"alert.id\": \"{{alert.id}}\",\n        \"context.alerts\"\
        : \"{{context.alerts}}\",\n        \"context.results.link\": \"{{context.results_link}}\"\
        ,\n        \"context.rule.description\": \"{{context.rule.description}}\"\
        ,\n        \"context.rule.id\": \"{{context.rule.id}}\",\n        \"context.rule.index\"\
        : \"{{context.rule.index}}\",\n        \"context.rule.name\": \"{{context.rule.name}}\"\
        ,\n        \"context.rule.severity\": \"{{context.rule.severity}}\",\n   \
        \     \"context.rule.threat\": \"{{context.rule.threat}}\",\n        \"context.rule.timeline.id\"\
        : \"{{context.rule.timeline_id}}\",\n        \"date\": \"{{date}}\",\n   \
        \     \"kibana.base.url\": \"{{kibanaBaseUrl}}\",\n        \"rule.name\":
        \"{{rule.name}}\",\n        \"rule.id\": \"{{rule.id}}\",\n        \"rule.tags\"\
        : \"{{rule.tags}}\",\n        \"rule.type\": \"{{rule.type}}\",\n        \"\
        rule.url\": \"{{rule.url}}\",\n        \"state.signals.count\": \"{{state.signals_count}}\"\
        ,\n        \"alert.flapping\": \"{{alert.flapping}}\",\n        \"context.rule.type\"\
        : \"{{context.rule.type}}\",\n        \"context.rule.query\": \"{{context.rule.query}}\"\
        ,\n        \"rule.space.id\": \"{{rule.spaceId}}\"\n    }"
exceptions_list:
  - list_id: exchange_mailbox_export_exceptions_list
    id: c0ab6fa0-4341-11ee-8e45-a7230eb1e864
    namespace_type: single
    type: detection

tsigouris007 commented 1 year ago

This looks right by eye. Would you mind try using a rule_id string like rule_id: exchange_mailbox_via_psh instead of the c1f26dc7-f383-425e-bdf0-d05845e8bc84 (following the ES logic in their API). Is this rule_id the detected changed or is it another one?

acumen-kevinr commented 1 year ago

Hi there,

So, I've done that and redeployed - exact same issue. I do NOT get the issue with your curated rules, but only with my rules...not sure what on earth would be causing this?

Even though both of these rules have rule_id: collection_mailbox_export

Really confused!

acumen-kevinr commented 1 year ago

You can also see the ID is totally different than my previous screenshot as I did a destroy/apply...this is set by the system, it's not an ID anywhere that I am setting manually. The only reference to any kind of ID is in the exceptions-lists and actions...

acumen-kevinr commented 1 year ago

Okay, I've found the issue.

The problem is because I have ID's configured under actions - however these are necessary for my deployment.

I can only presume you have some kind of exception in place for the ID under exception lists? Can that be applied to actions too? Thanks a lot.

acumen-kevinr commented 1 year ago

Hi @tsigouris007 did you see my above message on what the issue is?

Thanks

tsigouris007 commented 1 year ago

Good morning @silvercloud-kevinrobertson , Yeah I saw that. I 'll try fixing it this week.

tsigouris007 commented 1 year ago

Hi @silvercloud-kevinrobertson , would you mind trying out the newest version v0.0.6?

acumen-kevinr commented 1 year ago

This has fixed the issue - fantastic work.

However (yes, sorry), it seems like Timelines are not supported...is this something you would consider adding in?

acumen-kevinr commented 1 year ago

There also seems to be an issue with 'interval' as well, being picked up as not being necessary. However, adding this into the yaml files doesn't appear to do much good either.

tsigouris007 commented 1 year ago

This has fixed the issue - fantastic work.

However (yes, sorry), it seems like Timelines are not supported...is this something you would consider adding in?

Can you provide me with an example including the timeline fields? While reading the documentation I did not find any specification for timelines in detection rules as you can see: https://www.elastic.co/guide/en/security/current/rules-api-create.html

There is a whole different API for timelines which is beefy to implement right now but I can add it in my todos: https://www.elastic.co/guide/en/security/current/timeline-api-create.html

Also can you provide me an example with the interval? I don't need real data, just the types to be correct.

acumen-kevinr commented 1 year ago

Hi,

So, not asking to implement timelines specifically but when you create rules, you can specify a default timeline within a rule.

I don't think the API documentation is up to date, however the 'elastic-detection-rules' project definitely pushes the rules via TOML files: https://github.com/elastic/detection-rules/blob/a46797b987febb8f4db8748fcee33f6eb63df1d1/detection_rules/etc/api_schemas/8.10/8.10.query.json

Here's one rule example:

author:

Silver Cloud description: "This rule is triggered when an IP address indicator from the Threat Intel Filebeat module or integrations has a match\nagainst a network event.\n" from: now-125m index:
auditbeat-*
filebeat-*
logs-*
packetbeat-* interval: 2h language: kuery license: Silver Cloud name: Threat Intel IP Address Indicator Match references:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-threatintel.html
https://www.elastic.co/guide/en/security/master/es-threat-intel-integrations.html
https://www.elastic.co/security/tip risk_score: 50 rule_id: 0c41e478-5263-4c69-8f9e-7dfd2c22da64 severity: low tags:
'OS: Windows'
'Data Source: Elastic Endgame'
'Rule Type: Indicator Match' timeline_id: 36797360-a2db-11ed-9774-4fe286ec25ef timeline_title: Generic Threat Match Timeline type: threat_match threat_index:
filebeat-*
logs-ti_* threat_indicator_path: threat.indicator threatquery: @.*** >= \"now-30d/d\" and event.module:(threatintel or ti) and threat.indicator.ip: and\n not labels.is_ioc_transform_source:\"true\"\n" query: source.ip: or destination.ip: threat_filters:
$state: store: appState meta: negate: false disabled: false type: phrase key: event.category params: query: threat query: match_phrase: event.category: threat
$state: store: appState meta: negate: false disabled: false type: phrase key: event.kind params: query: enrichment query: match_phrase: event.kind: enrichment
$state: store: appState meta: negate: false disabled: false type: phrase key: event.type params: query: indicator query: match_phrase: event.type: indicator threat_mapping:
entries:
- field: source.ip type: mapping value: threat.indicator.ip
entries:
- field: destination.ip type: mapping value: threat.indicator.ip max_signals: 100 enabled: true actions:
group: default action_type_id: .webhook id: 2e8c29a0-32ae-11ee-b9e7-1b3169b63d26 params: body: "{\n \"alert.id\": \"{{alert.id}}\",\n \"context.alerts\"\ : \"{{context.alerts}}\",\n \"context.results.link\": \"{{context.results_link}}\"\ ,\n \"context.rule.description\": \"{{context.rule.description}}\"\ ,\n \"context.rule.id\": \"{{context.rule.id}}\",\n \"context.rule.index\"\ : \"{{context.rule.index}}\",\n \"context.rule.name\": \"{{context.rule.name}}\"\ ,\n \"context.rule.severity\": \"{{context.rule.severity}}\",\n \ \ \"context.rule.threat\": \"{{context.rule.threat}}\",\n \"context.rule.timeline.id\"\ : \"{{context.rule.timeline_id}}\",\n \"date\": \"{{date}}\",\n \ \ \"kibana.base.url\": \"{{kibanaBaseUrl}}\",\n \"rule.name\": \"{{rule.name}}\",\n \"rule.id\": \"{{rule.id}}\",\n \"rule.tags\"\ : \"{{rule.tags}}\",\n \"rule.type\": \"{{rule.type}}\",\n \"\ rule.url\": \"{{rule.url}}\",\n \"state.signals.count\": \"{{state.signals_count}}\"\ ,\n \"alert.flapping\": \"{{alert.flapping}}\",\n \"context.rule.type\"\ : \"{{context.rule.type}}\",\n \"context.rule.query\": \"{{context.rule.query}}\"\ ,\n \"rule.space.id\": \"{{rule.spaceId}}\"\n }" exceptions_list:
list_id: threat_intel_ip_exceptions_list id: e1ad78e0-4177-11ee-b77c-d59aedd05f79 namespace_type: single type: detection

Kevin Robertson Chief Security Officer The Exchange, 130 Cubie Street, Glasgow G40 2AF 0141 552 0000 | www.silvercloud.co.uk Certified for Quality and Integrity: Our Commitment to Higher Standards Disclaimer - This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Messages sent to and from us may be monitored. Internet communications cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, we do not accept responsibility for any errors or omissions that are present in this message, or any attachment, that have arisen as a result of e-mail transmission. If verification is required, please request a hard-copy version. Any views or opinions presented are solely those of the author and do not necessarily represent those of the company.

From: gokussj @.> Sent: Tuesday, October 10, 2023 08:35 To: tsigouris007/terraform-provider-elastic-siem-detection @.> Cc: Kevin Robertson @.>; Mention @.> Subject: Re: [tsigouris007/terraform-provider-elastic-siem-detection] Whitespace issues (Issue #28)

This has fixed the issue - fantastic work.

However (yes, sorry), it seems like Timelines are not supported...is this something you would consider adding in?

[image]https://user-images.githubusercontent.com/116726394/273702459-d8e42688-f6c7-48fd-8d83-55c00253d30a.png

Can you provide me with an example including the timeline fields?

While reading the documentation I did not find any specification for timelines in detection rules as you can see: https://www.elastic.co/guide/en/security/current/rules-api-create.html https://url.avanan.click/v2/___https://www.elastic.co/guide/en/security/current/rules-api-create.html___.YXAxZTpzaWx2ZXJjbG91ZDphOm86ZGIwNWNmNWZiYTI2ODBjNTc1NmU2NmU5ZDE1ZDEzOGU6NjoxM2MzOmE2MWQ2YWM5OGZiOTcyOTBmZjZjNWU3NWY3MmNlN2YxYTVjNzhlOTYzN2Y1MjViZTUzNWQwNzdiNzE1NGNmYWU6aDpU

There is a whole different API for timelines which is beefy to implement right now but I can add it in my todos: https://www.elastic.co/guide/en/security/current/timeline-api-create.html https://url.avanan.click/v2/___https://www.elastic.co/guide/en/security/current/timeline-api-create.html___.YXAxZTpzaWx2ZXJjbG91ZDphOm86ZGIwNWNmNWZiYTI2ODBjNTc1NmU2NmU5ZDE1ZDEzOGU6NjozZDkxOjZiNGNmNmFkZjExZTk2NGE1M2ViNTg1NDgzYWExZjlkMWZhMmM0M2RiNDliM2Y1MThkZWE4MzI3NWM3NTk0NmM6aDpU

Also can you provide me an example with the interval? I don't need real data, just the types to be correct.

— Reply to this email directly, view it on GitHubhttps://url.avanan.click/v2/___https://github.com/tsigouris007/terraform-provider-elastic-siem-detection/issues/28%23issuecomment-1754586750___.YXAxZTpzaWx2ZXJjbG91ZDphOm86ZGIwNWNmNWZiYTI2ODBjNTc1NmU2NmU5ZDE1ZDEzOGU6Njo3NzkxOjY2ZDIyMmQwZGViYTM2MTEwYjI0ZDJhNWMyNmVkM2M0YmNhODI1NmVlZGQwMmUzMDBkZTBlY2RhMjAyNDA5Y2Y6aDpU, or unsubscribehttps://url.avanan.click/v2/___https://github.com/notifications/unsubscribe-auth/A32RU6SZQE6LUZAOSLFN2LDX6T3DDAVCNFSM6AAAAAA47Q6IOSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJUGU4DMNZVGA___.YXAxZTpzaWx2ZXJjbG91ZDphOm86ZGIwNWNmNWZiYTI2ODBjNTc1NmU2NmU5ZDE1ZDEzOGU6NjpiM2FkOjRiMjk2NGVhN2ZlZjhiZWY1YmM1MzIwMTE2ZmZjNzM1NzEwM2UzMmVlYjAzOWUzOTJjY2ZiM2M1ZGU1ZmI0ZGM6aDpU. You are receiving this because you were mentioned.Message ID: @.***>

tsigouris007 commented 1 year ago

Hi @silvercloud-kevinrobertson , Try now with version v0.0.7. Timeline fields have been added. NOTE: You need to add both timeline_id and timeline_title otherwise it doesn't get updated.

tsigouris007 / terraform-provider-elastic-siem-detection

Whitespace issues #28