Test case to check updated_date

[shashank-elastic]

Issues

https://github.com/elastic/ia-trade-team/issues/118

Summary

• The purpose of the PR is to enforce the rule authors are updating the rule modified date in field updated_date before merging to main via pull_request.

Additional Information

• The test is not intended to cover direct commits to release branches, because as part of our back-porting workflows we unstage-incompatible-rules before committing to our release branches. Which means when we do the diff there is a discrepancy between what diff main report and what the release branch intends to have. Direct commits to release branches should run unit tests locally as a best practise before committing.
• We added an explicit skip check to check for GitHub events "push" and skip the test case on those events. This will allow local testing without any additional environment variables for dev environments

Code Changes

• Add a Test to ensure modified rules have updated_date field updated.
• The test case will be part of TestRuleMetadata class
• All rules under rules/ and rules_building_block are considered.
• Git diff will fetch us rules modified, and then git diff of each modified file is obtained to fetch the changes
• To handle usecase when a new rule is modified , we use '--diff-filter=M' where only modified files are considered for rule path rules/ and rules_building_block/
• If a rule is modified, then the git diff is examined to check for updated_date changes.
• If the Updated_date is never modified test case fails.
• If the diff changes does not contain "updated_date", then fail test case
• To handle failures when rule changes are backported to protected version branches explained here we have added a test case skip criteria

Edge Case

• Now the dev branch is open for x days and rule is tuned and updated_date is updated once, this test case will succeed if the rule is modified and updated_date is left unchanged after the first updation. All it checks is to contain one modification of updated_date.
• Also if the Branch is merged on a diff date the updated_date can differ from the commit date to main and this will always pass on main. All it checks is to contain one modification of updated_date.

Testing

• Updated_date check testing is the same as updated in PR - https://github.com/elastic/detection-rules/pull/3764
• Skip criteria Testing. Local Testing with no git hub event and base ref --> Test Case Skipped.

  tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic PASSED                                                                                                                                                        [ 57%]
  tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules PASSED                                                                                                                                                          [ 57%]
  tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules_modified PASSED                                                                                                                                                 [ 58%]
  tests/test_all_rules.py::TestRuleMetadata::test_event_dataset PASSED                                                                                                                                                             [ 59%]
  tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED                                                                                                                                                           [ 59%]
  tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries PASSED                                                                                                                                                           [ 60%]
  tests/test_all_rules.py::TestRuleMetadata::test_rule_change_has_updated_date SKIPPED (Skipping this test when not running on pull requests to main branch)             

  Local Testing with following git hub events Event- pull_request and Branch = 8.14 --> Test Case Skipped.

  ❯ export GITHUB_EVENT_NAME=pull_request  
  (.venv) 
  detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
  ❯ export GITHUB_BASE_REF=8.14
  (.venv) 
  detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
  ❯ echo $GITHUB_EVENT_NAME
  pull_request
  (.venv) 
  detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
  ❯ echo $GITHUB_BASE_REF
  8.14

  tests/test_all_rules.py::TestRuleFiles::test_bbr_in_correct_dir PASSED                                                                                                                                                           [ 55%]
  tests/test_all_rules.py::TestRuleFiles::test_non_bbr_in_correct_dir PASSED                                                                                                                                                       [ 56%]
  tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic PASSED                                                                                                                                                        [ 57%]
  tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules PASSED                                                                                                                                                          [ 57%]
  tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules_modified PASSED                                                                                                                                                 [ 58%]
  tests/test_all_rules.py::TestRuleMetadata::test_event_dataset PASSED                                                                                                                                                             [ 59%]
  tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED                                                                                                                                                           [ 59%]
  tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries PASSED                                                                                                                                                           [ 60%]
  tests/test_all_rules.py::TestRuleMetadata::test_rule_change_has_updated_date SKIPPED (Skipping this test when not running on pull requests to main branch)  

Local Testing with following git hub events Event- push and Branch = main--> Test Case Skipped.

detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ export GITHUB_EVENT_NAME=push        
(.venv) 
detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ export GITHUB_BASE_REF=main   
(.venv) 
detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ echo $GITHUB_EVENT_NAME       
push
(.venv) 
detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ echo $GITHUB_BASE_REF         
main

tests/test_all_rules.py::TestRuleFiles::test_bbr_in_correct_dir PASSED                                                                                                                                                           [ 55%]
tests/test_all_rules.py::TestRuleFiles::test_non_bbr_in_correct_dir PASSED                                                                                                                                                       [ 56%]
tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic PASSED                                                                                                                                                        [ 57%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules PASSED                                                                                                                                                          [ 57%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules_modified PASSED                                                                                                                                                 [ 58%]
tests/test_all_rules.py::TestRuleMetadata::test_event_dataset PASSED                                                                                                                                                             [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED                                                                                                                                                           [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries PASSED                                                                                                                                                           [ 60%]
tests/test_all_rules.py::TestRuleMetadata::test_rule_change_has_updated_date SKIPPED (Skipping this test when not running on pull requests to main branch)       

Local Testing with following git hub events Event- pull_request and Branch = main--> Test Case Executed

detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 1m23s 
❯ export GITHUB_EVENT_NAME=pull_request
(.venv) 
detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ echo $GITHUB_EVENT_NAME              
pull_request
(.venv) 
detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ echo $GITHUB_BASE_REF                
main

tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules_modified PASSED                                                                                                                                                 [ 58%]
tests/test_all_rules.py::TestRuleMetadata::test_event_dataset PASSED                                                                                                                                                             [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED                                                                                                                                                           [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries PASSED                                                                                                                                                           [ 60%]
tests/test_all_rules.py::TestRuleMetadata::test_rule_change_has_updated_date PASSED                                                                                                                                              [ 61%]
tests/test_all_rules.py::TestRuleMetadata::test_updated_date_newer_than_creation PASSED          

On Github PR Run. The testcase was executed - sample_run

Testing With Skip criteria for push

Locally this can be run without any additional settings

tests/test_all_rules.py::TestRuleTimelines::test_timeline_has_title PASSED                                                                                                                                                     [ 55%]
tests/test_all_rules.py::TestRuleFiles::test_bbr_in_correct_dir PASSED                                                                                                                                                         [ 55%]
tests/test_all_rules.py::TestRuleFiles::test_non_bbr_in_correct_dir PASSED                                                                                                                                                     [ 56%]
tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic PASSED                                                                                                                                                      [ 57%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules PASSED                                                                                                                                                        [ 57%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules_modified PASSED                                                                                                                                               [ 58%]
tests/test_all_rules.py::TestRuleMetadata::test_event_dataset PASSED                                                                                                                                                           [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED                                                                                                                                                         [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries PASSED                        

On PR the test is checked --> Refer Latest Unit Test Result.

When event is push the test case is skipped

detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 1m25s 
❯ export GITHUB_EVENT_NAME=push          
(.venv) 
detection-rules on  tarde_118 [$?] is 📦 v0.1.0 via 🐍 v3.12.3 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ echo $GITHUB_EVENT_NAME       
push

tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules PASSED                                                                                                                                                        [ 57%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules_modified PASSED                                                                                                                                               [ 58%]
tests/test_all_rules.py::TestRuleMetadata::test_event_dataset PASSED                                                                                                                                                           [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED                                                                                                                                                         [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries PASSED                                                                                                                                                         [ 60%]
tests/test_all_rules.py::TestRuleMetadata::test_rule_change_has_updated_date SKIPPED (Skipping this test when not running on pull requests.)       

[shashank-elastic]

Reconciling this PR design

• The skip test on non pull requests was added to ensure this test would be skipped when we push the backport commits.
• Edge Case like this can be tackled if this pull requests runs once only on main branch was the initial consideration for this PR.
• With this comes the problem of having certain variables set to be able to run it locally. [Refer the testing criteria]

• What are the possibilities that we can have without skipping -- We could dynamically change the base branch ref on PR / Push event to use those. -- My Point of view on this is that the test on push events is invariably wont fetch any events. -- There is a diff of the rule on the PR stage, on Push events ur already looking at the modified file and comparing it with its base branch wont fetch any diff. -- On the other hand, back porting branches wont have any PR(s) and are always on direct commit mode. -- So even if we have this test case to have diff on dynamic branch there will be no result as u will be diffing on the latest branch contents.

  • So we could use a simple negate and skip if the event type is "push" and rest of the cases the test case will run and we will be able to execute this seamlessly locally as well.

  opening this up to @Mikaayenson / @eric-forte-elastic / @traut to see if this is less disruptive for dev workflows.