Threat Intel Analytics

[eirsep]

Description

[Describe what this change achieves]

Issues Resolved

1117

Check List

• [ ] New functionality includes testing.
  • [ ] All tests pass
• [ ] New functionality has been documented.
  • [ ] New functionality has javadoc added
• [ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[AWSHurneyt]

Regarding the following jar hell issue in the CI: https://github.com/opensearch-project/security-analytics/actions/runs/9720020887/job/26830800682?pr=1098#step:6:125

#8 2.113 jar1: /usr/share/opensearch/plugins/opensearch-job-scheduler/guava-32.1.3-jre.jar
#8 2.113 jar2: /usr/share/opensearch/plugins/opensearch-alerting/guava-32.0.1-jre.jar

The google guava dependency was removed in PR https://github.com/opensearch-project/alerting/pull/1571

Tried a few means to exclude google guave in this dummy PR, but that did not resolve the issue.

1. https://github.com/AWSHurneyt/security-analytics/pull/3/files#diff-49a96e7eea8a94af862798a45174e6ac43eb4f8b4bd40759b5da63ba31ec3ef7R152
2. https://github.com/AWSHurneyt/security-analytics/pull/3/files#diff-49a96e7eea8a94af862798a45174e6ac43eb4f8b4bd40759b5da63ba31ec3ef7R178

Unsure why the alerting artifact would still include that dependency; reached out to @sbcd90 for insight.

[sbcd90]

Regarding the following jar hell issue in the CI: https://github.com/opensearch-project/security-analytics/actions/runs/9720020887/job/26830800682?pr=1098#step:6:125

#8 2.113 jar1: /usr/share/opensearch/plugins/opensearch-job-scheduler/guava-32.1.3-jre.jar
#8 2.113 jar2: /usr/share/opensearch/plugins/opensearch-alerting/guava-32.0.1-jre.jar

The google guava dependency was removed in PR opensearch-project/alerting#1571

Tried a few means to exclude google guave in this dummy PR, but that did not resolve the issue.

1. https://github.com/AWSHurneyt/security-analytics/pull/3/files#diff-49a96e7eea8a94af862798a45174e6ac43eb4f8b4bd40759b5da63ba31ec3ef7R152
2. https://github.com/AWSHurneyt/security-analytics/pull/3/files#diff-49a96e7eea8a94af862798a45174e6ac43eb4f8b4bd40759b5da63ba31ec3ef7R178

Unsure why the alerting artifact would still include that dependency; reached out to @sbcd90 for insight.

hi @AWSHurneyt , the test failures https://github.com/opensearch-project/security-analytics/actions/runs/9720020887/job/26830800682?pr=1098#step:6:125 are due to the latest docker image not being updated with alerting remote monitor changes. But i see some of the threat intel tests failing. https://github.com/opensearch-project/security-analytics/actions/runs/9727738710/job/26847616424?pr=1098 Please have a look at them.

[AWSHurneyt]

The EnabledIfSystemProperty annotation (example) in the TestS3ConnectionRestIT, and SATIFSourceConfigRestApiIT test suites should prevent them from running in the CIs.

Those test failures can be ignored.

As mentioned in the test doc comment (example), they require certain parameters to be provided in order to run successfully. Those parameters are not currently configured to be provided by the github CI, so the tests will fail.

Will investigate why the EnabledIfSystemProperty annotation is allowing those tests to run, and raise a separate PR to fix that.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 57.61076% with 2459 lines in your changes missing coverage. Please review.

Project coverage is 67.59%. Comparing base (e8d7879) to head (5b12588). Report is 1 commits behind head on main.

Files	Patch %	Lines
...el/service/SATIFSourceConfigManagementService.java	22.45%	292 Missing and 5 partials :warning:
.../threatIntel/service/SATIFSourceConfigService.java	36.86%	168 Missing and 5 partials :warning:
...atIntel/sacommons/monitor/ThreatIntelAlertDto.java	19.70%	162 Missing and 1 partial :warning:
.../threatIntel/iocscan/service/SaIoCScanService.java	47.28%	119 Missing and 17 partials :warning:
...lytics/threatIntel/model/SATIFSourceConfigDto.java	64.35%	83 Missing and 30 partials :warning:
...analytics/threatIntel/model/SATIFSourceConfig.java	66.03%	75 Missing and 32 partials :warning:
...curityanalytics/services/STIX2IOCFetchService.java	24.34%	87 Missing :warning:
.../securityanalytics/services/STIX2IOCFeedStore.java	49.27%	66 Missing and 4 partials :warning:
...g/opensearch/securityanalytics/model/STIX2IOC.java	62.94%	48 Missing and 15 partials :warning:
...threatIntel/iocscan/dao/BaseEntityCrudService.java	63.39%	49 Missing and 7 partials :warning:
... and 81 more

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1098 +/- ## ============================================ - Coverage 71.57% 67.59% -3.99% - Complexity 2490 3424 +934 ============================================ Files 297 403 +106 Lines 13615 19393 +5778 Branches 1461 1969 +508 ============================================ + Hits 9745 13108 +3363 - Misses 3218 5338 +2120 - Partials 652 947 +295 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[opensearch-trigger-bot[bot]]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.x
# Create a new branch
git switch --create backport-1098-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ad621863cb491e5ea6d666486d39246b6893aa4b
# Push it to GitHub
git push --set-upstream origin backport-1098-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport-1098-to-2.x.