[zscaler_zia] Support Audit, Endpoint DLP and Sandbox Report logs and add support of new fields in existing integration

janvi-elastic commented 3 months ago

Type of change

Enhancement

Proposed commit message

Added Sandbox Report, Audit and Endpoint DLP data streams and supported new fields in existing integration.
Added data collection logic for Sandbox Report, Audit and Endpoint DLP data streams.
Added the ingest pipeline for Sandbox Report, Audit and Endpoint DLP data streams and updated ingest pipeline for existing integration.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added new dashboards and visualizations.
Added test for pipeline for Sandbox Report, Audit and Endpoint DLP data stream.
Added system test cases for Sandbox Report, Audit and Endpoint DLP data stream.

Checklist

[x] I have reviewed tips for building integrations and this pull request is aligned with them.
[x] I have verified that all data streams collect metrics or logs.
[x] I have added an entry to my package's changelog.yml file.
[x] I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/zscaler_zia directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: zscaler_zia - START ---
╭─────────────┬────────────────┬───────────┬───────────────┬────────┬───────────────╮
│ PACKAGE     │ DATA STREAM    │ TEST TYPE │ TEST NAME     │ RESULT │  TIME ELAPSED │
├─────────────┼────────────────┼───────────┼───────────────┼────────┼───────────────┤
│ zscaler_zia │ alerts         │ system    │ default       │ PASS   │ 46.373272088s │
│ zscaler_zia │ audit          │ system    │ default       │ PASS   │ 40.485803637s │
│ zscaler_zia │ audit          │ system    │ http-endpoint │ PASS   │ 41.322675753s │
│ zscaler_zia │ dns            │ system    │ http-endpoint │ PASS   │ 38.333538235s │
│ zscaler_zia │ dns            │ system    │ tcp           │ PASS   │ 39.109006052s │
│ zscaler_zia │ endpoint_dlp   │ system    │ default       │ PASS   │ 41.008146931s │
│ zscaler_zia │ endpoint_dlp   │ system    │ http-endpoint │ PASS   │ 39.978748865s │
│ zscaler_zia │ firewall       │ system    │ http-endpoint │ PASS   │ 40.694056324s │
│ zscaler_zia │ firewall       │ system    │ tcp           │ PASS   │ 38.442839218s │
│ zscaler_zia │ sandbox_report │ system    │ default       │ PASS   │ 40.302376001s │
│ zscaler_zia │ tunnel         │ system    │ http-endpoint │ PASS   │ 40.059225026s │
│ zscaler_zia │ tunnel         │ system    │ tcp           │ PASS   │ 36.278997431s │
│ zscaler_zia │ web            │ system    │ http-endpoint │ PASS   │ 39.176605828s │
│ zscaler_zia │ web            │ system    │ tcp           │ PASS   │ 40.074749443s │
╰─────────────┴────────────────┴───────────┴───────────────┴────────┴───────────────╯
--- Test results for package: zscaler_zia - END   ---
Done
--- Test results for package: zscaler_zia - START ---
╭─────────────┬────────────────┬───────────┬─────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM    │ TEST TYPE │ TEST NAME                           │ RESULT │ TIME ELAPSED │
├─────────────┼────────────────┼───────────┼─────────────────────────────────────┼────────┼──────────────┤
│ zscaler_zia │ alerts         │ pipeline  │ test-alerts.log                     │ PASS   │  67.962649ms │
│ zscaler_zia │ alerts         │ pipeline  │ (ingest pipeline warnings)          │ PASS   │ 328.357416ms │
│ zscaler_zia │ audit          │ pipeline  │ test-audit-http-endpoint.log        │ PASS   │  47.639687ms │
│ zscaler_zia │ audit          │ pipeline  │ test-audit.log                      │ PASS   │   6.546759ms │
│ zscaler_zia │ audit          │ pipeline  │ (ingest pipeline warnings)          │ PASS   │  326.60558ms │
│ zscaler_zia │ dns            │ pipeline  │ test-dns-http-endpoint.log          │ PASS   │  31.405321ms │
│ zscaler_zia │ dns            │ pipeline  │ test-dns.log                        │ PASS   │    6.69093ms │
│ zscaler_zia │ dns            │ pipeline  │ (ingest pipeline warnings)          │ PASS   │ 327.823959ms │
│ zscaler_zia │ endpoint_dlp   │ pipeline  │ test-endpoint-dlp-http-endpoint.log │ PASS   │   9.165028ms │
│ zscaler_zia │ endpoint_dlp   │ pipeline  │ test-endpoint-dlp.log               │ PASS   │   4.023019ms │
│ zscaler_zia │ endpoint_dlp   │ pipeline  │ (ingest pipeline warnings)          │ PASS   │ 397.464081ms │
│ zscaler_zia │ firewall       │ pipeline  │ test-firewall-http-endpoint.log     │ PASS   │  28.383836ms │
│ zscaler_zia │ firewall       │ pipeline  │ test-firewall.log                   │ PASS   │   5.257008ms │
│ zscaler_zia │ firewall       │ pipeline  │ (ingest pipeline warnings)          │ PASS   │ 313.003642ms │
│ zscaler_zia │ sandbox_report │ pipeline  │ test-sandbox.log                    │ PASS   │  15.592491ms │
│ zscaler_zia │ sandbox_report │ pipeline  │ (ingest pipeline warnings)          │ PASS   │ 323.250863ms │
│ zscaler_zia │ tunnel         │ pipeline  │ test-tunnel-http-endpoint.log       │ PASS   │   7.658297ms │
│ zscaler_zia │ tunnel         │ pipeline  │ test-tunnel.log                     │ PASS   │   8.421268ms │
│ zscaler_zia │ tunnel         │ pipeline  │ (ingest pipeline warnings)          │ PASS   │ 334.043019ms │
│ zscaler_zia │ web            │ pipeline  │ test-web-http-endpoint.log          │ PASS   │  19.744757ms │
│ zscaler_zia │ web            │ pipeline  │ test-web.log                        │ PASS   │   7.631604ms │
│ zscaler_zia │ web            │ pipeline  │ (ingest pipeline warnings)          │ PASS   │ 331.712924ms │
╰─────────────┴────────────────┴───────────┴─────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: zscaler_zia - END   ---
Done
--- Test results for package: zscaler_zia - START ---
╭─────────────┬────────────────┬───────────┬──────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM    │ TEST TYPE │ TEST NAME                                                            │ RESULT │ TIME ELAPSED │
├─────────────┼────────────────┼───────────┼──────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-03b699de-70f6-4ef0-9fa9-49c035b62635 is loaded │ PASS   │       3.28µs │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-44e6d836-f55d-495d-93e0-79da0637042e is loaded │ PASS   │        455ns │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-579d9380-382a-11ed-aa11-3bf35d6f0a84 is loaded │ PASS   │        589ns │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-5de5ab5a-fd15-40bf-8f4c-5206f0ed7416 is loaded │ PASS   │        648ns │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-78eeb3f0-381d-11ed-aa11-3bf35d6f0a84 is loaded │ PASS   │        625ns │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-7b9f74b0-3820-11ed-aa11-3bf35d6f0a84 is loaded │ PASS   │        687ns │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-b335cb40-3811-11ed-aa11-3bf35d6f0a84 is loaded │ PASS   │        656ns │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-c12fb7c5-0f5c-4341-98fb-12dd197f00a6 is loaded │ PASS   │        738ns │
│ zscaler_zia │                │ asset     │ dashboard zscaler_zia-eec7265e-2cd3-422a-924a-c3db6f5d0a42 is loaded │ PASS   │        710ns │
│ zscaler_zia │                │ asset     │ search zscaler_zia-0a8c8745-4147-4027-9017-f8170336d48b is loaded    │ PASS   │        576ns │
│ zscaler_zia │                │ asset     │ search zscaler_zia-46bd26cc-0e3e-4d8c-aa1a-38ce9e006e2f is loaded    │ PASS   │        622ns │
│ zscaler_zia │                │ asset     │ search zscaler_zia-5290e0b2-4324-47f3-beb6-0af3a078b856 is loaded    │ PASS   │        638ns │
│ zscaler_zia │                │ asset     │ search zscaler_zia-69b19dc6-a9dc-4009-b93e-485e77b96886 is loaded    │ PASS   │        658ns │
│ zscaler_zia │                │ asset     │ search zscaler_zia-8fa90910-8a79-48c7-b9e5-a0c137a2f174 is loaded    │ PASS   │        750ns │
│ zscaler_zia │                │ asset     │ search zscaler_zia-c6228fe8-e312-4c72-88f0-ef46959ac84c is loaded    │ PASS   │        746ns │
│ zscaler_zia │                │ asset     │ search zscaler_zia-ce6524a6-67c1-466d-bd99-2ec9223b7f0c is loaded    │ PASS   │        740ns │
│ zscaler_zia │ alerts         │ asset     │ index_template logs-zscaler_zia.alerts is loaded                     │ PASS   │        700ns │
│ zscaler_zia │ alerts         │ asset     │ ingest_pipeline logs-zscaler_zia.alerts-2.21.0 is loaded             │ PASS   │        237ns │
│ zscaler_zia │ audit          │ asset     │ index_template logs-zscaler_zia.audit is loaded                      │ PASS   │        348ns │
│ zscaler_zia │ audit          │ asset     │ ingest_pipeline logs-zscaler_zia.audit-2.21.0 is loaded              │ PASS   │        240ns │
│ zscaler_zia │ dns            │ asset     │ index_template logs-zscaler_zia.dns is loaded                        │ PASS   │        393ns │
│ zscaler_zia │ dns            │ asset     │ ingest_pipeline logs-zscaler_zia.dns-2.21.0 is loaded                │ PASS   │        315ns │
│ zscaler_zia │ endpoint_dlp   │ asset     │ index_template logs-zscaler_zia.endpoint_dlp is loaded               │ PASS   │        518ns │
│ zscaler_zia │ endpoint_dlp   │ asset     │ ingest_pipeline logs-zscaler_zia.endpoint_dlp-2.21.0 is loaded       │ PASS   │        353ns │
│ zscaler_zia │ firewall       │ asset     │ index_template logs-zscaler_zia.firewall is loaded                   │ PASS   │        472ns │
│ zscaler_zia │ firewall       │ asset     │ ingest_pipeline logs-zscaler_zia.firewall-2.21.0 is loaded           │ PASS   │        279ns │
│ zscaler_zia │ sandbox_report │ asset     │ index_template logs-zscaler_zia.sandbox_report is loaded             │ PASS   │        468ns │
│ zscaler_zia │ sandbox_report │ asset     │ ingest_pipeline logs-zscaler_zia.sandbox_report-2.21.0 is loaded     │ PASS   │        388ns │
│ zscaler_zia │ tunnel         │ asset     │ index_template logs-zscaler_zia.tunnel is loaded                     │ PASS   │        661ns │
│ zscaler_zia │ tunnel         │ asset     │ ingest_pipeline logs-zscaler_zia.tunnel-2.21.0 is loaded             │ PASS   │        369ns │
│ zscaler_zia │ web            │ asset     │ index_template logs-zscaler_zia.web is loaded                        │ PASS   │        551ns │
│ zscaler_zia │ web            │ asset     │ ingest_pipeline logs-zscaler_zia.web-2.21.0 is loaded                │ PASS   │        372ns │
╰─────────────┴────────────────┴───────────┴──────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: zscaler_zia - END   ---
Done
Run static tests for the package
--- Test results for package: zscaler_zia - START ---
╭─────────────┬────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM    │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────────┼────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ zscaler_zia │ alerts         │ static    │ Verify sample_event.json │ PASS   │ 129.859849ms │
│ zscaler_zia │ audit          │ static    │ Verify sample_event.json │ PASS   │ 125.784243ms │
│ zscaler_zia │ dns            │ static    │ Verify sample_event.json │ PASS   │ 201.728126ms │
│ zscaler_zia │ endpoint_dlp   │ static    │ Verify sample_event.json │ PASS   │ 158.378334ms │
│ zscaler_zia │ firewall       │ static    │ Verify sample_event.json │ PASS   │  142.66921ms │
│ zscaler_zia │ sandbox_report │ static    │ Verify sample_event.json │ PASS   │ 151.864611ms │
│ zscaler_zia │ tunnel         │ static    │ Verify sample_event.json │ PASS   │ 134.488033ms │
│ zscaler_zia │ web            │ static    │ Verify sample_event.json │ PASS   │ 142.401073ms │
╰─────────────┴────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: zscaler_zia - END   ---
Done

Related issues

Closes https://github.com/elastic/integrations/issues/9960

Screenshot

SS_1 SS_2

elasticmachine commented 3 months ago

:rocket: Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine commented 3 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy commented 2 months ago

Hey @piyush-elastic,

I think this could have benefited from being four separate PRs.

Regarding @efd6 request: This is a general ask from our team and helps in improving quality of the reviews. Please consider splitting these large PRs into smaller ones.

jamiehynds commented 2 months ago

I think we can derive useful info by splitting events under sandbox report categories like SystemSummary, Networking, SecurityBypass, Stealth, etc. Right now they are all array of objects and we cant derive much info from them. The only thing useful right now is at the top level report information, like # of Reports completed, Avg. time taken to complete, etc.

If its not part of current ask, maybe we could take up as an enhancement.

@jamiehynds WDYT?

100% - we should be parsing and normalising as much as we can from these reports including category and categorisation fields (as shown in the Zscaler example reports here). those fields are far more valuable than the number of reports completed, etc. @piyush-elastic we can add those before merging the PR - we can't take it as an enhancement for a future release.

piyushw-crest commented 2 months ago

Hey @piyush-elastic,

I think this could have benefited from being four separate PRs.

Regarding @efd6 request: This is a general ask from our team and helps in improving quality of the reviews. Please consider splitting these large PRs into smaller ones.

@kcreddy - Do you want us to split PR module wise like Data collection, FM , viz? or data-stream wise?

kcreddy commented 2 months ago

Do you want us to split PR module wise like Data collection, FM , viz? or data-stream wise?

@piyushw-crest I would like if PRs were separated by datastream (for new datastreams/integrations). I think thats what Dan also meant by 4 PRs. cc: @efd6