Add automated tests for security packages

[andrewkroh]

Automated tests need to be added to the security packages that we migrated over from Filebeat. This will prevent future regressions and save us from having to manually test. The tests should verify:

• Data can be collected through each supported input type.
• Fields contained in the final documents are defined in package.
• There are no data type conflicts with latest ECS release.

I think it would be ideal if all of this testing could be accomplished through system tests with the elastic-package tool. I'll work through setting up a test and see what additional features will be needed and work that via issues in the elastic-package repo.

Input Types

These are the different input types used in packages.

• logfile
• udp
• tcp
• tcp with tls
• aws-s3
• gcp-pubsub
• netflow
• httpjson
• o365audit
• windows event log (via .evtx file)

Data Streams to Test

This is every data stream and the inputs that they support.

• AWS
  • CloudTrail (pipeline test https://github.com/elastic/integrations/pull/408)
  • s3
  • VPC Flow
  • s3
• Auditd
  • Log
  • logfile (system & pipeline)
• CEF
  • Log (has pipeline test)
  • logfile (system test https://github.com/elastic/integrations/pull/575)
  • syslog (system test https://github.com/elastic/integrations/pull/575)
• CheckPoint
  • Firewall (pipeline test https://github.com/elastic/integrations/pull/409)
  • logfile (system test https://github.com/elastic/integrations/pull/574)
  • udp (system test https://github.com/elastic/integrations/pull/553)
  • tcp (system test https://github.com/elastic/integrations/pull/553)
• Cisco
  • ASA
  • logfile https://github.com/elastic/integrations/pull/414
  • udp (system https://github.com/elastic/integrations/pull/578)
  • FTD
  • logfile (system https://github.com/elastic/integrations/pull/578)
  • udp (system https://github.com/elastic/integrations/pull/578)
  • IOS
  • logfile (system test https://github.com/elastic/integrations/pull/416)
  • syslog (system https://github.com/elastic/integrations/pull/578)
  • Meraki
  • logfile (system test #428)
  • udp (system https://github.com/elastic/integrations/pull/578)
  • tcp (system https://github.com/elastic/integrations/pull/578)
  • Nexus
  • logfile (system https://github.com/elastic/integrations/pull/578)
  • udp (system https://github.com/elastic/integrations/pull/578)
  • tcp (system https://github.com/elastic/integrations/pull/578)
• CrowdStrike
  • Falcon
  • logfile (system test https://github.com/elastic/integrations/pull/429) (add it again https://github.com/elastic/integrations/pull/583)
• Fortinet
  • FortiClient Endpoint security
  • udp (system https://github.com/elastic/integrations/pull/585)
  • tcp (system https://github.com/elastic/integrations/pull/585)
  • logfile (system test https://github.com/elastic/integrations/pull/432)
  • Firewall
  • udp (system https://github.com/elastic/integrations/pull/585)
  • tcp (system https://github.com/elastic/integrations/pull/585)
  • logfile (pipeline test https://github.com/elastic/integrations/pull/437) (system https://github.com/elastic/integrations/pull/585)
  • FortiMail
  • udp (system https://github.com/elastic/integrations/pull/585)
  • tcp (system https://github.com/elastic/integrations/pull/585)
  • logfile (system test https://github.com/elastic/integrations/pull/438)
  • Manager/Analyzer
  • udp (system https://github.com/elastic/integrations/pull/585)
  • tcp (system https://github.com/elastic/integrations/pull/585)
  • logfile (system test https://github.com/elastic/integrations/pull/439)
• Google Workspace
  • admin
  • httpjson
  • logfile (system test ✅)
  • drive
  • httpjson
  • logfile (system test ✅)
  • groups
  • httpjson
  • logfile (system test ✅)
  • login
  • httpjson
  • logfile (system test ✅)
  • saml
  • httpjson
  • logfile (system test ✅)
  • user_accounts
  • httpjson
  • logfile (system test ✅)
• iptables
  • log (pipeline test ✅)
  • logfile (system test ✅)
  • syslog (udp) (system test ✅)
• Juniper
  • JUNOS (rsa2elk, no pipeline test))
  • udp (system test ✅ https://github.com/elastic/integrations/pull/588)
  • tcp (system test ✅ https://github.com/elastic/integrations/pull/588)
  • logfile (system test ✅ https://github.com/elastic/integrations/pull/588)
  • Netscreen (rsa2elk, no pipeline test)
  • udp (system test ✅ https://github.com/elastic/integrations/pull/588)
  • tcp (system test ✅ https://github.com/elastic/integrations/pull/588)
  • logfile (system test #444)
  • SRX (pipeline test https://github.com/elastic/integrations/pull/443)
  • udp (system test ✅ https://github.com/elastic/integrations/pull/588)
  • tcp (system test ✅ https://github.com/elastic/integrations/pull/588)
  • logfile (system test ✅ https://github.com/elastic/integrations/pull/588)
• Microsoft
  • DHCP
  • logfile (system test https://github.com/elastic/integrations/pull/447)
• Netflow
  • Log
  • netflow (system tests ✅ https://github.com/elastic/integrations/pull/590)
• O365
  • Audit
  • o365audit
  • logfile (system test https://github.com/elastic/integrations/pull/449)
• Okta
  • System
  • httpjson
  • logfile (system test https://github.com/elastic/integrations/pull/452)
• PANW
  • PAN-OS
  • syslog (system test https://github.com/elastic/integrations/pull/594)
  • logfile (system+pipeline https://github.com/elastic/integrations/pull/454)
• Suricata
  • EVE
  • logfile (system+pipeline https://github.com/elastic/integrations/pull/457)
• System
  • Auth
  • logfile (pipeliine https://github.com/elastic/integrations/pull/463)
• Windows
  • Forwarded
  • winlog
  • PowerShell
  • winlog
  • PowerShell Operational
  • winlog
  • Security
  • winlog
  • Sysmon
  • winlog
• Zeek (@leehinman)
  • logfile (systems test #448)
• Zoom
  • Webhook (pipeline https://github.com/elastic/integrations/pull/598)
  • http_endpoint (system https://github.com/elastic/integrations/pull/598)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

Issues encountered while implementing testing (I will open issues and link here):

• Need to be able to execute multiple system tests for a data stream. Some data streams have several input options (syslog, tcp, udp) that need exercised. https://github.com/elastic/elastic-package/issues/208
• Need to be able to select one of several docker-compose services to use during the tests. The service requirements are different when testing logs vs. syslog+tcp vs. syslog+udp.
• Some input types are not easily testable like s3+sqs. In order to add more test coverage of these data streams it would be nice if we could add an input for testing that's not exposed in the UI (like show_user: false). We could expose it in the UI and have it disabled, but I think that would likely lead to confusion. https://github.com/elastic/package-spec/issues/97
• Golden file testing for system tests. This would be useful for detecting breaking changes in agent/collector/inputs.