Test Elastic 7.10.2 - Githubissues

dougburks commented 3 years ago

List of packages to be tested:

securityonion-elastic - 20190510-1ubuntu1securityonion135

List of Docker images to be tested:

Elasticsearch 7.10.2 (both OSS and Elastic Features versions)
Logstash 7.10.2 (both OSS and Elastic Features versions)
Kibana 7.10.2 (both OSS and Elastic Features versions)
Freqserver
Domainstats
ElastAlert
Curator

Please review the issues to be tested: https://github.com/Security-Onion-Solutions/security-onion/issues/1809 https://github.com/Security-Onion-Solutions/security-onion/issues/1810 https://github.com/Security-Onion-Solutions/security-onion/issues/1811 https://github.com/Security-Onion-Solutions/security-onion/issues/1812 https://github.com/Security-Onion-Solutions/security-onion/issues/1813 https://github.com/Security-Onion-Solutions/security-onion/issues/1817

An overview of the testing process can be found in the comments below.

Please record all testing results via comments on this issue.

Thanks in advance for your time and effort!

dougburks commented 3 years ago

How To Start Testing

install the 16.04.4.2 ISO image (has old Logstash templates which we need to test removal of): https://github.com/Security-Onion-Solutions/security-onion/blob/master/old/Verify_ISO_16.04.4.2.md
snapshot the VM if possible
run Setup
snapshot the VM if possible

add the test PPA:

sudo add-apt-repository -y ppa:securityonion/test

change DOCKERHUB from securityonionsolutions to securityonionsolutionstest (OSS license):
```
sudo sed -i 's|DOCKERHUB="securityonionsolutions"|DOCKERHUB="securityonionsolutionstest"|g' /etc/nsm/elasticdownload.conf
```
(OR change DOCKERHUB to securityonionsolutionselastest for Elastic Features license)
update:
```
sudo soup
```

dougburks commented 3 years ago

How To Verify Proper Elastic Operation

Please test in as many different combinations as possible:

verify that traditional Logstash parsing correctly parses Zeek logs in JSON or TSV format and Sysmon logs via Winlogbeat and Wazuh
verify that Elasticsearch Ingest parsing correctly parses Zeek logs in JSON format and Sysmon logs via Winlogbeat and Wazuh
verify that ElastAlert works properly
verify Kibana dashboards visualize those parsed logs correctly (for dashboards that have search hits, visualizations should show data and NO errors...for dashboards that have NO search hits, visualizations should show NO data and NO errors)
verify Squert and Logout links work properly
verify pivoting to Indicator dashboard
verify that each Kibana dashboard has a default query in the upper left
verify that lucene is still the default query language for both Dashboards and Discover
verify that you can now switch from dark mode to light mode via Kibana Advanced Settings and that the old dark and light scripts are gone
verify pivoting to CapMe works from all network data types
verify templates look correct
verify Curator close and delete work properly
check elasticsearch, logstash, and kibana logs in /var/log/ for anything out of the ordinary
so-import-pcap vs sosetup-minimal vs traditional Setup
Setup GUI vs CLI
Evaluation Mode vs Production Mode - when testing Evaluation Mode, make sure that Domainstats and Freqserver are generating data properly, here is a pcap that should generate data on the DomainStats dashboard: https://www.malware-traffic-analysis.net/2021/01/12/2021-01-12-Hancitor-infection-with-Cobalt-Strike.pcap.zip
standalone vs distributed deployments
new installation vs upgrade
Elastic OSS vs Elastic Features license (use so-elastic-features to switch from OSS to Features)
SSO vs Elastic native auth (use so-elastic-auth to switch to Elastic native auth)
fully test all features in Kibana (both OSS and Features) to make sure we've got all the new URLs that Kibana added
test upgrading a machine that already has Elastic native auth enabled
test upgrading to Elastic 7.10.2 and then doing a full upgrade to Security Onion 2
16.04.4.2 includes the old Logstash templates. When you install this update, it should automatically remove those old Logstash templates so new logs should come in without having to manually remove old templates.

Please make sure so-curator-closed-delete-delete gets tested thoroughly in at least the following scenarios:

if we haven't reached LOG_SIZE_LIMIT, it should do nothing
if we have reached LOG_SIZE_LIMIT but there are no closed indices, it should do nothing
if there are closed indices but we haven't reached LOG_SIZE_LIMIT, it should do thing
if we have reached LOG_SIZE_LIMIT and there are closed indices, it should delete closed indices until we are below LOG_SIZE_LIMIT or there are no more closed indices

cm-ops commented 3 years ago

All testing conducted using the above guidance.

verify that traditional Logstash parsing correctly parses Zeek logs in JSON or TSV format and Sysmon logs via Winlogbeat and Wazuh - No issues
verify that Elasticsearch Ingest parsing correctly parses Zeek logs in JSON format and Sysmon logs via Winlogbeat and Wazuh - No issues
verify that ElastAlert works properly - No issues
verify Kibana dashboards visualize those parsed logs correctly - No issues
verify Squert and Logout links work properly - No issues
verify pivoting to Indicator dashboard - No issues
verify that each Kibana dashboard has a default query in the upper left - No issues
verify that lucene is still the default query language for both Dashboards and Discover - No issues
verify that you can now switch from dark mode to light mode via Kibana Advanced Settings and that the old dark and light scripts are gone - No issues
verify pivoting to CapMe works from all network data types - No issues
verify templates look correct - No issues
check elasticsearch, logstash, and kibana logs in /var/log/ for anything out of the ordinary - No issues
so-import-pcap vs traditional Setup - No issues
Setup GUI vs CLI - No issues
Evaluation Mode vs Production Mode - No issues
standalone vs distributed deployments - No issues
Elastic OSS vs Elastic Features license (use so-elastic-features to switch from OSS to Features) - No issues
SSO vs Elastic native auth (use so-elastic-auth to switch to Elastic native auth) - No issues
fully test all features in Kibana (both OSS and Features) to make sure we've got all the new URLs that Kibana added - No issues
test upgrading to Elastic 7.10.2 and then doing a full upgrade to Security Onion 2 - No issues (Standalone and Distributed)
16.04.4.2 includes the old Logstash templates. When you install this update, it should automatically remove those old Logstash templates so new logs should come in without having to manually remove old templates. - No issues
verify Curator close and delete work properly - No issues

Please make sure so-curator-closed-delete-delete gets tested thoroughly in at least the following scenarios:

if we haven't reached LOG_SIZE_LIMIT, it should do nothing - No issues
if we have reached LOG_SIZE_LIMIT but there are no closed indices, it should do nothing - No issues
if there are closed indices but we haven't reached LOG_SIZE_LIMIT, it should do nothing - No issues
if we have reached LOG_SIZE_LIMIT and there are closed indices, it should delete closed indices until we are below LOG_SIZE_LIMIT or there are no more closed indices - No issues

dougburks commented 3 years ago

Published: https://blog.securityonion.net/2021/02/elastic-stack-7102-now-available-for.html

Security-Onion-Solutions / security-onion

Test Elastic 7.10.2 #1816