search

clong / DetectionLab

Automate the creation of a lab environment complete with security tooling and logging best practices
MIT License
4.6k stars 980 forks source link

Error building DetectionLab via WSL2 #692

Closed benheise closed 3 years ago

benheise commented 3 years ago

Description of the issue:

While using WSL2 and building the logger host, I'm running into the following error message that causes the build to stop.

The first is that the bash script copied over to the logger host has windows control characters in it for some reason.

TASK [logger : Install Splunk] **********************************************************************************************************************************************************************************
fatal: [192.168.1.68]: FAILED! => {"changed": false, "cmd": "# Get a free Maxmind license here: https://www.maxmind.com/en/geolite2/signup\n# Required for the ASNgen app to work: https://splunkbase.splunk.com/app/3531/\nexport MAXMIND_LICENSE=\nif [ -z $MAXMIND_LICENSE ]; then\n  echo \"Note: You have not entered a MaxMind license key in the 'Install Splunk' Ansible task, so the ASNgen Splunk app may not work correctly.\"\n  echo \"However, it is not required and everything else should function correctly.\"\nfi\n\n# Check if Splunk is already installed\nif [ -f \"/opt/splunk/bin/splunk\" ]; then\n  echo \"[$(date +%H:%M:%S)]: Splunk is already installed\"\nelse\n  echo \"[$(date +%H:%M:%S)]: Installing Splunk...\"\n  # Get download.splunk.com into the DNS cache. Sometimes resolution randomly fails during wget below\n  dig @8.8.8.8 download.splunk.com >/dev/null\n  dig @8.8.8.8 splunk.com >/dev/null\n  dig @8.8.8.8 www.splunk.com >/dev/null\n\n  # Try to resolve the latest version of Splunk by parsing the HTML on the downloads page\n  echo \"[$(date +%H:%M:%S)]: Attempting to autoresolve the latest version of Splunk...\"\n  LATEST_SPLUNK=$(curl https://www.splunk.com/en_us/download/splunk-enterprise.html | grep -i deb | grep -Eo \"data-link=\\\"................................................................................................................................\" | cut -d '\"' -f 2)\n  # Sanity check what was returned from the auto-parse attempt\n  if [[ \"$(echo $LATEST_SPLUNK | grep -c \"^https:\")\" -eq 1 ]] && [[ \"$(echo $LATEST_SPLUNK | grep -c \"\\.deb$\")\" -eq 1 ]]; then\n    echo \"[$(date +%H:%M:%S)]: The URL to the latest Splunk version was automatically resolved as: $LATEST_SPLUNK\"\n    echo \"[$(date +%H:%M:%S)]: Attempting to download...\"\n    wget --progress=bar:force -P /opt \"$LATEST_SPLUNK\"\n  else\n    echo \"[$(date +%H:%M:%S)]: Unable to auto-resolve the latest Splunk version. Falling back to hardcoded URL...\"\n    # Download Hardcoded Splunk\n    wget --progress=bar:force -O /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb 'https://download.splunk.com/products/splunk/releases/8.0.2/linux/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb&wget=true'\n  fi\n  if ! ls /opt/splunk*.deb 1> /dev/null 2>&1; then\n    echo \"Something went wrong while trying to download Splunk. This script cannot continue. Exiting.\"\n    exit 1\n  fi\n  if ! dpkg -i /opt/splunk*.deb > /dev/null; then\n    echo \"Something went wrong while trying to install Splunk. This script cannot continue. Exiting.\"\n    exit 1\n  fi\n  /opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt --seed-passwd changeme\n  /opt/splunk/bin/splunk add index wineventlog -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index osquery -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index osquery-status -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index sysmon -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index powershell -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index zeek -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index suricata -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index threathunting -auth 'admin:changeme'\n  /opt/splunk/bin/splunk add index evtx_attack_samples -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_forwarder/splunk-add-on-for-microsoft-windows_700.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/splunk-add-on-for-microsoft-sysmon_1062.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/asn-lookup-generator_110.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/lookup-file-editor_331.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/splunk-add-on-for-zeek-aka-bro_400.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/force-directed-app-for-splunk_200.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/punchcard-custom-visualization_130.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/sankey-diagram-custom-visualization_130.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/link-analysis-app-for-splunk_161.tgz -auth 'admin:changeme'\n  /opt/splunk/bin/splunk install app /vagrant/resources/splunk_server/threathunting_1492.tgz -auth 'admin:changeme'\n  \n  # Fix ASNGen App - https://github.com/doksu/TA-asngen/issues/18#issuecomment-685691630\n  echo 'python.version = python2' >> /opt/splunk/etc/apps/TA-asngen/default/commands.conf\n\n  # Install the Maxmind license key for the ASNgen App\n  if [ ! -z $MAXMIND_LICENSE ]; then\n    mkdir /opt/splunk/etc/apps/TA-asngen/local \n    cp /opt/splunk/etc/apps/TA-asngen/default/asngen.conf /opt/splunk/etc/apps/TA-asngen/local/asngen.conf\n    sed -i \"s/license_key =/license_key = $MAXMIND_LICENSE/g\" /opt/splunk/etc/apps/TA-asngen/local/asngen.conf\n  fi\n\n  # Install a Splunk license if it was provided\n  if [ -n $BASE64_ENCODED_SPLUNK_LICENSE ]; then\n    echo \"$BASE64_ENCODED_SPLUNK_LICENSE\" | base64 -d > /tmp/Splunk.License\n    /opt/splunk/bin/splunk add licenses /tmp/Splunk.License\n    rm /tmp/Splunk.License\n  fi\n\n  # Replace the props.conf for Sysmon TA and Windows TA\n  # Removed all the 'rename = xmlwineventlog' directives\n  # I know youre not supposed to modify files in \"default\",\n  # but for some reason adding them to \"local\" wasnt working\n  cp /vagrant/resources/splunk_server/windows_ta_props.conf /opt/splunk/etc/apps/Splunk_TA_windows/default/props.conf\n  cp /vagrant/resources/splunk_server/sysmon_ta_props.conf /opt/splunk/etc/apps/TA-microsoft-sysmon/default/props.conf\n\n  # Add custom Macro definitions for ThreatHunting App\n  cp /vagrant/resources/splunk_server/macros.conf /opt/splunk/etc/apps/ThreatHunting/default/macros.conf\n  # Fix props.conf in ThreatHunting App\n  sed -i 's/EVAL-host_fqdn = Computer/EVAL-host_fqdn = ComputerName/g' /opt/splunk/etc/apps/ThreatHunting/default/props.conf\n  # Fix Windows TA macros\n  mkdir /opt/splunk/etc/apps/Splunk_TA_windows/local\n  cp /opt/splunk/etc/apps/Splunk_TA_windows/default/macros.conf /opt/splunk/etc/apps/Splunk_TA_windows/local\n  sed -i 's/wineventlog_windows/wineventlog/g' /opt/splunk/etc/apps/Splunk_TA_windows/local/macros.conf\n  # Fix Force Directed App until 2.0.1 is released (https://answers.splunk.com/answers/668959/invalid-key-in-stanza-default-value-light.html#answer-669418)\n  rm /opt/splunk/etc/apps/force_directed_viz/default/savedsearches.conf\n\n  # Add a Splunk TCP input on port 9997\n  echo -e \"[splunktcp://9997]\\nconnection_host = ip\" >/opt/splunk/etc/apps/search/local/inputs.conf\n  # Add props.conf and transforms.conf\n  cp /vagrant/resources/splunk_server/props.conf /opt/splunk/etc/apps/search/local/\n  cp /vagrant/resources/splunk_server/transforms.conf /opt/splunk/etc/apps/search/local/\n  cp /opt/splunk/etc/system/default/limits.conf /opt/splunk/etc/system/local/limits.conf\n  # Bump the memtable limits to allow for the ASN lookup table\n  sed -i.bak 's/max_memtable_bytes = 10000000/max_memtable_bytes = 30000000/g' /opt/splunk/etc/system/local/limits.conf\n\n  # Skip Splunk Tour and Change Password Dialog\n  echo \"[$(date +%H:%M:%S)]: Disabling the Splunk tour prompt...\"\n  touch /opt/splunk/etc/.ui_login\n  mkdir -p /opt/splunk/etc/users/admin/search/local\n  echo -e \"[search-tour]\\nviewed = 1\" >/opt/splunk/etc/system/local/ui-tour.conf\n  # Source: https://answers.splunk.com/answers/660728/how-to-disable-the-modal-pop-up-help-us-to-improve.html\n  if [ ! -d \"/opt/splunk/etc/users/admin/user-prefs/local\" ]; then\n    mkdir -p \"/opt/splunk/etc/users/admin/user-prefs/local\"\n  fi\n  echo '[general]\n  render_version_messages = 1\n  dismissedInstrumentationOptInVersion = 4\n  notification_python_3_impact = false\n  display.page.home.dashboardId = /servicesNS/nobody/search/data/ui/views/logger_dashboard' > /opt/splunk/etc/users/admin/user-prefs/local/user-prefs.conf\n  # Enable SSL Login for Splunk\n  echo -e \"[settings]\\nenableSplunkWebSSL = true\" >/opt/splunk/etc/system/local/web.conf\n  # Copy over the Logger Dashboard\n  if [ ! -d \"/opt/splunk/etc/apps/search/local/data/ui/views\" ]; then\n    mkdir -p \"/opt/splunk/etc/apps/search/local/data/ui/views\"\n  fi\n  cp /vagrant/resources/splunk_server/logger_dashboard.xml /opt/splunk/etc/apps/search/local/data/ui/views || echo \"Unable to find dashboard\"\n  # Reboot Splunk to make changes take effect\n    /opt/splunk/bin/splunk restart\n    /opt/splunk/bin/splunk enable boot-start\n  fi\n", "delta": "0:00:00.474273", "end": "2021-08-10 01:06:28.729666", "msg": "non-zero return code", "rc": 1, "start": "2021-08-10 01:06:28.255393", "stderr": "/bin/bash: line 20: curl: command not found\n--2021-08-10 01:06:28--  https://download.splunk.com/products/splunk/releases/8.0.2/linux/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb&wget=true\nResolving download.splunk.com (download.splunk.com)... 13.225.206.15, 13.225.206.79, 13.225.206.41, ...\nConnecting to download.splunk.com (download.splunk.com)|13.225.206.15|:443... connected.\nHTTP request sent, awaiting response... 404 Not Found\n2021-08-10 01:06:28 ERROR 404: Not Found.\n\ndpkg-deb: error: '/opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb' is not a Debian format archive\ndpkg: error processing archive /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb (--install):\n dpkg-deb --control subprocess returned error exit status 2\nErrors were encountered while processing:\n /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb", "stderr_lines": ["/bin/bash: line 20: curl: command not found", "--2021-08-10 01:06:28--  https://download.splunk.com/products/splunk/releases/8.0.2/linux/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb&wget=true", "Resolving download.splunk.com (download.splunk.com)... 13.225.206.15, 13.225.206.79, 13.225.206.41, ...", "Connecting to download.splunk.com (download.splunk.com)|13.225.206.15|:443... connected.", "HTTP request sent, awaiting response... 404 Not Found", "2021-08-10 01:06:28 ERROR 404: Not Found.", "", "dpkg-deb: error: '/opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb' is not a Debian format archive", "dpkg: error processing archive /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb (--install):", " dpkg-deb --control subprocess returned error exit status 2", "Errors were encountered while processing:", " /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb"], "stdout": "Note: You have not entered a MaxMind license key in the 'Install Splunk' Ansible task, so the ASNgen Splunk app may not work correctly.\nHowever, it is not required and everything else should function correctly.\n[01:06:28]: Installing Splunk...\n[01:06:28]: Attempting to autoresolve the latest version of Splunk...\n[01:06:28]: Unable to auto-resolve the latest Splunk version. Falling back to hardcoded URL...\nSomething went wrong while trying to install Splunk. This script cannot continue. Exiting.", "stdout_lines": ["Note: You have not entered a MaxMind license key in the 'Install Splunk' Ansible task, so the ASNgen Splunk app may not work correctly.", "However, it is not required and everything else should function correctly.", "[01:06:28]: Installing Splunk...", "[01:06:28]: Attempting to autoresolve the latest version of Splunk...", "[01:06:28]: Unable to auto-resolve the latest Splunk version. Falling back to hardcoded URL...", "Something went wrong while trying to install Splunk. This script cannot continue. Exiting."]}

Installing and running dos2unix resolved this issue.

The second issue is that the script appears to correctly parse the splunk .deb for installation, however the file downloaded is empty.

vagrant@logger:~$ sudo ./logger_bootstrap.sh
Unable to locate logger_variables.sh
Note: You have not entered a MaxMind API key in logger_variables.sh, so the ASNgen Splunk app may not work correctly.
However, it is optional and everything else should function correctly.
[01:15:19]: Adding apt repositories...
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
<snip...>
Get:54 http://ubuntu.securedservers.com bionic-updates/multiverse Translation-en [6,988 B]
Fetched 40.0 MB in 7s (5,491 kB/s)
Reading package lists... Done
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
<snip...>
Get:12 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 418 kB in 1s (349 kB/s)
Reading package lists... Done
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
<snip...>
Get:7 http://mirrors.lug.mtu.edu/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 330 kB in 1min 1s (5,405 B/s)
Reading package lists... Done
[01:16:41]: Running apt-get clean...
[01:16:41]: Running apt-get update...
[01:16:45]: Running apt-fast install...
[01:16:47]: Updating the MOTD...
[01:16:47]: [TEST] Validating that jq is correctly installed...
[+] jq was successfully installed!
[01:16:47]: [TEST] Validating that whois is correctly installed...
[+] whois was successfully installed!
[01:16:47]: [TEST] Validating that build-essential is correctly installed...
[+] build-essential was successfully installed!
[01:16:47]: [TEST] Validating that git is correctly installed...
[+] git was successfully installed!
[01:16:47]: [TEST] Validating that unzip is correctly installed...
[+] unzip was successfully installed!
[01:16:47]: [TEST] Validating that yq is correctly installed...
[+] yq was successfully installed!
[01:16:47]: [TEST] Validating that mysql-server is correctly installed...
[+] mysql-server was successfully installed!
[01:16:47]: [TEST] Validating that redis-server is correctly installed...
[+] redis-server was successfully installed!
[01:16:47]: [TEST] Validating that python-pip is correctly installed...
[+] python-pip was successfully installed!
sudo: unable to resolve host logger
[01:16:47]: Installing Splunk...
[01:16:52]: Attempting to autoresolve the latest version of Splunk...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  190k    0  190k    0     0   848k      0 --:--:-- --:--:-- --:--:--  852k
[01:16:53]: The URL to the latest Splunk version was automatically resolved as: https://d7wz6hmoaavd0.cloudfront.net/products/splunk/releases/8.2.1/linux/splunk-8.2.1-ddff1c41e5cf-linux-2.6-amd64.deb
[01:16:53]: Attempting to download...
--2021-08-10 01:16:53--  https://d7wz6hmoaavd0.cloudfront.net/products/splunk/releases/8.2.1/linux/splunk-8.2.1-ddff1c41e5cf-linux-2.6-amd64.deb
Resolving d7wz6hmoaavd0.cloudfront.net (d7wz6hmoaavd0.cloudfront.net)... 13.225.71.84, 13.225.71.215, 13.225.71.11, ...
Connecting to d7wz6hmoaavd0.cloudfront.net (d7wz6hmoaavd0.cloudfront.net)|13.225.71.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 437791450 (418M) [binary/octet-stream]
Saving to: ‘/opt/splunk-8.2.1-ddff1c41e5cf-linux-2.6-amd64.deb’

splunk-8.2.1-ddff1c41e5cf-linux-2.6-amd64.deb        100%[===================================================================================================================>] 417.51M  81.6MB/s    in 5.3s

2021-08-10 01:16:58 (79.3 MB/s) - ‘/opt/splunk-8.2.1-ddff1c41e5cf-linux-2.6-amd64.deb’ saved [437791450/437791450]

dpkg-deb: error: '/opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb' is not a Debian format archive
dpkg: error processing archive /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb (--install):
 dpkg-deb --control subprocess returned error exit status 2

Errors were encountered while processing:
 /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb
Something went wrong while trying to install Splunk. This script cannot continue. Exiting.
vagrant@logger:~$ file /opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb
/opt/splunk-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb: empty

Manually running the splunk installer .deb and supplying the default username/password completed this step and allowed Ansible to move on to the next steps.

clong commented 3 years ago

Hey @benheise, sorry for the delay here. Is the Splunk issue repeatable? Maybe it was a corrupted download or an intermittent issue with Splunk servers at the time?

Also, I can't repro the windows control characters issue (but I'm running this on a Mac):

$ md5sum logger_bootstrap.sh
82fc33098b4f20e55df55561b1056b9e  logger_bootstrap.sh
$ dos2unix logger_bootstrap.sh
dos2unix: converting file logger_bootstrap.sh to Unix format...
$ md5sum logger_bootstrap.sh
82fc33098b4f20e55df55561b1056b9e  logger_bootstrap.sh

Maybe you opened the file in a Windows text editor at some point which caused Windows control chars to get added?

benheise commented 3 years ago

I didn't think so, but I'm not running into that same issue so that must be the case.