Error "Unable to determine retention policy"

[Dedofugaz]

🐛 Summary

I am running into an error at step 3.2.2 when running the deploy.sh script. The command I am running is: sudo ./deploy.sh install even after following steps on issue 19 (https://github.com/cisagov/LME/issues/19)

To reproduce

Steps to reproduce the behavior:

1. Have a clean install of Ubuntu 22.04.3 LTS
2. Run apt update and apt upgrade
3. Reboot
4. Run "lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv"
5. Then "resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv"
6. Reboot (just in case)
7. Then running steps on 3.2.2 (# Install Git client to be able to clone the LME repository sudo apt update sudo apt install git -y

   Download a copy of the LME files

   sudo git clone https://github.com/cisagov/lme.git /opt/lme/

   Change to the LME directory containing files for the Linux server

   cd /opt/lme/Chapter\ 3\ Files/

   Execute script with root privileges

   sudo ./deploy.sh install )

Expected behavior

I expected it should detect correctly the disk space and set a retention policy and proceed with install and show the passwords for access to the dashboard, seems like the process setups elastic and I can see the page but as I don't have any user and password I cannot use it.

Any helpful log output or screenshots

Paste the results here:

administrator@vmlme01:/opt/lme/Chapter 3 Files$ sudo ./deploy.sh install
Will execute the following intrusive actions:
        - apt update/upgrade
        - install docker (please uninstall before proceeding, or indicate skipping the install)
        - initialize docker swarm (execute `sudo docker swarm leave --force`  before proceeding if you are part of a swarm
        - automatic os updates via unattened-upgrades)
Proceed ([y]es/[n]o):y
[X] Updating OS software
Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:2 http://es.archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 http://es.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://es.archive.ubuntu.com/ubuntu jammy-backports InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[X] Installing prerequisites
Reading package lists...
Building dependency tree...
Reading state information...
net-tools is already the newest version (1.60+git20181103.0eebece-1ubuntu5).
zip is already the newest version (3.0-12build2).
curl is already the newest version (7.81.0-1ubuntu1.14).
curl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
This OS was detected as: ubuntu
[X] Configuring Auto Updates
Reading package lists...
Building dependency tree...
Reading state information...
unattended-upgrades is already the newest version (2.8ubuntu1).
unattended-upgrades set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Enter the IP of this Linux server: 172.16.xx.xx
Enter the Fully Qualified Domain Name (FQDN) of this Linux server. This needs to be resolvable from the Windows Event Collector: vmlme01.xxx.xxx
[X] Configuring winlogbeat config and certificates to use 172.16.xx.xx as the IP and vmlme01.xxx.xxx as the DNS
This script will use self signed certificates for communication and encryption. Do you want to continue with self signed certificates? ([y]es/[n]o): y
Skip Docker Install? ([y]es/[n]o): n
Do you have an old elastic user password? ([y]es/[n]o): n
[!] Note: Depending on your OpenSSL configuration you may see an error opening a .rnd file into RNG, this will not block the installation
[X] Making root Certificate Authority
[X] Signing root CA
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = Swarm
[X] Making Logstash certificate
[X] Signing logstash cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = vmlme01.ijarque.com
[X] Making Winlogbeat client certificate
[X] Signing wlbclient cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = wlbclient
[X] Making Elasticsearch certificate
[X] Sign elasticsearch cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = elasticsearch
[X] Making Kibana certificate
[X] Sign kibana cert
Certificate request self-signature ok
subject=C = US, ST = DC, L = Washington, O = CISA, CN = kibana
[X] Installing Docker
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c install -m 0755 -d /etc/apt/keyrings
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
+ sh -c chmod a+r /etc/apt/keyrings/docker.gpg
+ sh -c echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null
+ sh -c docker version
[X] Configuring Docker swarm
Swarm initialized: current node (gsznx0x0q5fccf74s4nyncrhl) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-1a5q6dbwujblpohkhdqeboxh7e2yttd1vh1ljkx172rkfl1wvd-edv6pu5n2oeldvtg6dg9uofji 172.16.xx.xx:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

[X] Adding certificates and keys to Docker
5izatgmr7qhbz1h4d3aobk2iq
bjcw7cbl3x4t4eagtzthuad3x
yi8msvau8t0p1tbwxkwlyetoy
gk5twe4ny6146pirtstlza1gb
0zguahmk5yyf3k7cqjztzp78d
xv5qer0y61t7zyi8rcu115ygb
dezgkymbhdeivdxovuqqe6tie
[X] Updating logstash configuration with logstash writer
blhag7qyeidsbciabcs5tw54y
[X] Creating custom logstash conf
xd492kw80q2zw5l0uzcr1gdfm
vm.max_map_count = 262144
Creating network lme_esnet
Creating service lme_logstash
Creating service lme_elasticsearch
Creating service lme_kibana
[X] Waiting for elasticsearch to be ready
[X] Setting elastic user password
{}
[X] Setting kibana system password
{}
[X] Setting logstash system password
{}
[X] Setting logstash writer role
{"role":{"created":true}}
[X] Setting dashboard update role
{"role":{"created":true}}
[X] Creating logstash writer user
{"created":true}
[X] Setting logstash writer password
{}
[X] Creating dashboard update user
{"created":true}
[X] Setting dashboard update user password
{}
[X] Configuring elasticsearch Replica settings
{"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"_na_","index":"[_all]"}],"type":"index_not_found_exception","reason":"no such index [[_all]]","index_uuid":"_na_","index":"[_all]"},"status":404}
[X] Generating files_for_windows zip
  adding: tmp/lme/ (stored 0%)
  adding: tmp/lme/wlbclient.crt (deflated 24%)
  adding: tmp/lme/wlbclient.key (deflated 24%)
  adding: tmp/lme/root-ca.crt (deflated 25%)
  adding: tmp/lme/winlogbeat.yml (deflated 52%)
test of /opt/lme/files_for_windows.zip OK

[X] Setting Elastic pipelines
{"acknowledged":true}[X] We think your main disk is 111G
[!] Unable to determine retention policy - exiting

Add any screenshots of the problem here.

[llwaterhouse]

Hello,

Our suspicion is that your disk size is too small. In our deployments, our disk size is 128Gb.

Please try a larger disk size and let us know. Thank you.

[Dedofugaz]

Seems like that did the trick... I don't understand though, on the documentation you say it requires 90G of disk for the application + OS, I know I had plenty of space for the OS, maybe it would be good to update the documentation specifing hardware requirements used for the tests. Thanks for the help!

[llwaterhouse]

Thanks for the suggestion, we'll take a look to see how we can update the documentation.

Thank you for responding.