search

wazuh / wazuh-cloudformation

Wazuh - Amazon AWS Cloudformation
https://wazuh.com
40 stars 22 forks source link

Instance setup fails due to invalid JRE download URL #1

Closed r0yfire closed 5 years ago

r0yfire commented 5 years ago

Description

When creating a new stack using CloudFormation, the Kibana instance fails to run the setup script /tmp/wazuh_cf_kibana.sh. Upon further investigation I believe it fails to download the JRE package on line 37 elastic/wazuh_cf_kibana.sh

I assume the same problem exists with elastic/wazuh_cf_elastic.sh, though I did not test it.

Log example

[root@ip-10-0-1-9 ~]# cat /var/log/cfn-init.log 
2019-04-22 16:42:43,326 [INFO] -----------------------Starting build-----------------------
2019-04-22 16:42:43,327 [INFO] Running configSets: KibanaInstallationSet
2019-04-22 16:42:43,328 [INFO] Running configSet KibanaInstallationSet
2019-04-22 16:42:43,329 [INFO] Running config KibanaFiles
2019-04-22 16:42:43,431 [INFO] Running config KibanaInstall
2019-04-22 16:42:43,973 [ERROR] Command 01_RunInstallationScript (/tmp/wazuh_cf_kibana.sh) failed
2019-04-22 16:42:43,973 [ERROR] Error encountered during build of KibanaInstall: Command 01_RunInstallationScript failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 01_RunInstallationScript failed
2019-04-22 16:42:43,975 [ERROR] -----------------------BUILD FAILED!------------------------
2019-04-22 16:42:43,978 [ERROR] Unhandled exception during build: Command 01_RunInstallationScript failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 01_RunInstallationScript failed

Steps to test manually

url_jre="https://download.oracle.com/otn-pub/java/jdk/8u202-b08/1961070e4c9b4e26a04e7f5a083f551e/jre-8u202-linux-x64.rpm"
jre_rpm="/tmp/jre-8-linux-x64.rpm"
curl -Lo ${jre_rpm} --header "Cookie: oraclelicense=accept-securebackup-cookie" ${url_jre}
rpm -qlp /tmp/jre-8-linux-x64.rpm
snaow commented 5 years ago

Hi @0xF1,

You are right. We corrected our documentation a couple of days ago: https://github.com/wazuh/wazuh-documentation/commit/bbfdfee4d399cb5f634c37f889da03fad7311e28#diff-48202c8838fbf009b28d50af33cd263cR21

Oracle new policy does not allow direct download as it was before. We recommend using Java OpenJDK instead which is totally open and it's shipped in main distributions.

I have just modified master branch, let me know if the problem persists. Commit: https://github.com/wazuh/wazuh-cloudformation/commit/830f3bfab55d7f3538b7e648d721d75354f5c1a3

r0yfire commented 5 years ago

Thanks for the quick response and your contributions.

I tested the changes this morning, but found a new issue. The current Amazon Linux AMI comes with OpenJDK 1.7 pre-installed, which breaks the ES plugin install.

Solution:

Add yum -y remove java-1.7.0-openjdk before installing OpenJDK-1.8.0.

CF Log output:

[root@ip-10-0-1-43 ~]# cat /var/log/cfn-init.log 
2019-04-23 14:08:35,387 [INFO] -----------------------Starting build-----------------------
2019-04-23 14:08:35,388 [INFO] Running configSets: KibanaInstallationSet
2019-04-23 14:08:35,389 [INFO] Running configSet KibanaInstallationSet
2019-04-23 14:08:35,390 [INFO] Running config KibanaFiles
2019-04-23 14:08:35,470 [INFO] Running config KibanaInstall
2019-04-23 14:08:50,839 [ERROR] Command 01_RunInstallationScript (/tmp/wazuh_cf_kibana.sh) failed
2019-04-23 14:08:50,839 [ERROR] Error encountered during build of KibanaInstall: Command 01_RunInstallationScript failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 01_RunInstallationScript failed
2019-04-23 14:08:50,840 [ERROR] -----------------------BUILD FAILED!------------------------
2019-04-23 14:08:50,841 [ERROR] Unhandled exception during build: Command 01_RunInstallationScript failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command 01_RunInstallationScript failed

Running the setup script manually produces this error:

[root@ip-10-0-1-43 ~]# /tmp/wazuh_cf_kibana.sh 
<SNIP TEXT>
+ yum -y install java-1.8.0-openjdk
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                                                                                                                 | 2.1 kB  00:00:00     
amzn-updates                                                                                                                                              | 2.5 kB  00:00:00     
Package 1:java-1.8.0-openjdk-1.8.0.201.b09-0.43.amzn1.x86_64 already installed and latest version
Nothing to do
+ rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
++ echo 6.5.4
++ cut -d. -f1
+ elastic_major_version=6
+ cat
+ yum -y install elasticsearch-6.5.4
Loaded plugins: priorities, update-motd, upgrade-helper
elasticsearch-6.x                                                                                                                                         | 1.3 kB  00:00:00     
Package elasticsearch-6.5.4-1.noarch already installed and latest version
Nothing to do
+ chkconfig --add elasticsearch
+ /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch discovery-ec2
the minimum required Java version is 8; your Java version from [/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.211.x86_64/jre] does not meet this requirement

Bonus request The setup script fails to run when executed a second time on line 28 when creating the user. If you think it's appropriate, add a check like this: if ! id -u ${ssh_username} > /dev/null 2>&1; then adduser ${ssh_username}; fi

snaow commented 5 years ago

Hi @0xF1,

I am happy to help. Thanks for your contributions and feedback. I've sent a new commit with the fixes you mention: https://github.com/wazuh/wazuh-cloudformation/commit/02349f4798dc4a3373c3358225d6c6bc6da5faf0

Let me know if you need further changes. Consider as well send a PR with all the changes you need.

Thanks again for the feedback. Regards, Pedro.

manuasir commented 5 years ago

Hello @0xF1,

We just merged those changes to the master branch, so this issue will be closed. Feel free to open a new one whenever you may need it.

Thanks for this feedback.

Cheers