Elsticsearch 6.3 Upgrade Broken (in Debian based Systems)

roock commented 6 years ago

Module version: 6.3.1
Puppet version: 4.10.12 and 5.5.4
OS and version: Debian 9 and Ubuntu 18.04

Bug description

Install any old version of elasticsearch before 6.3.0 e.g.

    include ::java

    class { 'elasticsearch':
      version => '6.2.0'
    }

    elasticsearch::instance { 'es-01': }

Then try to upgrade Elasticsearch to version 6.3.x. The upgrade will fail because it will try to create an initial keystore (this feature was introduced in version 6.3.0 https://github.com/elastic/elasticsearch/pull/28928 ). The creating of the keystore will fail because elasticsearch-keystore doesn't find ES_PATH_CONF and exists with an errorcode.

==> puppet-test01.local: ES_PATH_CONF must be set to the configuration path
==> puppet-test01.local: dpkg: error processing package elasticsearch (--configure):
==> puppet-test01.local:  installed elasticsearch package post-installation script subprocess returned error exit status 1

The problem does not happen if the initial version is already 6.3.x because then the keystore is created before ES_PATH_CONF is removed from /etc/default/elasticsearch.

archeious commented 6 years ago

Did you figure out a workaround for this? I have not been able to figure out how to proceed.

olevole commented 6 years ago

same issue.

I used to for work-around global env:

file_line { 'es-path-conf':
          path  => '/etc/environment',
          line  => "ES_PATH_CONF=/etc/elasticsearch",
          match => 'ES_PATH_CONF=',
}

tylerjl commented 6 years ago

Thank you for the report, I'm looking into potential solutions in this branch. This is another victim of the complications this module has to make in order to support the concept of instances, unfortunately (and another potential nail in the coffin of continuing to support multiple instances in the future.)

benedikt-haug commented 6 years ago

I also got the problem when upgrading from 6.3.0 => 6.4.0

Had to copy ES_PATH_CONF from /etc/default/elasticseach-$instance_name => /etc/default/elasticsearch

smortex commented 6 years ago

@tylerjl setting ES_PATH_CONF in /etc/environment seems to not do the trick:

$ sudo apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  elasticsearch-oss kibana-oss
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 110 MB of archives.
After this operation, 73.7 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 elasticsearch-oss all 6.4.1 [33.7 MB]
Get:2 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 kibana-oss amd64 6.4.1 [76.2 MB]
Fetched 110 MB in 7s (14.6 MB/s)                                                                                                                                                                                                            
(Reading database ... 237721 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.4.1_all.deb ...
Unpacking elasticsearch-oss (6.4.1) over (6.4.0) ...
Preparing to unpack .../kibana-oss_6.4.1_amd64.deb ...
Stopping kibana service... OK
Unpacking kibana-oss (6.4.1) over (6.4.0) ...
Setting up elasticsearch-oss (6.4.1) ...
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
ES_PATH_CONF must be set to the configuration path
dpkg: error processing package elasticsearch-oss (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for systemd (232-25+deb9u4) ...
Setting up kibana-oss (6.4.1) ...
Errors were encountered while processing:
 elasticsearch-oss
E: Sub-process /usr/bin/dpkg returned an error code (1)

Setting ES_PATH_CONF=/etc/elasticsearch/logs in /etc/default/elasticsearch (the actual path of my instance) may work, but apt ask for a confirmation I was not confident to accept: An elasticsearch keystore already exists. Overwrite? [y/N] :scream: .

Setting ES_PATH_CONF=/etc/elasticsearch in /etc/default/elasticsearch (the default path) make the update pass. The configuration is then somewhat broken (service does not start), but applying the Puppet catalog fixes everything.

tylerjl commented 6 years ago

@smortex out of curiosity, did you re-login/re-source your shell after making the change to /etc/elasticsearch before running apt? From what I can tell it seems to resolve some things on my end as well.

If it really doesn't work for some reason, we'll need to dramatically change some stuff with the module - such as supporting dedicated Elasticsearch instances at all - since setting ES_PATH_CONF in /etc/defaults/elasticsearch clobbers instance-specific ES_PATH_CONF values when invoking any CLI tools in the elasticsearch-env executable.

smortex commented 6 years ago

@tylerjl good point… I sudo-edited /etc/environment, and sudo-apt-upgraded elasticsearch… My regular-user shell was not reloaded after the change (so had not $ES_PATH_CONF set), but if it was, the sudo configuration would have unset $ES_PATH_CONF, so maybe this was a no-op in my workflow.

I assumed the unit would gather all information it needed (including sourcing /etc/environment), but I have not checked if it was actually doing so. Assuming is generally a good way to not find what is actually happening :confused:

roock commented 6 years ago

I just checked the postinstall script again and in 6.4.x, it looks like this: 6.4

# the equivalent code for rpm is in posttrans
if [ "$PACKAGE" = "deb" -a ! -f /etc/elasticsearch/elasticsearch.keystore ]; then
    /usr/share/elasticsearch/bin/elasticsearch-keystore create
    chown root:elasticsearch /etc/elasticsearch/elasticsearch.keystore
    chmod 660 /etc/elasticsearch/elasticsearch.keystore
    md5sum /etc/elasticsearch/elasticsearch.keystore > /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum
fi

It ssems that it would also be sufficient to have the puppet module create an empty file at /etc/elasticsearch/elasticsearch.keystore to fix the issue.

minorOffense commented 6 years ago

Just mentioning that when I run puppet the first time I do see this line here:

Notice: Augeas[/etc/sysconfig/elasticsearch](provider=augeas):
--- /etc/sysconfig/elasticsearch    2018-06-11 18:44:24.000000000 -0500
+++ /etc/sysconfig/elasticsearch.augnew 2018-10-03 13:12:32.590030428 -0500
@@ -9,8 +9,6 @@
 #JAVA_HOME=

 # Elasticsearch configuration directory
-ES_PATH_CONF=/etc/elasticsearch
-
 # Elasticsearch PID directory
 #PID_DIR=/var/run/elasticsearch

Looks like it's being removed explicitly.

Not sure if that's helpful.

smortex commented 6 years ago

Okay, new ElasticSearch update today, explicitly setting ES_PATH_CONF in the environment, and all went well.

# env ES_PATH_CONF=/etc/elasticsearch apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  elasticsearch-oss kibana-oss
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 110 MB of archives.
After this operation, 8192 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 elasticsearch-oss all 6.4.2 [33.7 MB]
Get:2 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 kibana-oss amd64 6.4.2 [76.2 MB]
Fetched 110 MB in 6s (17.4 MB/s)                                                                                                                                                                                                          
(Reading database ... 237851 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.4.2_all.deb ...
Unpacking elasticsearch-oss (6.4.2) over (6.4.1) ...
Preparing to unpack .../kibana-oss_6.4.2_amd64.deb ...
Stopping kibana service... OK
Unpacking kibana-oss (6.4.2) over (6.4.1) ...
Setting up elasticsearch-oss (6.4.2) ...
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for systemd (232-25+deb9u4) ...
Setting up kibana-oss (6.4.2) ...
#

This is probably the simplest workaround :tada:.

Puppet still fixes a lot of things after the update.

zeha commented 6 years ago

Are there any news on this? It's really annoying to run into this problem on each and every elasticsearch upgrade, and recently there have been more releases in a shorter time frame, so we're hitting this more often. The workaround ... works, but I don't see how I can persist it.

tkuther commented 6 years ago

Just mentioning that when I run puppet the first time I do see this line here:

Notice: Augeas[/etc/sysconfig/elasticsearch](provider=augeas):
--- /etc/sysconfig/elasticsearch  2018-06-11 18:44:24.000000000 -0500
+++ /etc/sysconfig/elasticsearch.augnew   2018-10-03 13:12:32.590030428 -0500
@@ -9,8 +9,6 @@
 #JAVA_HOME=

 # Elasticsearch configuration directory
-ES_PATH_CONF=/etc/elasticsearch
-
 # Elasticsearch PID directory
 #PID_DIR=/var/run/elasticsearch

Looks like it's being removed explicitly.

Not sure if that's helpful.

It seems to be the main cause of this trouble.

tylerjl commented 6 years ago

So here are the options to fix this that I can think of:

If anyone can confirm that making this change to /etc/environment resolves the issue, I can merge that fix it and release that to try and resolve it.
If option 1 doesn't work, we'll have to rewrite the module to drop support for running dedicated elasticsearch::instance resources entirely in a backwards-incompatible release (i.e., upgrading to the new Puppet module will likely not be possible.)

If anybody can provide feedback regarding whether setting /etc/environment via that Puppet change works - it seems to make things happy in my testing, but I'd like more confirmation - then we can potentially move forward with option 1) which will take significantly less time to fix.

tkuther commented 5 years ago

I can confirm that setting ES_PATH_CONF=/etc/elasticsearch in /etc/environment fixes installation/upgrade of the elasticsearch package.

But given other issues like #892 I'd vote for 2. in the long term.

kontinuity commented 5 years ago

If anybody can provide feedback regarding whether setting /etc/environment via that Puppet change

This indeed works as expected.

TuningYourCode commented 5 years ago

+1 for 1. which probably solves the problem. Today all our development / integration maschines failed at automatic upgrade to 6.6.0 because of this problem.

shamil commented 5 years ago

/etc/environment, doesn't work for me, this is what solved in my case

include ::elasticsearch

augeas { "add ${::elasticsearch::configdir} to defaults":
    incl    => "${::elasticsearch::defaults_location}/elasticsearch",
    lens    => 'Shellvars.lns',
    changes => "set ES_PATH_CONF ${::elasticsearch::configdir}"
}
-> Package[$::elasticsearch::package_name]

tinder-tder commented 5 years ago

it looks like any of these solutions of managing that variable get undone and redone resulting in a file bucket action each puppet run

Info: Applying configuration version '1548896438'
Notice: Augeas[/etc/default/elasticsearch update](provider=augeas): 
--- /etc/default/elasticsearch  2019-01-30 21:40:47.421206297 +0000
+++ /etc/default/elasticsearch.augnew   2019-01-31 01:00:42.209489373 +0000
@@ -47,3 +47,4 @@
 # When using Systemd, this setting is ignored and the 'vm.max_map_count'
 # property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
 #MAX_MAP_COUNT=262144
+ES_PATH_CONF=/etc/elasticsearch

Notice: /Stage[main]/Profiles::Elasticsearch::Metrics/Augeas[/etc/default/elasticsearch update]/returns: executed successfully
Notice: Augeas[/etc/default/elasticsearch](provider=augeas): 
--- /etc/default/elasticsearch  2019-01-31 01:00:42.269489396 +0000
+++ /etc/default/elasticsearch.augnew   2019-01-31 01:00:45.969490829 +0000
@@ -47,4 +47,3 @@
 # When using Systemd, this setting is ignored and the 'vm.max_map_count'
 # property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
 #MAX_MAP_COUNT=262144
-ES_PATH_CONF=/etc/elasticsearch

Notice: /Stage[main]/Elasticsearch::Config/Augeas[/etc/default/elasticsearch]/returns: executed successfully
Notice: Applied catalog in 4.89 seconds

shamil commented 5 years ago

@tinder-tder yep, it does this on each time catalog applied, because the module removes this setting, and in my workaround I add it.

I only care that when I upgrade ES, puppet will not fail, and this 2 changes each time are not really bother me.

Besides this file (/etc/default/elasticsearch) not really used, I guess it's just to make dpkg happy, there is defaults file per-instance which is what actually used and must not be patched...

Of course all these are not solutions, but workarounds/hacks. We should have proper solution. My bet is removing concept of instances is the best solution!

tylerjl commented 5 years ago

Based upon the feedback here, my hunch is that taking the approach of:

Adding the /etc/environment value as a short-term fix, and
Long-term, dropping support for multiple Elasticsearch instances

Is probably the best approach. I've discussed the potential of removing multiple-instance support in the module with people both inside and outside Elastic, and it seems like doing so is really the only way to keep the module highly stable and reliable for people (there's so many issues that it causes - this is just one, but out-of-date logging config files, jvm options, init scripts, and so many more are also plagued by this).

lvempati commented 5 years ago

For CentOS 7 RPM upgrade, noticed the same issue. Some research on ES_PATH_CONF : ES_PATH_CONF was introduced in 6.0. It was added to /etc/sysconfig/elasticsearch. RPM spec file defines this file as noreplace. So, upgrade from <6.0 ->6.3 created all the noreplace files are .rpmnew leaving the original file ( /etc/sysconfig/elasticsearch ) untouched. List of .rpmnew files warning: /etc/elasticsearch/elasticsearch.yml created as /etc/elasticsearch/elasticsearch.yml.rpmnew warning: /etc/elasticsearch/jvm.options created as /etc/elasticsearch/jvm.options.rpmnew warning: /etc/elasticsearch/log4j2.properties created as /etc/elasticsearch/log4j2.properties.rpmnew warning: /etc/init.d/elasticsearch created as /etc/init.d/elasticsearch.rpmnew warning: /etc/sysconfig/elasticsearch created as /etc/sysconfig/elasticsearch.rpmnew warning: /usr/lib/systemd/system/elasticsearch.service created as /usr/lib/systemd/system/elasticsearch.service.rpmnew

For 6.3, %posttrans section of RPM contains script to create /etc/elasticsearch/elasticsearch.keystore. This fails because it can't find ES_PATH_CONF. So, it prints the above message and quits.

Rebuild RPM and add a %post section scriptlet like this and installing newly build rpm works.
grep -q -x -F 'ES_PATH_CONF' /etc/sysconfig/elasticsearch || echo 'ES_PATH_CONF=/etc/elasticsearch' >> /etc/sysconfig/elasticsearch

seidler2547 commented 5 years ago

At least for 6.6 just

touch /etc/elasticsearch/elasticsearch.keystore

still worked fine for me, as reported in https://github.com/elastic/puppet-elasticsearch/issues/977#issuecomment-425073134

voxpupuli / puppet-elasticsearch

Elsticsearch 6.3 Upgrade Broken (in Debian based Systems) #977

Bug description