cookbook does not include support for xpack

[jb4x]

This cookbook doesn't seem to have included support for xpack as it exists in 5.0. Using the default recipe won't result in success if you've got a subscription and try this for 5.0. I ran into this when I realized that an instance using this cookbook had no role mapping.

@jjfalling asked to be added.

[martinb3]

Hi there -- if you ask this cookbook to install x-pack, it will install successfully. It's actually one of the tests we run against this cookbook. I'm not aware of any special "support for x-pack" that this cookbook would need, but we don't install it by default.

Are you saying that you'd like this cookbook to install x-pack by default? We're generally following the same defaults as the packages and archive files shipped by Elastic.

[praveenuchiha]

Hey, I have installed x-pack using this cookbook using the lines

elasticsearch_plugin 'x-pack' do
  notifies :restart, 'elasticsearch_service[elasticsearch]', :delayed
end

However i was not able to find role_mappings.yml file either in /usr/share/elasticsearch or in /etc/elasticsearch. Am i miising something or is this a bug?

[karmi]

@jjfalling, can you maybe provide more context and specifics?

[martinb3]

I went and tested this specifically on Ubuntu 16.04, with ES v5.0.1, and Chef 12.x, and I'm seeing the file at:

root@repository-ubuntu-1604:~# find /etc -type f | grep role_mapping.yml
/etc/elasticsearch/x-pack/role_mapping.yml

That's using this recipe -- note that we don't install x-pack by default, just like Elastic's packages and tarballs don't include it:

include_recipe 'elasticsearch::default'

elasticsearch_plugin 'x-pack' do
  notifies :restart, 'elasticsearch_service[elasticsearch]', :delayed
end

If you could provide some more information, such as the cookbook version and elasticsearch version, and potentially even the Chef version, I'd be happy to test those particulars again and be sure the plugin installs and the file is present.

[praveenuchiha]

Hi Martin,

Sorry for the late response. I have used v3.0.1 of the cookbook. And I upgraded the cluster from 2.2. We are using ubuntu 14.04 and chef 12.x. I also wanna mention that I see two elasticsearch folders under /usr/share (elasticsearch/ and /elasticsearch-5.0.5/) with the same content (files) in both of them. And this is the file structure of /etc/elasticsearch in all the nodes.

/etc/elasticsearch/
├── elasticsearch.yml
├── elasticsearch.yml.dpkg-dist
├── jvm.options
├── log4j2.properties
├── logging.yml
└── scripts

And this is the full recepie I have used.

`elasticsearch_user 'elasticsearch' do
  username 'elasticsearch'
  groupname 'elasticsearch'
  shell '/bin/bash'
  comment 'Elasticsearch User'

  action :create
end

elasticsearch_install 'my_es_installation' do
  type 'tarball' # type of install
  version '5.0.0'
  action :install # could be :remove as well
end

elasticsearch_configure 'elasticsearch' do
    allocated_memory '4g'
    path_data        "/data"
        path_conf        "/etc/elasticsearch"
        path_logs        "/var/log/elasticsearch"
    configuration ({
        'cluster.name' => 'cluster',
        'bootstrap.memory_lock' => true,
        'discovery.zen.ping.unicast.hosts' => ["node-1", "node-2", "node-3"],
        'network.bind_host' => '0.0.0.0',
        'network.publish_host' => '_eth0_',     
        'discovery.zen.minimum_master_nodes' => 2,
    })
end

elasticsearch_service 'elasticsearch' do
  service_actions [:enable, :restart]
end

elasticsearch_plugin 'x-pack' do
  notifies :restart, 'elasticsearch_service[elasticsearch]', :delayed
end`

Thank you for all your effort so far. Let me know if you are able to reproduce this on your end or if you need more details.

[martinb3]

Hi there -- thank you for the update with specifics. I'll try to address those as much as possible below.

I also wanna mention that I see two elasticsearch folders under /usr/share (elasticsearch/ and /elasticsearch-5.0.5/

One of these is a symlink to the other (since you're using the tarball install method). You should be able to see that with ls -la:

# ls -ld /usr/share/elasticsearch
lrwxrwxrwx 1 root root 30 Nov 28 17:39 /usr/share/elasticsearch -> /usr/share/elasticsearch-5.0.0

We are using ubuntu 14.04 and chef 12.x

I copied and pasted your example, and I see the various yml files at:

root@tarball-ubuntu-1404:/usr/share/elasticsearch# find /usr/share | grep role
/usr/share/elasticsearch-5.0.0/config/x-pack/users_roles
/usr/share/elasticsearch-5.0.0/config/x-pack/role_mapping.yml
/usr/share/elasticsearch-5.0.0/config/x-pack/roles.yml

It seems likely that the files aren't being found by Elasticsearch (you'd need to copy them to /etc/elasticsearch/x-pack/...), but this seems like a limitation of installation via tarball -- we don't currently set CONF_DIR or ES_HOME before running bin/elasticsearch-plugin, and until I went to look at it just now, wasn't even aware it respected those values.

I think the biggest root issue is that bin/elasticsearch-plugin doesn't source /etc/default/elasticsearch by default, but it needs to know if CONF_DIR or ES_HOME has been overridden.

I think the best thing to do is open an issue against upstream to ask about this. I'll do that now.

[praveenuchiha]

Hey, Thank you for the response. I have installed elastic on 3 different nodes in two different clusters using the same recepie. But couldnt find the roles.yml file. Here is the directory structure in all my nodes

/usr/share/elasticsearch/
├── ["
│   └── mnt
│       └── nfs"]
├── bin
│   ├── elasticsearch
│   ├── elasticsearch.bat
│   ├── elasticsearch.in.bat
│   ├── elasticsearch.in.sh
│   ├── elasticsearch-plugin
│   ├── elasticsearch-plugin.bat
│   ├── elasticsearch-service.bat
│   ├── elasticsearch-service-mgr.exe
│   ├── elasticsearch-service-x64.exe
│   ├── elasticsearch-service-x86.exe
│   ├── elasticsearch-systemd-pre-exec
│   ├── elasticsearch-translog
│   ├── elasticsearch-translog.bat
│   └── x-pack
│       ├── certgen
│       ├── certgen.bat
│       ├── croneval
│       ├── croneval.bat
│       ├── extension
│       ├── extension.bat
│       ├── migrate
│       ├── migrate.bat
│       ├── syskeygen
│       ├── syskeygen.bat
│       ├── users
│       └── users.bat
├── lib
│   ├── elasticsearch-5.0.0.jar
│   ├── HdrHistogram-2.1.6.jar
│   ├── hppc-0.7.1.jar
│   ├── jackson-core-2.8.1.jar
│   ├── jackson-dataformat-cbor-2.8.1.jar
│   ├── jackson-dataformat-smile-2.8.1.jar
│   ├── jackson-dataformat-yaml-2.8.1.jar
│   ├── jna-4.2.2.jar
│   ├── joda-convert-1.2.jar
│   ├── joda-time-2.9.4.jar
│   ├── jopt-simple-5.0.2.jar
│   ├── jts-1.13.jar
│   ├── log4j-1.2-api-2.6.2.jar
│   ├── log4j-api-2.6.2.jar
│   ├── log4j-core-2.6.2.jar
│   ├── lucene-analyzers-common-6.2.0.jar
│   ├── lucene-backward-codecs-6.2.0.jar
│   ├── lucene-core-6.2.0.jar
│   ├── lucene-grouping-6.2.0.jar
│   ├── lucene-highlighter-6.2.0.jar
│   ├── lucene-join-6.2.0.jar
│   ├── lucene-memory-6.2.0.jar
│   ├── lucene-misc-6.2.0.jar
│   ├── lucene-queries-6.2.0.jar
│   ├── lucene-queryparser-6.2.0.jar
│   ├── lucene-sandbox-6.2.0.jar
│   ├── lucene-spatial3d-6.2.0.jar
│   ├── lucene-spatial-6.2.0.jar
│   ├── lucene-spatial-extras-6.2.0.jar
│   ├── lucene-suggest-6.2.0.jar
│   ├── securesm-1.1.jar
│   ├── snakeyaml-1.15.jar
│   ├── spatial4j-0.6.jar
│   └── t-digest-3.0.jar
├── LICENSE.txt
├── modules
│   ├── aggs-matrix-stats
│   │   ├── aggs-matrix-stats-5.0.0.jar
│   │   └── plugin-descriptor.properties
│   ├── ingest-common
│   │   ├── ingest-common-5.0.0.jar
│   │   ├── jcodings-1.0.12.jar
│   │   ├── joni-2.1.6.jar
│   │   └── plugin-descriptor.properties
│   ├── lang-expression
│   │   ├── antlr4-runtime-4.5.1-1.jar
│   │   ├── asm-5.0.4.jar
│   │   ├── asm-commons-5.0.4.jar
│   │   ├── asm-tree-5.0.4.jar
│   │   ├── lang-expression-5.0.0.jar
│   │   ├── lucene-expressions-6.2.0.jar
│   │   ├── plugin-descriptor.properties
│   │   └── plugin-security.policy
│   ├── lang-groovy
│   │   ├── groovy-2.4.6-indy.jar
│   │   ├── lang-groovy-5.0.0.jar
│   │   ├── plugin-descriptor.properties
│   │   └── plugin-security.policy
│   ├── lang-mustache
│   │   ├── compiler-0.9.3.jar
│   │   ├── lang-mustache-5.0.0.jar
│   │   ├── plugin-descriptor.properties
│   │   └── plugin-security.policy
│   ├── lang-painless
│   │   ├── antlr4-runtime-4.5.1-1.jar
│   │   ├── asm-debug-all-5.1.jar
│   │   ├── lang-painless-5.0.0.jar
│   │   ├── plugin-descriptor.properties
│   │   └── plugin-security.policy
│   ├── percolator
│   │   ├── percolator-5.0.0.jar
│   │   └── plugin-descriptor.properties
│   ├── reindex
│   │   ├── commons-codec-1.10.jar
│   │   ├── commons-logging-1.1.3.jar
│   │   ├── httpasyncclient-4.1.2.jar
│   │   ├── httpclient-4.5.2.jar
│   │   ├── httpcore-4.4.5.jar
│   │   ├── httpcore-nio-4.4.5.jar
│   │   ├── plugin-descriptor.properties
│   │   ├── reindex-5.0.0.jar
│   │   └── rest-5.0.0.jar
│   ├── transport-netty3
│   │   ├── netty-3.10.6.Final.jar
│   │   ├── plugin-descriptor.properties
│   │   ├── plugin-security.policy
│   │   └── transport-netty3-5.0.0.jar
│   └── transport-netty4
│       ├── netty-buffer-4.1.5.Final.jar
│       ├── netty-codec-4.1.5.Final.jar
│       ├── netty-codec-http-4.1.5.Final.jar
│       ├── netty-common-4.1.5.Final.jar
│       ├── netty-handler-4.1.5.Final.jar
│       ├── netty-resolver-4.1.5.Final.jar
│       ├── netty-transport-4.1.5.Final.jar
│       ├── plugin-descriptor.properties
│       ├── plugin-security.policy
│       └── transport-netty4-5.0.0.jar
├── NOTICE.txt
├── plugins
│   ├── discovery-ec2
│   │   ├── aws-java-sdk-core-1.10.69.jar
│   │   ├── aws-java-sdk-ec2-1.10.69.jar
│   │   ├── commons-codec-1.10.jar
│   │   ├── commons-logging-1.1.3.jar
│   │   ├── discovery-ec2-5.0.0.jar
│   │   ├── httpclient-4.5.2.jar
│   │   ├── httpcore-4.4.5.jar
│   │   ├── jackson-annotations-2.5.0.jar
│   │   ├── jackson-databind-2.5.3.jar
│   │   ├── plugin-descriptor.properties
│   │   └── plugin-security.policy
│   └── x-pack
│       ├── activation-1.1.jar
│       ├── automaton-1.11-8.jar
│       ├── bcpkix-jdk15on-1.55.jar
│       ├── bcprov-jdk15on-1.55.jar
│       ├── commons-codec-1.10.jar
│       ├── commons-logging-1.1.3.jar
│       ├── guava-16.0.1.jar
│       ├── httpasyncclient-4.1.2.jar
│       ├── httpclient-4.5.2.jar
│       ├── httpcore-4.4.5.jar
│       ├── httpcore-nio-4.4.5.jar
│       ├── javax.mail-1.5.3.jar
│       ├── LICENSE.txt
│       ├── netty-3.10.6.Final.jar
│       ├── netty-buffer-4.1.5.Final.jar
│       ├── netty-codec-4.1.5.Final.jar
│       ├── netty-codec-http-4.1.5.Final.jar
│       ├── netty-common-4.1.5.Final.jar
│       ├── netty-handler-4.1.5.Final.jar
│       ├── netty-resolver-4.1.5.Final.jar
│       ├── netty-transport-4.1.5.Final.jar
│       ├── NOTICE.txt
│       ├── owasp-java-html-sanitizer-r239.jar
│       ├── plugin-descriptor.properties
│       ├── plugin-security.policy
│       ├── rest-5.0.0.jar
│       ├── sniffer-5.0.0.jar
│       ├── transport-netty3-5.0.0.jar
│       ├── transport-netty4-5.0.0.jar
│       ├── unboundid-ldapsdk-3.1.1.jar
│       └── x-pack-5.0.0.jar
└── README.textile

Am i doing something wrong? I dont know why these files doesn't get created in my nodes.

[martinb3]

At this point, I'm not sure if @jjfalling and @jb4x's issue is the same as the one @praveenuchiha is asking about. @praveenuchiha, can you open a separate issue, and we can move this thread there? @jjfalling can you provide some context based on my responses? I've shown that the mapping file does exist in my examples, and I'd like to understand your issue better.

[martinb3]

Hi folks -- please also give the latest release a try (v3.0.2) and see if it addresses your issue. bin/elasticsearch-plugin respects environment variables, but we weren't necessarily setting them (it's default guess was usually correct, but isn't now). We are setting them explicitly now.

[praveenuchiha]

Hi @martinb3 Thank you for the release. I also wanna address that @jb4x ,@jjfalling and me encountered this issue while working on the same project. So its the same problem. But if you want me to still open a new issue i can do that as well.

[martinb3]

I also wanna address that @jb4x ,@jjfalling and me encountered this issue while working on the same project

Oh, okay, no problem then -- we can continue in the same issue. I couldn't tell that from the comments :)

[praveenuchiha]

I tried to use package install for 5.0.1 and it fails everytime with an error

================================================================================
      Error executing action `install` on resource 'dpkg_package[/var/chef/cache/elasticsearch-5.0.1.deb]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received '1'
      ---- Begin output of dpkg -i /var/chef/cache/elasticsearch-5.0.1.deb ----
      STDOUT: (Reading database ... 145944 files and directories currently installed.)
      Preparing to unpack .../cache/elasticsearch-5.0.1.deb ...
      Unpacking elasticsearch (5.0.1) over (5.0.1) ...
      Setting up elasticsearch (5.0.1) ...
      Processing triggers for systemd (229-4ubuntu12) ...
      Processing triggers for ureadahead (0.100.0-19) ...
      STDERR: Configuration file '/etc/elasticsearch/elasticsearch.yml'
       ==> Deleted (by you or by a script) since installation.
       ==> Package distributor has shipped an updated version.
         What would you like to do about it ?  Your options are:
          Y or I  : install the package maintainer's version
          N or O  : keep your currently-installed version
            D     : show the differences between the versions
            Z     : start a shell to examine the situation
       The default action is to keep your current version.
      *** elasticsearch.yml (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package elasticsearch (--install):
       end of file on stdin at conffile prompt
      Errors were encountered while processing:
       elasticsearch
      ---- End output of dpkg -i /var/chef/cache/elasticsearch-5.0.1.deb ----
      Ran dpkg -i /var/chef/cache/elasticsearch-5.0.1.deb returned 1

      Cookbook Trace:
      ---------------
      /var/chef/cache/cookbooks/elasticsearch/libraries/provider_install.rb:121:in `install_package_wrapper_action'
      /var/chef/cache/cookbooks/elasticsearch/libraries/provider_install.rb:17:in `block in <class:InstallProvider>'

      Resource Declaration:
      ---------------------
      # In /var/chef/cache/cookbooks/elasticsearch/libraries/provider_install.rb

      110:               dpkg_package "#{Chef::Config[:file_cache_path]}/#{filename}" do
      111:                 options package_options
      112:                 action :nothing
      113:               end
      114:             else

      Compiled Resource:
      ------------------
      # Declared in /var/chef/cache/cookbooks/elasticsearch/libraries/provider_install.rb:110:in `install_package_wrapper_action'

      dpkg_package("/var/chef/cache/elasticsearch-5.0.1.deb") do
        package_name "/var/chef/cache/elasticsearch-5.0.1.deb"
        action [:nothing]
        retries 0
        retry_delay 2
        default_guard_interpreter :default
        declared_type :dpkg_package
        cookbook_name "elasticv5"
      end

      Platform:
      ---------
      x86_64-linux

Here's the only change i made from the previously posted cookbook .

elasticsearch_install 'my_es_installation' do
  type 'package' # type of install
  version "5.0.1"
  action :install # could be :remove as well
end

[martinb3]

@praveenuchiha This is a totally different / unrelated error, but you'll need to deal with the package manager if you don't specify a version to install -- it's going to try to upgrade every time and you already have config files the package manager won't overwrite. You also can't switch from using the tar ball to using the packages after the fact.

[martinb3]

It seems like this issue has three different discussions happening now:

1. What does "support for x-pack" mean
2. Does the cookbook allow installation of the x-pack plugin? (may or may not be the same as previous item)
3. Upgrading Elasticsearch in place via the package manager

I'm not really sure where we go from here. I'd love to get back to the first item and understand what the folks involved expect when "supporting x-pack," so I can understand if the cookbook does or doesn't address that.

[karmi]

As @martinb3 says, we do install X-Pack as part of our integration test suite, and the handling of plugin installation should be the same.

@jb4x, @praveenuchiha, can you please isolate the issue and try installing Elasticsearch with X-Pack based on the integration test mentioned above on a clean virtual machine, so can have a shared starting point?

[martinb3]

Hi folks -- I'd like to resolve this, to be sure we're doing everything needed by the OPs. Is there any update on examples we can use to understand the problem?

[praveenuchiha]

Hi Martin,

Sorry for the late response. I was running ES 2.x installed from the 2.4 cookbook version. And when i tried 3.x cookbook to upgrade to 5.x using package install method, it failed. So i used tarball method and it was successful (i thought it was). But then the above problem appeared (directory structures). Now that i know the 3.x cookbook doesn't support upgrade. i tried a fresh install with the same recipe and everything works flawlessly. I think you can go ahead and close this issue. And thank you for including the note about update in the Readme :)

[martinb3]

Thank you for the update @praveenuchiha! I'll go ahead and close this issue.