confusing config directories when using the Shield plugin

[babadofar]

I'm not sure whether this issue mostly belongs to Shield or the the puppet-elasticsearch repo. My problem relates to using the esusers authentication realm, and problem with finding the shield configuration directory. The puppet module creates a directory for each instance, and this seems to come into conflict with esusers auth.

Installing the license and shield plugin without any additional options creates a "shield" folder straight below /etc/elasticsearch. Using the esusers command line tool also operates on the files in the /etc/elasticsearch folder. However, when I try to log in as the user, that location is no longer valid, at least not when it comes to checking roles. ( I can log in, but get a message about not valid permissions or something)

My solution has been to move the "shield" folder into the instance folder, (ie /etc/elasticsearch/es-01/shield) and to change the CONF_DIR option in /etc/defaults to /etc/elasticsearch/es-01

You can test it out by cloning this repo: https://github.com/comperiosearch/vagrant-elk-box vagrant up and

vagrant ssh 
sudo /usr/share/elasticsearch/bin/plugin -i elasticsearch/license/latest  
sudo /usr/share/elasticsearch/bin/plugin -i elasticsearch/shield/latest  
sudo elasticsearch restart
sudo /usr/share/elasticsearch/bin/shield/esusers useradd admin01  -r admin

curl http://localhost:9200/_cat/nodes -u admin01

moving the shield directory from /etc/elasticsearch to /etc/elasticsearch/es-01 and modifiying CONF_DIR in /etc/defaults/elasticsearch to /etc/elasticsearch/es-01 fixes the problem

Any ideas how to fix this properly?

[electrical]

Hi,

There are still some problems with regards to the shield plugin indeed that i am trying to solve. I'm in the progress of building a separate shield plugin that hopefully will handle all aspects of shield management. The main problem is that the files need to be shared between the different instances, which is bit of a pain using the esusers binary if only used once.

[bilsch]

@electrical how far along are you on this and #399 - I just created a fork and want to create defined types for esusers user/role management in puppet. I don't want to create conflicting work. My plan is to get this going and at least get a prelim pr up today or tomorrow. Also happy to work off of your notes/design ideas

[electrical]

@bilsch i haven't done any solid work yet on the user/role management so far. only on ssl cert generation for shield. I'm planning to build a separate elasticsearch-shield module for it though. Do you have the fork somewhere i can check out?

Cheers.

[bilsch]

@electrical I have only begun working on a basic module. I was originally going to create this as an extension to the puppet-elasticsearch module.

I'd be happy to collaborate on a separate shield module if you would be interested. if you want to - create a new empty repo in the elastic org with some design ideas and your key code and I can take a stab at it implementation.

I'm not sure we will be in a position to own the module out-right ( would have to ask others just not sure )

[msimos]

Also need to add documentation about setting -Des.path.conf=/etc/elasticsearch to the plugin command so the Shield install works properly when using RPM or DEB install of Elasticsearch.

[electrical]

@msimos that specific issue should be solved already with version 0.9.7 and up of the module.

[fgimian]

@electrical Hey there Richard, I'm afraid that we are still having an issue with this.

  elasticsearch::instance { 'client-node':
  ...
  }

  elasticsearch::plugin { 'license':
    url       => "http://mydomain.com/pub/elasticsearch/license-2.3.1.zip",
    instances => 'marvel',
  }

  elasticsearch::plugin { 'shield':
    url       => "http://mydomain.com/pub/elasticsearch/shield-2.3.1.zip",
    instances => 'marvel',
  }

However, when I run the esusers command that ships with Shield, it creates configuration under /etc/elasticsearch/shield instead of /etc/elasticsearch/client-node/shield.

Any help would be greatly appreciated :smile: Fotis

[tylerjl]

Hi @fgimian, Richard has moved on from this module so I (along with other Elastic employees) are working on this.

Fully supported Shield resources are in progress in my feaure branch which you can preview. The tl;dr for Shield esusers operations is that I've altered the plugin resource to manage plugin files under /usr/share/elasticsearch. You can then pass --default.path.conf to the esusers executable to point to that shield directory and use puppet to keep the /usr/share/elasticsearch/shield and /etc/elasticsearch/$instance/shield directories in sync.

The branch is in a working state but under heavy development. Keeping the shield configs in /usr and copying to /etc seems weird at first, but it 1) is more compatible than hard linking, 2) simplifies multi-instance deployments and 3) preserves default role settings shipped with the shield install.

[fgimian]

Thank you so much for the tip about --default.path.conf!! I was trying to figure out how to override the config directory for esusers and couldn't find anything. 😄

[tylerjl]

This should be resolved with the 0.11.0 release; tl;dr the /usr/share/elasticsearch/shield directory is synced between instances since symlinking is unsupported in the java security manager.

[fgimian]

Thanks mate 😄

[sandeepkanabar]

Hi @tylerjl , I use puppet to setup elasticsearch and shield. In our case, our config directory is /data/elasticsearch/conf. So I copied the shield folder from /etc/elasticsearch/shield folder to /data/elasticsearch/conf/ and all worked well. But say, if I add a user, then I again need to RE-COPY the folder. Isn't there a way so that I can instruct shield to use /data/elasticsearch/conf/shield rather than /etc/elasticsearch/shield?

[sandeepkanabar]

Ah bummer! added --default.path.conf=/data/elasticsearch/conf and worked like a charm. Is it necessary that I need to copy the shield files from /data/elasticsearch/conf/shield to /etc/elasticsearch/shield and that they should be in sync? Any harm if the the /etc/elasticsearch/shield doesn't contain the new users I added using --default.path.conf?

[tylerjl]

So I not 100% certain what your setup is try to achieve - if you're using the puppet module resources, there really shouldn't be a need to invoke the esusers command or manage directories, so I'll just summarize the way things currently work in the module:

• The esusers command points to /usr/share/elasticsearch/shield to perform all operations/manage users/and so on.
• A file resource ensures that /usr/share/elasticsearch/shield is kept in-sync with elasticsearch::instance resources, basically like so: rsync /usr/share/elasticsearch/shield /etc/elasticsearch/es-01/shield
• That order is enforced every puppet run so that any users or roles that are changed get propagated to all instances.

If you're manually calling esusers and pointing at directories that aren't managed by puppet, you just need to make sure that individual instance config directories have the necessary shield configuration files - because the module sets up separate instance directories for each Elasticsearch service, configuration files in either /etc/elasticsearch/shield or /usr/share/elasticsearch/shield aren't really live/actively applied; only once they're placed into a service's configuration directory does shield pick them up.

[sandeepkanabar]

Hi @tylerjl , I agree that with proper usage of puppet, I don't need to cp the directories. I'm refactoring puppet code (learning puppet along the way) and will replace that with file resource as you suggested. Thank you!

I was a tad surprised when you said why I'm invoking 'esusers' manually. It turns out, that when I downloaded the elasticsearch module from puppetforge in 1st week of May, the latest version was 0.10.3 and shield support was implemented from next version onwards. Thank you for indirectly pointing it out. I'll download the latest version and do away with the esusers command.

By the way, I'm using puppet to set up Elasticsearch, Logstash, Kibana, shield and marvel.

Thanks for all the inputs. Very much appreciate it.

[sandeepkanabar]

@tylerjl - I have a question. So, I'm still using elasticsearch puppetforge module v 0.10.3. I have puppet code that invokes esusers using curl command. Not that I'm against upgrading to latest module, but I had already written code before I found that you've released a module supporting shield. And as you said earlier, I've done rsync /etc/elasticsearch/shield to /data/elasticsearch/conf/shield, since /data/elasticsearch/conf is my config directory. The question I have is: I want to create a few custom roles. What would be the best way to create in 0.10.3? Should I write a template that lists my custom roles and privileges and then before invoking esusers, append the contents of the template to the file /etc/elasticsearch/shield? Then when rsync happens, it would get copied to /data/elasticsearch/conf/shield. Is this the correct way?

[tylerjl]

That's probably the simplest way. If you're already syncing the Shield directory where it's needed, then just managing the entire roles.yml file should do it.

[sandeepkanabar]

Thanks @tylerjl . Managing the modified roles.yml sounds better than adding contents. Also, I tried to upgrade to the latest elasticsearch puppetforge module but I am running into errors as shown below:

Debug: Puppet::Type::Elasticsearch_plugin::ProviderElasticsearch_plugin: file /usr/share/elasticsearch/bin/elasticsearch-plugin does not exist Debug: Puppet::Type::Elasticsearch_plugin::ProviderPlugin: file /usr/share/elasticsearch/bin/plugin does not exist Debug: Creating default schedules Error: Could not find dependent File[/usr/share/elasticsearch/plugins] for File[/usr/share/elasticsearch/plugins/shield

Below is a snippet of my profile class. The same code works fine against 0.10.3. When I checked, I found that even installation of Elasticsearch hadn't started and that could explain why the directory is not present.

class profile::elasticsearch_cluster (
  $es_version                   = hiera('profile::elasticsearch::version'),  
  .... //hiera lookups
  $config  = {
  }
){
  user{'elasticsearch':
      ensure => present,
      shell  => "/sbin/nologin"
  }->

  exec { 'create-elasticsearch-snapshot-repo-directory':
    path        => "/bin",
    command     => "mkdir -p ${snapshot_repo_dir}",
  }->
  exec { 'create-elasticsearch-log-directory':
    path        => "/bin",
    command     => "mkdir -p ${log_dir}",
  }->
  file { $snapshot_repo_dir:
    ensure => "directory",
    owner  => "elasticsearch",
    group  => "elasticsearch",
    mode   => 0644,
    recurse => true,
  }

  $config_hash = {
    'ES_HEAP_SIZE' => $es_heap_size,
    'MAX_OPEN_FILES' => '65535',
    'MAX_LOCKED_MEMORY' => 'unlimited'
  }

  #Ensure jdk_oracle
  if ! defined(Class['::profile::oracle_jdk']) {
    class {'profile::oracle_jdk':}
  }

  class { '::elasticsearch':
    version       => $es_version,
    manage_repo  => $es_manage_repo,
    repo_version => $es_repo_version,
    package_pin  => $es_package_pin,
    init_defaults => $config_hash,
    #package_url => $package_url,
    config  => 
    {
        'cluster.name'                          => $es_cluster_name,
        'node.name'                             => $::fqdn,
        'node.master'                           => $es_node_master,
        'node.data'                             => $es_node_data,
        'node.tag'                              => $es_node_tag,
        'node.zone'                             => $es_node_zone,
        'network.host'                          => $es_network_host,
        'bootstrap.mlockall'                    => true,
        'index.number_of_shards'                => $es_index_number_of_shards,
        'index.number_of_replicas'              => $es_index_number_of_replicas,
        'discovery.zen.ping.unicast.hosts'      => $discovery_zen_ping_unicast_hosts,
        'discovery.zen.ping.multicast.enabled'  => $discovery_zen_ping_multicast_enabled,
        'discovery.zen.ping.timeout'            => $discovery_zen_ping_timeout,
        'marvel.agent.enabled'                  => $marvel_agent_enabled,
        'marvel.agent.exporters.id1.type'       => $marvel_agent_exporters,
        'marvel.agent.exporters.id1.host'        => $marvel_agent_exporters_host,
        'marvel.agent.exporters.id1.auth.username' => $marvel_agent_username,
        'marvel.agent.exporters.id1.auth.password' => $marvel_agent_password,
        'path.repo'                             => $snapshot_repo_dir,      
    },
    require  => [Class['::profile::oracle_jdk']]
  }

  elasticsearch::instance { 'es-01':
    datadir => $data_dir,
    logdir => $log_dir,
    configdir => $conf_dir,
    config  => $config
  }

  elasticsearch::plugin{'mobz/elasticsearch-head':
    instances  => 'es-01'
  }

  elasticsearch::plugin {'license':
      instances  => 'es-01',
  }->
  file { 'es-license.json':
        path => '/etc/elasticsearch/es-license.json',
        ensure => file,
        mode => '0755',
        source => 'puppet:///modules/profile/license-file.json'
  }

elasticsearch::plugin {'shield':
      instances  => 'es-01',
  }->
  # add user/role and password for shield authentication

  exec{"add_shield_admin_user":
    command => "/usr/share/elasticsearch/bin/shield/esusers useradd $es_shield_username -r admin -p $es_shield_password",
    require => [Elasticsearch::Plugin['shield'], Elasticsearch::Plugin['license']],
    path => "/usr/bin/:/usr/sbin/:/bin/:/usr/share/elasticsearch/bin/shield",
    onlyif => "test $(/usr/share/elasticsearch/bin/shield/esusers list | grep '$es_shield_username' | wc -l) -eq 0",
    logoutput => true
  }->

  #work around shield limitation. Shield doesn't recognize custom config dirs
  rsync::get { '/data/elasticsearch/conf':
    source  => '/etc/elasticsearch/shield',
    purge   => true,
  }->
  exec { 'chown-shield-folder':
    path      => '/usr/bin:/usr/sbin:/bin',
    command => "ls -al /data/elasticsearch/conf/shield && chown -R elasticsearch:elasticsearch /data/elasticsearch/conf/shield",
  }   
...
}

What could I be missing?

[tylerjl]

@sandeepkanabar if the bin/plugin or bin/elasticsearch-plugin commands aren't found, then yes, Elasticsearch probably never got installed - I'd suggest posting a gist with the entirety of the puppet logs, as the original, root reason for the failure (failing to install Elasticsearch in the first place) may be in there.

I'd also suggest taking our debug session to the forums (I'm @tylerjl there, too) and we can revisit the repo issues if we dig up a bug. 👍

[sandeepkanabar]

Thanks @tylerjl . I've posted in the forum as suggested by you.