Error opening zip file or JAR manifest missing : /usr/sbin/../lib/jamm-0.3.0.jar

[palmertime]

This is probably not an issue with the module but rather an issue with me not knowing what i am doing. Basically, i am setting up a RHEL7 system with the DataStax Enterprise stack of software. My first choice would be to do a TAR install but that doesn't seem possible so i have been using the RPM install. So far i have been unsuccessful getting it work. Below is my manifest, puppet output, systemctl and messages log.

Any help would be appreciated.

Here is my manifest.pp: group { 'cassandra': ensure => present, name => 'cassandra', gid => '220', before => User['cassandra'], } user { 'cassandra': ensure => present, name => 'cassandra', uid => '220', comment => 'Cassandra service account', gid => '220', home => '/usr/local/cassandra', managehome => true, before => Class['cassandra'], } file { 'dse-data': ensure => directory, path => '/usr/local/cassandra/dse-data/', group => '220', mode => '0775', owner => '220', before => Class['cassandra'], require => User['cassandra'], } class { 'cassandra::java': #package_name => 'java-1.8.0-oracle', require => Class['yumclient'], } class {'cassandra::datastax_repo': descr => 'DataStax Repo for DataStax Enterprise', pkg_url => 'https://user%40gmail.com:password@rpm.datastax.com/enterprise', before => Class['cassandra'], } class { 'cassandra': cluster_name => 'Test', commitlog_directory => '/usr/local/cassandra/dse-data/commitlog', commitlog_directory_mode => '700', config_path => '/etc/dse/cassandra', data_file_directories => ['/usr/local/cassandra/dse-data/data'], dc => 'Corp', endpoint_snitch => 'GossipingPropertyFileSnitch', listen_address => $fqdn, package_ensure => 'latest', package_name => 'dse-full', rack => 'rack1', service_name => 'dse', service_systemd => true, saved_caches_directory => '/usr/local/cassandra/dse-data/saved_caches', }

The puppet apply works without error: Notice: /Stage[main]/Cassandra::Datastax_repo/Yumrepo[datastax]/ensure: created Notice: /Stage[main]/Cassandra::Java/Package[jna]/ensure: created Notice: /Stage[main]/Main/Node[cass1.build.networkfleet.com]/Group[cassandra]/ensure: created Notice: /Stage[main]/Main/Node[cass1.build.networkfleet.com]/User[cassandra]/ensure: created Notice: /Stage[main]/Main/Node[cass1.build.networkfleet.com]/File[dse-data]/ensure: created Notice: /Stage[main]/Cassandra/File[/usr/lib/systemd/system/dse.service]/ensure: created Notice: /Stage[main]/Cassandra/Package[dse-full]/ensure: created Notice: /Stage[main]/Cassandra/File[/usr/local/cassandra/dse-data/saved_caches]/ensure: created Notice: /Stage[main]/Cassandra/Ini_setting[rackdc.properties.dc]/value: value changed 'DC1' to 'Corp' Notice: /Stage[main]/Cassandra/File[/usr/local/cassandra/dse-data/commitlog]/ensure: created Notice: /Stage[main]/Cassandra/Ini_setting[rackdc.properties.rack]/value: value changed 'RAC1' to 'rack1' Notice: /Stage[main]/Cassandra/File[/etc/dse/cassandra/cassandra.yaml]/content: content changed '{md5}a27249baf78b2ebe23dc89bde1d865c3' to '{md5}c93e3fc181e26f153c018bad8f56b2a0' Notice: /Stage[main]/Cassandra/File[/etc/dse/cassandra/cassandra.yaml]/mode: mode changed '0755' to '0644' Notice: /Stage[main]/Cassandra/Cassandra::Private::Data_directory[/usr/local/cassandra/dse-data/data]/File[/usr/local/cassandra/dse-data/data]/ensure: created Notice: /Stage[main]/Cassandra/Service[cassandra]/ensure: ensure changed 'stopped' to 'running'

But the service is in a failed state and errors in the /var/log/messages

systemctl status:

[root@cass1 ~]# systemctl status dse ● dse.service - Cassandra Loaded: loaded (/usr/lib/systemd/system/dse.service; enabled; vendor preset: disabled) Active: failed (Result: start-limit) since Wed 2016-02-10 13:21:25 EST; 3min 8s ago Process: 4329 ExecStart=/usr/sbin/cassandra -f -p /var/run/dse/dse.pid (code=exited, status=1/FAILURE) Main PID: 4329 (code=exited, status=1/FAILURE)

Feb 10 13:21:24 cass1.build.networkfleet.com systemd[1]: dse.service: main process exited, code=exited, status=1/FAILURE Feb 10 13:21:24 cass1.build.networkfleet.com systemd[1]: Unit dse.service entered failed state. Feb 10 13:21:24 cass1.build.networkfleet.com systemd[1]: dse.service failed. Feb 10 13:21:25 cass1.build.networkfleet.com systemd[1]: dse.service holdoff time over, scheduling restart. Feb 10 13:21:25 cass1.build.networkfleet.com systemd[1]: start request repeated too quickly for dse.service Feb 10 13:21:25 cass1.build.networkfleet.com systemd[1]: Failed to start Cassandra. Feb 10 13:21:25 cass1.build.networkfleet.com systemd[1]: Unit dse.service entered failed state. Feb 10 13:21:25 cass1.build.networkfleet.com systemd[1]: dse.service failed.

/var/log/messages: Feb 10 12:21:40 cass1 systemd: Started Cassandra. Feb 10 12:21:40 cass1 systemd: Starting Cassandra... Feb 10 12:21:40 cass1 cassandra: Error opening zip file or JAR manifest missing : /usr/sbin/../lib/jamm-0.3.0.jar Feb 10 12:21:40 cass1 cassandra: Error occurred during initialization of VM Feb 10 12:21:40 cass1 cassandra: agent library failed to init: instrument Feb 10 12:21:40 cass1 systemd: dse.service: main process exited, code=exited, status=1/FAILURE

[dallinb]

Hi,

This problem has just been noticed by my client. Working on a fix for it ASAP.

For some reason, manually running service dse start works fine so need to know what systemd is doing. Struggling a bit seeing as we all swear that this did used to work.

Best wishes,

Ben

[palmertime]

Thanks for the update. Let me know if you need me to provide further info or to test. My Vagrant environment is ready!

[dallinb]

This problem is referred to here http://stackoverflow.com/questions/31656479/missing-jar-manifest-after-running-datastax-enterprise-4-7 although, it seems specific to Debian. However, my client site is having the same issue with CentOS 7.1. Taking a look.

On a system that is not affected (i.e. that does not use service_systemd) I get the following:

# systemctl status dse
dse.service - LSB: DataStax Enterprise
   Loaded: loaded (/etc/rc.d/init.d/dse)
   Active: active (exited) since Fri 2016-02-12 07:09:24 UTC; 3min 50s ago
  Process: 23852 ExecStop=/etc/rc.d/init.d/dse stop (code=exited, status=0/SUCCESS)
  Process: 24054 ExecStart=/etc/rc.d/init.d/dse start (code=exited, status=0/SUCCESS)

[dallinb]

OK, the following file seems to work fine for DSE (/usr/lib/systemd/system/dse.service):

# Managed by Puppet.  Changes are likely to be overwritten.

[Unit]
Description=Cassandra
After=network.target

[Service]
PIDFile=/var/run/dse/dse.pid
#User=cassandra
#Group=cassandra
ExecStart=/etc/init.d/dse start
ExecStop=/etc/init.d/dse stop
StandardOutput=journal
StandardError=journal
LimitNOFILE=100000
LimitMEMLOCK=infinity
LimitNPROC=32768
LimitAS=infinity
Restart=always
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

Need to get this into the puppet module and then tested against the community edition.

[dallinb]

That file also fixes a problem that even when the cassandra/dse service shut down successfully, it was reported as a failed service. This is because an exit status of 143 (common for Java) can be interpreted as a success.

[dallinb]

Releasing v1.12.2, follow #167 and please let me know how it goes. If still not fixed, I'll reopen this issue.