redis hangs on fresh install

[brettswift]

I've tried this with and without the system_sysctl option. Both will hang.

Puppet Code:

  require epel

  class { 'redis':
    system_sysctl => true
  }

  Class['epel']->
  Class['redis']

Last few lines of puppet output before it hangs:

==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Filebucketed /etc/redis.conf to puppet with sum 1b59697dc32ec93399fb1828553e5f97
==> masterofpuppets: Notice: /Stage[main]/Redis/File[/etc/redis.conf]/content: content changed '{md5}1b59697dc32ec93399fb1828553e5f97' to '{md5}68b329da9893e34099c7d8ad5cb9c940'
==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Scheduling refresh of Service[redis]
==> masterofpuppets: Notice: /Stage[main]/Redis/File[/var/lib/redis/]/group: group changed 'root' to 'redis'

--> hangs here

If I kill the puppet process and run it again, it completes very quickly.

I saw the same issue raised in ticket #38.

I quite like this module, as it seems quite feature complete, but for the time being I may have to start using a different one if I can't get this installed properly. Is there a work-around ?

Environment:

• Puppet Enterprise 3.3.2
• CentOS 6.5

Yum info:

yum info redis
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: centos.mirror.ca.planethoster.net
 * epel: mirror.clarkson.edu
 * extras: centos.bhs.mirrors.ovh.net
 * updates: mirror.its.dal.ca
Installed Packages
Name        : redis
Arch        : x86_64
Version     : 2.4.10
Release     : 1.el6

[fsalum]

Hi Brett,

I'm on vacation so I can take a look only in Jan for you.

You can try this. Once it hangs ssh to the server and run ps to find out what is stuck, if it is the yum cmd or something else.

If you run it manually after the process is killed, does it execute any other pending step from the redis module installation?

I don't see the output of the package being installed. At what point did it install it?

Sent from my iPhone

On Dec 23, 2014, at 10:23 AM, Brett notifications@github.com wrote:

I've tried this with and without the system_sysctl option. Both will hang.

Puppet Code:

require epel

class { 'redis': system_sysctl => true }

Class['epel']-> Class['redis'] Last few lines of puppet output before it hangs:

==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Filebucketed /etc/redis.conf to puppet with sum 1b59697dc32ec93399fb1828553e5f97 ==> masterofpuppets: Notice: /Stage[main]/Redis/File[/etc/redis.conf]/content: content changed '{md5}1b59697dc32ec93399fb1828553e5f97' to '{md5}68b329da9893e34099c7d8ad5cb9c940' ==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Scheduling refresh of Service[redis] ==> masterofpuppets: Notice: /Stage[main]/Redis/File[/var/lib/redis/]/group: group changed 'root' to 'redis'

--> hangs here

If I kill the puppet process and run it again, it completes very quickly.

I saw the same issue raised in ticket #38.

I quite like this module, as it seems quite feature complete, but for the time being I may have to start using a different one if I can't get this installed properly. Is there a work-around ?

Environment:

Puppet Enterprise 3.3.2 CentOS 6.5 Yum info:

yum info redis Loaded plugins: fastestmirror, security Loading mirror speeds from cached hostfile

• base: centos.mirror.ca.planethoster.net
• epel: mirror.clarkson.edu
• extras: centos.bhs.mirrors.ovh.net
• updates: mirror.its.dal.ca Installed Packages Name : redis Arch : x86_64 Version : 2.4.10 Release : 1.el6 — Reply to this email directly or view it on GitHub.

[brettswift]

This actually happens with another redis module as well: thomasvandoren/redis which doesn't use yum, it installs from source.

Here is a whole bunch of info:

First run again, more complete:

==> masterofpuppets: Info: Applying configuration version '1419435776'
==> masterofpuppets: Notice: /Stage[main]/Epel/File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6]/ensure: defined content as '{md5}d865e6b948a74cb03bc3401c0b01b785'
==> masterofpuppets: Notice: environment: vagrant
==> masterofpuppets: Notice: /Stage[main]/Puppet/Notify[environment: vagrant]/message: defined 'message' as 'environment: vagrant'
==> masterofpuppets: Notice: apptier:
==> masterofpuppets: Notice: /Stage[main]/Puppet/Notify[apptier: ]/message: defined 'message' as 'apptier: '
==> masterofpuppets: Notice: /Stage[main]/Epel/Epel::Rpm_gpg_key[EPEL-6]/Exec[import-EPEL-6]/returns: executed successfully
==> masterofpuppets: Notice: /Stage[main]/Epel/Yumrepo[epel-testing]/ensure: created
==> masterofpuppets: Info: changing mode of /etc/yum.repos.d/epel-testing.repo from 600 to 644
==> masterofpuppets: Notice: /Stage[main]/Epel/Yumrepo[epel-testing-debuginfo]/ensure: created
==> masterofpuppets: Info: changing mode of /etc/yum.repos.d/epel-testing-debuginfo.repo from 600 to 644
==> masterofpuppets: Notice: /Stage[main]/Epel/Yumrepo[epel-testing-source]/ensure: created
==> masterofpuppets: Info: changing mode of /etc/yum.repos.d/epel-testing-source.repo from 600 to 644
==> masterofpuppets: Notice: /Stage[main]/Epel/Yumrepo[epel]/ensure: created
==> masterofpuppets: Info: changing mode of /etc/yum.repos.d/epel.repo from 600 to 644
==> masterofpuppets: Notice: /Stage[main]/Epel/Yumrepo[epel-debuginfo]/ensure: created
==> masterofpuppets: Info: changing mode of /etc/yum.repos.d/epel-debuginfo.repo from 600 to 644
==> masterofpuppets: Notice: /Stage[main]/Epel/Yumrepo[epel-source]/ensure: created
==> masterofpuppets: Info: changing mode of /etc/yum.repos.d/epel-source.repo from 600 to 644
==> masterofpuppets: Notice: /Stage[main]/Redis/Package[redis]/ensure: created
==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Filebucketed /etc/redis.conf to puppet with sum 1b59697dc32ec93399fb1828553e5f97
==> masterofpuppets: Notice: /Stage[main]/Redis/File[/etc/redis.conf]/content: content changed '{md5}1b59697dc32ec93399fb1828553e5f97' to '{md5}68b329da9893e34099c7d8ad5cb9c940'
==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Scheduling refresh of Service[redis]
==> masterofpuppets: Notice: /Stage[main]/Redis/File[/var/lib/redis/]/group: group changed 'root' to 'redis'
~~~ hangs here ~~~~

ps:

13969 /bin/sh /sbin/service redis start
13976 /bin/sh /etc/init.d/redis start
13979 runuser -s /bin/bash redis -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/redis-server /etc/redis.conf
13980 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/redis-server /etc/redis.conf
13981 /usr/sbin/redis-server /etc/redis.conf

redis service:

[root@masterofpuppets vagrant]# /etc/init.d/redis status
redis-server is stopped

[root@masterofpuppets vagrant]# /etc/init.d/redis start
Starting redis-server: [14230] 24 Dec 07:47:06 # Opening port 6379: bind: Address already in use
                                                           [FAILED]

netstat:

[root@masterofpuppets vagrant]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      13981/redis-server

top:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13981 redis     20   0 39940 1584  800 S  0.0  0.1   0:00.73 redis-server

force quit vagrant up, and run vagrant provision again:

==> masterofpuppets: Info: Applying configuration version '1419436356'
==> masterofpuppets: Notice: environment: vagrant
==> masterofpuppets: Notice: /Stage[main]/Puppet/Notify[environment: vagrant]/message: defined 'message' as 'environment: vagrant'
==> masterofpuppets: Notice: apptier:
==> masterofpuppets: Notice: /Stage[main]/Puppet/Notify[apptier: ]/message: defined 'message' as 'apptier: '
==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Filebucketed /etc/redis.conf to puppet with sum 68b329da9893e34099c7d8ad5cb9c940
==> masterofpuppets: Notice: /Stage[main]/Redis/File[/etc/redis.conf]/content: content changed '{md5}68b329da9893e34099c7d8ad5cb9c940' to '{md5}e124ffcc27724697e852ec4bffc5458d'
==> masterofpuppets: Info: /Stage[main]/Redis/File[/etc/redis.conf]: Scheduling refresh of Service[redis]
==> masterofpuppets: Notice: /Stage[main]/Redis/Service[redis]/ensure: ensure changed 'stopped' to 'running'
==> masterofpuppets: Info: /Stage[main]/Redis/Service[redis]: Unscheduling refresh on Service[redis]
==> masterofpuppets: Notice: /Stage[main]/Puppet::Enc/File[/etc/puppetlabs/puppet/enc/]/ensure: created
==> masterofpuppets: Notice: /Stage[main]/Puppet::Enc/File[redis enc rb]/ensure: defined content as '{md5}278f79dbbf36d6644dbe34c8a4676fac'
==> masterofpuppets: Notice: /Stage[main]/Puppet::Enc/File[redis enc sh]/ensure: defined content as '{md5}0174999d53e8878ed0d36cdaebeba660'
==> masterofpuppets: Notice: /Stage[main]/Puppet::Enc/Ini_setting[node_terminus]/value: value changed 'console' to 'exec'
==> masterofpuppets: Notice: /Stage[main]/Puppet::Enc/Ini_setting[external_nodes]/ensure: created
==> masterofpuppets: Notice: Finished catalog run in 4.96 seconds

Interesting bits:

The checksum of the /etc/redis.conf file causes a refresh on the service. When it hangs it has a different checksum than when it starts properly. I should go back to double check but I have noticed the /etc/redis.conf file is empty at one point. Puppet may clear that file somehow, and schedule a refresh of the service. Instead of the service failing to start it just hangs. But I'm not sure why a re-provision would change the file content.. or why the file would ever be empty. That task 'should' be idempotent across two runs.. so what is causing it to not be?

/etc/redis.conf checksum from where it hung

content changed '{md5}1b59697dc32ec93399fb1828553e5f97' to '{md5}68b329da9893e34099c7d8ad5cb9c940'

/etc/redis.conf checksum from second provision

content changed '{md5}68b329da9893e34099c7d8ad5cb9c940' to '{md5}e124ffcc27724697e852ec4bffc5458d'

[brettswift]

Confirmed that when it hangs, /etc/redis.conf is empty.

Before reprovisioning if I try to start the process manually, with the empty .conf file i get this:

[root@masterofpuppets vagrant]# /sbin/service redis start
Starting redis-server: [14312] 24 Dec 08:40:37 * Server started, Redis version 2.4.10
[14312] 24 Dec 08:40:37 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[14312] 24 Dec 08:40:37 * The server is now ready to accept connections on port 6379
[14312] 24 Dec 08:40:37 - 0 clients connected (0 slaves), 717448 bytes in use
[14312] 24 Dec 08:40:42 - 0 clients connected (0 slaves), 717448 bytes in use
[14312] 24 Dec 08:40:47 - 0 clients connected (0 slaves), 717448 bytes in use
[14312] 24 Dec 08:40:52 - 0 clients connected (0 slaves), 717448 bytes in use

and it keeps outputting 0 clients connected every 5 seconds.. which is what the Puppet process is waiting on.

Kinda crappy that it's possible for a linux service start sequence to behave like this.

I tried applying this with the service_restart => false as well, hangs just the same.

I'm eyeballing this code and nothing jumps out at me as to how this file could be empty.

[fsalum]

That's what I suspected the blank redis.conf means the module can't find the redis version to use the right config file.

In the redis lib directory look at the yum cmd line it uses to find out the redis version. Manually run it and check if it is working.

Also there is a parameter to manually enforce a version to use, you can also try that.

Bad I don't have a computer with me :(

Sent from my iPhone

On Dec 24, 2014, at 11:50 AM, Brett notifications@github.com wrote:

Confirmed that when it hangs, /etc/redis.conf is empty.

Before reprovisioning if I try to start the process manually, with the empty .conf file i get this:

[root@masterofpuppets vagrant]# /sbin/service redis start Starting redis-server: [14312] 24 Dec 08:40:37 * Server started, Redis version 2.4.10 [14312] 24 Dec 08:40:37 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. [14312] 24 Dec 08:40:37 * The server is now ready to accept connections on port 6379 [14312] 24 Dec 08:40:37 - 0 clients connected (0 slaves), 717448 bytes in use [14312] 24 Dec 08:40:42 - 0 clients connected (0 slaves), 717448 bytes in use [14312] 24 Dec 08:40:47 - 0 clients connected (0 slaves), 717448 bytes in use [14312] 24 Dec 08:40:52 - 0 clients connected (0 slaves), 717448 bytes in use and it keeps outputting 0 clients connected every 5 seconds.. which is what the Puppet process is waiting on.

Kinda crappy that it's possible for a linux service start sequence to behave like this.

I tried applying this with the service_restart => false as well, hangs just the same.

I'm eyeballing this code and nothing jumps out at me as to how this file could be empty.

— Reply to this email directly or view it on GitHub.

[brettswift]

Redis 2.4.10 installed ok. That's all that's available on epel.

The remi repo has redis 2.8 so trying that.

Thanks! Hope this isn't getting in the way of your holidays.

Brett Sent from my iPhone

On Dec 24, 2014, at 10:28, Felipe Salum notifications@github.com wrote:

That's what I suspected the blank redis.conf means the module can't find the redis version to use the right config file.

In the redis lib directory look at the yum cmd line it uses to find out the redis version. Manually run it and check if it is working.

Also there is a parameter to manually enforce a version to use, you can also try that.

Bad I don't have a computer with me :(

Sent from my iPhone

On Dec 24, 2014, at 11:50 AM, Brett notifications@github.com wrote:

Confirmed that when it hangs, /etc/redis.conf is empty.

Before reprovisioning if I try to start the process manually, with the empty .conf file i get this:

[root@masterofpuppets vagrant]# /sbin/service redis start Starting redis-server: [14312] 24 Dec 08:40:37 * Server started, Redis version 2.4.10 [14312] 24 Dec 08:40:37 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. [14312] 24 Dec 08:40:37 * The server is now ready to accept connections on port 6379 [14312] 24 Dec 08:40:37 - 0 clients connected (0 slaves), 717448 bytes in use [14312] 24 Dec 08:40:42 - 0 clients connected (0 slaves), 717448 bytes in use [14312] 24 Dec 08:40:47 - 0 clients connected (0 slaves), 717448 bytes in use [14312] 24 Dec 08:40:52 - 0 clients connected (0 slaves), 717448 bytes in use and it keeps outputting 0 clients connected every 5 seconds.. which is what the Puppet process is waiting on.

Kinda crappy that it's possible for a linux service start sequence to behave like this.

I tried applying this with the service_restart => false as well, hangs just the same.

I'm eyeballing this code and nothing jumps out at me as to how this file could be empty.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

[brettswift]

FYI, installing Remi first, and then letting your module do it's thing, worked for a basic install.

[xcompass]

I got similar problem on CentOS 6.5. redis.conf was empty and hang at /sbin/service redis start. Kill redis process and break puppet, then rerun puppet. It will generate the correct conf file. Using EPEL and IUS repo.

The log shows: ==> centos65: Notice: /Stage[main]/Redis/File[/etc/redis.conf]/content: content changed '{md5}1b59697dc32ec93399fb1828553e5f97' to '{md5}68b329da9893e34099c7d8ad5cb9c940'

The 68b329da9893e34099c7d8ad5cb9c940 is the empty redis.conf md5. I thought puppet suppose to replace the redis.conf with the one in the module. Why is it empty?

[xcompass]

I think I got some clue. It is because the facter redis_version is empty. So there is no condition in redis.conf met and an empty file is generated.

It seems the facter is calculated before puppet execute any statement. And facter uses installed redis version or try to figure it out through yum (in CentOS). Becuase redis package is in their party repo, e.g. EPEL or IUS (2.8.x in my case). Facter couldn't find any package, thus empty version is returned.

My current workaround is to set redis_version_override. It will force facter to use it. Tested on CentOS.

Maybe there should be a default version number for each distro.

[zeroecco]

@fsalum I think I have a fix for this. The if block is setting the redis_version but the case statement is overwriting later. If we combine the two it should work (dropping the else at the end of the case):

    redis_version = Facter::Util::Resolution.exec('/usr/sbin/redis-server --version')
        if redis_version.nil?
          redis_version = Facter::Util::Resolution.exec(yum+" info redis 2> /dev/null | /bin/grep '^Version' | /bin/awk -F ':' '{printf(\"%s\",$2)}' | sort -nr | head -1")
        else
          case redis_version
              when /2\.8\.[0-9]/
                  #set version to 2.8
                  redis_version = '2.8.x'
              when /2\.6\.[0-9]/
                  #set version to 2.6
                  redis_version = '2.6.x'
              when /2\.4\.[0-9]/
                  #set version to 2.4
                  redis_version = '2.4.x'
              when /2\.2\.[0-9]/
                  #set version to 2.2
                  redis_version = '2.2.x'
          end
        end
        redis_version

[fsalum]

@zeroecco I don't think that would work because we need to set the version to that specific A.B.x to use in the template.

What if the redis_version inside the if-block we rename the variable to something else?

    redis_lookup = Facter::Util::Resolution.exec('/usr/bin/redis-server --version')
    if redis_lookup.nil?
        redis_lookup = Facter::Util::Resolution.exec(dpkg+" show redis-server 2> /dev/null | /bin/grep -i 'version:' | /usr/bin/awk '{printf(\"%s\",$2)}' | sort -nr | head -1")
    end

    case redis_lookup
        when /2\.8\.[0-9]/
            #set version to 2.8
            redis_version = '2.8.x'
        when /2\.6\.[0-9]/
            #set version to 2.6
            redis_version = '2.6.x'
        when /2\.4\.[0-9]/
            #set version to 2.4
            redis_version = '2.4.x'
        when /2\.2\.[0-9]/
            #set version to 2.2
            redis_version = '2.2.x'
        else
            redis_version = 'nil'
    end
    redis_version