log settings confusion under centos sysv

[petere]

Under centos with the sysv init style, the settings for log directory and log level end up in two different places and they disagree with each other by default.

In /etc/default/zookeeper, there are

ZOO_LOG_DIR="<%= node[:zookeeper][:log_dir] %>"
ZOO_LOG4J_PROP="INFO,ROLLINGFILE"

But this file is only read by the init script, which calls zkServer.sh, which does not use these settings. The only effect of these settings is that the init script creates the configured log directory if necessary (shouldn't be, because the install recipe already does it).

zkServer.sh instead uses zookeeper-env.sh, which pulls its data from node[:zookeeper][:env_vars], which are not set by default. As a result, the hardcoded default values are used and the call ends up being

java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE

Somehow, these settings should be moved to one place only.

[jeffbyrnes]

@petere thanks so much for spotting this. The log dir changes are recent. I knew there was a bit of a snarl with how these things are set, but it’s tricky to suss out, and the init scripts don’t all agree on how things get kicked off.

I’ll try to spend some cycles on this, but I won’t have any time for a little while. Pretty sure @jakedavis is also fully loaded with other things.

Pull requests are always welcome!

[jeffbyrnes]

@petere so this is really confusing. However, I can confirm that ZK is started with the settings you mention on CentOS.

Reading through zkServer.sh on a converged CentOS box, I spy this a little ways in:

case $1 in
start)
    echo  -n "Starting zookeeper ... "
    if [ -f "$ZOOPIDFILE" ]; then
      if kill -0 `cat "$ZOOPIDFILE"` > /dev/null 2>&1; then
         echo $command already running as process `cat "$ZOOPIDFILE"`.
         exit 0
      fi
    fi
    nohup "$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
    -cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

So the ZOO_LOG_DIR value should be respected.

However, right at the top of zkServer.sh is this:

if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then
  . "$ZOOBINDIR/../libexec/zkEnv.sh"
else
  . "$ZOOBINDIR/zkEnv.sh"
fi

Which in turn leads to:

if [ "x$ZOOCFGDIR" = "x" ]
then
  if [ -e "${ZOOKEEPER_PREFIX}/conf" ]; then
    ZOOCFGDIR="$ZOOBINDIR/../conf"
  else
    ZOOCFGDIR="$ZOOBINDIR/../etc/zookeeper"
  fi
fi

if [ -f "${ZOOCFGDIR}/zookeeper-env.sh" ]; then
  . "${ZOOCFGDIR}/zookeeper-env.sh"
fi

So it would seem that zookeeper-env.sh is overriding things. However, it doesn’t exist if you don’t populate that attribute.

However, if you continue looking through zkEnv.sh:

if [ "x${ZOO_LOG_DIR}" = "x" ]
then
    ZOO_LOG_DIR="."
fi

Which appears to be the culprit. So the real trick is to ensure that ZOO_LOG_DIR has been populated, to avoid it being overwritten. Yet it should be, courtesy of what’s in /etc/default/zookeeper.

Frankly, the ZK folks have made this absolutely ridiculously hard to configure.

[jeffbyrnes]

No response in some time, closing out.