Not working with volume "/etc/rundeck/" mounted

[jolbax]

Hi,

Please help

I find your rundeck container awesome. In order to keep some things persistent I am using the external MariaDB container. All volumes mounted on the host with exception of "/etc/rundeck" work without any problem. When I mount "/etc/rundeck" I am just not able to access the web gui.

I already tested with Docker 1.9.1 and 1.10.3 but it does not seams to be a issue of docker.

In this way I am running the container:

 docker run --name rundeck -p 4440:4440  \
        -v /opt/containers/var/rundeck:/var/rundeck \
        -v /opt/containers/var/log:/var/log/rundeck \
        -v /opt/containers/etc/rundeck:/etc/rundeck \
        -v /opt/containers/var/lib/ssh:/var/lib/rundeck/.ssh \
        --link rundeck-db:rundeck-db \
        -e RUNDECK_STORAGE_PROVIDER=db \
        -e RUNDECK_PROJECT_STORAGE_TYPE=db \
        -e RUNDECK_PASSWORD=rundeck-pass \
        -e DATABASE_URL="jdbc:mysql://rundeck-db/rundeck?autoReconnect=true" \
        -e NO_LOCAL_MYSQL=True \
        -e SERVER_URL=http://rundeck01.domain.int:4440 -t jordan/rundeck:latest 

The logs says nothing but:

tail -f  service.log
2016-04-01 12:39:02.056:INFO:/:Initializing Spring root WebApplicationContext
INFO  BootStrap: Starting Rundeck 2.6.4-1...
INFO  BootStrap: using rdeck.base config property: /var/lib/rundeck
INFO  BootStrap: loaded configuration: /etc/rundeck/framework.properties
INFO  BootStrap: RSS feeds disabled
INFO  BootStrap: Rundeck is ACTIVE: executions can be run.
2016-04-01 12:40:12.381:INFO:oejsh.ContextHandler:started o.e.j.w.WebAppContext{/,file:/var/lib/rundeck/exp/webapp/},/var/lib/rundeck/exp/webapp
2016-04-01 12:40:12.591:INFO:/:Initializing Spring FrameworkServlet 'grails'
2016-04-01 12:40:14.137:INFO:oejus.SslContextFactory:Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
2016-04-01 12:40:14.230:INFO:oejs.AbstractConnector:Started SslSelectChannelConnector@0.0.0.0:4443

Looking forward for advice

Cheers jolbax

[jjethwa]

Hi @jolbax

For the /etc/rundeck volume, are all of the default configuration files in /opt/containers/etc/rundeck ? Thanks :)

[jolbax]

Hi @jjethwa Yes I think so. What I have done a couple of times is to run the container and mount the volumes on a plain system (without already existing directories or files under /etc/rundeck). That means that the container creates the needed folders and files on the hosting system. The container seemed to start correctly but I cannot access to the web gui. So I destroyed the container and run it again but now with the existing directories and files under "/etc/rundeck". But I get the same behavior.

Something strange is that even when I run the container for a fist time, without existing /etc/rundeck directory, the DB information is written twice:

/etc/rundeck/rundeck-config.properties

#loglevel.default is the default log level for jobs: ERROR,WARN,INFO,VERBOSE,DEBUG
loglevel.default=INFO
rdeck.base=/var/lib/rundeck

#rss.enabled if set to true enables RSS feeds that are public (non-authenticated)
rss.enabled=false

# change hostname here
grails.serverURL=http://rundeck01.mappuls.int:4440

dataSource.url = jdbc:mysql://rundeck-db/rundeck?autoReconnect=true
dataSource.username = rundeck
dataSource.password = rundeck-pass
rundeck.storage.provider.1.type=db
rundeck.storage.provider.1.path=/
rundeck.projectsStorageType=db
dataSource.username = rundeck
dataSource.password = rundeck-pass
rundeck.storage.provider.1.type=db
rundeck.storage.provider.1.path=/
rundeck.projectsStorageType=db

This does not happen when /etc/rundeck is not mounted.

I already tried deleting the repeated unneeded lines and restarting rundeck but I still cannot access to the web-gui.

Thanks

[jolbax]

Actually it does not even work when I start the container like this

docker run --name rundeck -p 4440:4440 -v /opt/containers/etc/rundeck:/etc/rundeck -e SERVER_URL=http://axonlu-asvr.mappuls.int:4440 -t jordan/rundeck:latest

With empty "/opt/containers/etc/rundeck" directory and with populated one...

[jolbax]

Success! I could figure out the configuration mistake!

When mounting the /etc/rundeck directory the container is somehow not able to edit "/etc/framework.properties". The port remains 4443 and other values stay untouched. So I started the container with "HTTPS" and port mapping 4443:4443 and it worked.

Since I spent enough time finding this out I wont go on trying to run it with HTTP and port 4440. But It would be nice to see it working :)

Cheers

[jjethwa]

Hi @jolbax

That's great news! The doubling up of the datasource username and password in /etc/rundeck/rundeck-config.properties shouldn't cause any issues. It's a bug in the logic when the container is started up. I'll eventually get around to fixing it hehe.

Good catch with the framework.properties issue. I had missed that test case. I'll have to modify the container run script so it can make the changes to the config as needed. Should have time to get to it a little later today. Thanks so much for your troubleshooting :)

[jjethwa]

Hi @jolbax

This commit should do it: https://github.com/jjethwa/rundeck/commit/915ef1932bac6d4e6cbe07dda80a8ccefd218232

Latest container was automatically built after the push and is available via Docker Hub :)

[jolbax]

Hi @jjethwa I gave it a try. The framework.properties are updated now. But it keeps trying to start on port 4443. Is this defined somewhere else?

docker exec  rundeck cat /var/log/rundeck/service.log
2016-04-05 05:41:27.605:INFO:oejs.Server:jetty-7.6.0.v20120127
2016-04-05 05:41:33.090:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2016-04-05 05:41:36.302:INFO:/:Initializing Spring root WebApplicationContext
INFO  BootStrap: Starting Rundeck 2.6.4-1...
INFO  BootStrap: using rdeck.base config property: /var/lib/rundeck
INFO  BootStrap: loaded configuration: /etc/rundeck/framework.properties
INFO  BootStrap: RSS feeds disabled
INFO  BootStrap: Rundeck is ACTIVE: executions can be run.
2016-04-05 05:42:37.228:INFO:oejsh.ContextHandler:started o.e.j.w.WebAppContext{/,file:/var/lib/rundeck/exp/webapp/},/var/lib/rundeck/exp/webapp
2016-04-05 05:42:37.354:INFO:/:Initializing Spring FrameworkServlet 'grails'
2016-04-05 05:42:38.493:INFO:oejus.SslContextFactory:Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
2016-04-05 05:42:38.580:INFO:oejs.AbstractConnector:Started SslSelectChannelConnector@0.0.0.0:4443

[jjethwa]

Hi @jolbax

It should still start with HTTP over port 4440 as defined by the SERVER_URL var you defined and also start HTTPS over port 4443 if you want to use it. Is that not the case? :(

[jolbax]

Hi @jjethwa Im running this:

docker run --name rundeck -p 4440:4440 -v /opt/containers/etc/rundeck:/etc/rundeck -e SERVER_URL=http://rundeck.domain.int:4440 -t jordan/rundeck:latest

The framework.properties file is being updated correctly. But the service.log shows this output:

...
2016-04-05 05:42:38.580:INFO:oejs.AbstractConnector:Started SslSelectChannelConnector@0.0.0.0:4443
...

So the Web GUI is not reachable over the port 4440 and HTTP. :-(

[jjethwa]

Hi @jolbax

Thanks for your patience. Still trying to reproduce the issue on my side. Our rundeck (production) environment uses a volume for /etc/rundeck but does not exhibit the same behaviour. I tried a few tests, but could not reproduce it as well :(

Here is what I tried:

1. Start a rundeck container with the default (out of the box settings)

sudo docker run --name rundeck -p 4440:4440 -e SERVER_URL=http://my.domain.com:4440 -t jordan/rundeck:latest

1. Once rundeck started, I copied over the contents of /etc/rundeck to the host directory I wanted to use as a volume

sudo docker cp rundeck:/etc/rundeck /home/core/volumes/rundeck/etc

1. Stopped the existing instance

sudo docker stop rundeck && sudo docker rm rundeck

1. Started a new rundeck instance using the volume

sudo docker run --name rundeck -p 4440:4440 -e SERVER_URL=http://my.domain.com:4440 -v /home/core/volumes/rundeck/etc/rundeck:/etc/rundeck -t jordan/rundeck:latest

1. rundeck starts up on 4440:

2016-04-05 21:20:01.886:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:4440

[fia5000]

I'm seeing this exact same behavior @jolbax reported.

[jjethwa]

Hi @jolbax or @fia5000

Can one of you provide /etc/rundeck/profile ? Thanks :)

[jolbax]

sorry for the late answer. So I followed the steps you described above and that worked for me now. Rundeck is running well on port 4440 and with mounted /etc/rundeck volume.

I think my mistake was not mount the /etc/rundeck volume on a empty directory. The container wasn't able to reconfigure all the files. But doing the way you explained worked well. Thanks!

[jolbax]

and here my profile file

RDECK_BASE=/var/lib/rundeck
export RDECK_BASE

JAVA_CMD=java
RUNDECK_TEMPDIR=/tmp/rundeck

RDECK_HTTP_PORT=4440
RDECK_HTTPS_PORT=4443

#
# If JAVA_HOME is set, then add it to home and set JAVA_CMD to use the version specified in that
# path.  JAVA_HOME can be set in the rundeck profile.  Or set in this file.
#JAVA_HOME=<path/to/JDK or JRE/install>

if [ ! -z $JAVA_HOME ]; then
    PATH=$PATH:$JAVA_HOME/bin
    export PATH
    JAVA_CMD=$JAVA_HOME/bin/java
fi

export CLI_CP=$(find /var/lib/rundeck/cli -name \*.jar -printf %p:)
export BOOTSTRAP_CP=$(find /var/lib/rundeck/bootstrap -name \*.jar -printf %p:)
export RDECK_JVM="${RDECK_JVM} -Djava.security.auth.login.config=/etc/rundeck/jaas-loginmodule.conf \
    -Dloginmodule.name=RDpropertyfilelogin \
    -Drdeck.config=/etc/rundeck \
    -Drdeck.base=/var/lib/rundeck \
    -Drundeck.server.configDir=/etc/rundeck \
    -Dserver.datastore.path=/var/lib/rundeck/data \
    -Drundeck.server.serverDir=/var/lib/rundeck \
    -Drdeck.projects=/var/rundeck/projects \
    -Drdeck.runlogs=/var/lib/rundeck/logs \
    -Drundeck.config.location=/etc/rundeck/rundeck-config.properties \
    -Djava.io.tmpdir=$RUNDECK_TEMPDIR"
#
# Set min/max heap size
#
RDECK_JVM="$RDECK_JVM -Xmx1024m -Xms256m -XX:MaxPermSize=256m -server"
#
# SSL Configuration - Uncomment the following to enable.  Check SSL.properties for details.
#
#export RDECK_JVM="${RDECK_JVM} $RDECK_JVM -Drundeck.ssl.config=/etc/rundeck/ssl/ssl.properties -Dserver.https.port=${RDECK_HTTPS_PORT}"

export RDECK_SSL_OPTS="-Djavax.net.ssl.trustStore=/etc/rundeck/ssl/truststore -Djavax.net.ssl.trustStoreType=jks -Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol"

if test -t 0 -a -z "$RUNDECK_CLI_TERSE"
then
  RUNDECK_CLI_TERSE=true
  export RUNDECK_CLI_TERSE
fi

if test -n "$JRE_HOME"
then
   unset JRE_HOME
fi

umask 002

[jjethwa]

Hi @jolbax and @fia5000

Thanks to both of you, I've figured out the issue. If you try to use an empty /etc/rundeck volume, the container will set some configurations which forces Rundeck to start up over SSL. I think most users will already have their configurations in /etc/rundeck or start with the defaults then docker cp then over to the volume. If more users complain, I can make some changes to try to handle this edge case :)

[flypenguin]

Hi there, I am trying to run rundeck with an empty /etc/rundeck volume (host-mounted). I actually expected the container to start up all by itself in a clean configuration. But it didn't. I tried uncommenting or commenting the SSL directive in profile, no help. The config files do appear in the mounted .../etc if mounted. Mounting / not mounting does not make a difference, tried. Also tried mounting / not mounting /var/rundeck from the container. No change. I'd appreciate any help here :)

I am not sure if this is actually the bug here, but I think it might. I'm also not sure how to verify this :)

---

starting rundeck with puppet:

docker::run { 'rundeck':
    image                    => $docker_image,
    volumes                  => [
      #'/var/docker-apps/rundeck/etc:/etc/rundeck',    # with or without makes no difference
      '/var/docker-apps/rundeck/var:/var/rundeck',
      '/var/docker-apps/rundeck/ssh:/var/lib/rundeck/.ssh',
      '/var/docker-apps/rundeck/log:/var/log/rundeck',
    ],
    env                      => [
      "SERVER_URL=${public_url}",    # is: http://rundeck.mycompany.int, and resolves
      "DATABASE_URL=${mysql_url}",   # is: jdbc:mysql://maria.mycompany.int/DBNAME?autoReconnect=true
      "RUNDECK_PASSWORD=${mysql_pass}",
      "NO_LOCAL_MYSQL=true",
    ],
    ports                    => ["${public_port}:4440",],
  }

rundeck logs:

2016-04-26 20:00:24.896:INFO:oejs.Server:jetty-7.6.0.v20120127
2016-04-26 20:00:27.278:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2016-04-26 20:00:28.573:INFO:/:Initializing Spring root WebApplicationContext
2016-04-26 20:01:51.308:INFO:oejs.Server:jetty-7.6.0.v20120127
2016-04-26 20:01:53.598:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2016-04-26 20:01:54.951:INFO:/:Initializing Spring root WebApplicationContext
2016-04-26 20:16:21.018:INFO:oejs.Server:jetty-7.6.0.v20120127
2016-04-26 20:16:23.263:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2016-04-26 20:16:24.555:INFO:/:Initializing Spring root WebApplicationContext
2016-04-26 20:24:25.241:INFO:oejs.Server:jetty-7.6.0.v20120127
2016-04-26 20:24:27.627:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2016-04-26 20:24:29.012:INFO:/:Initializing Spring root WebApplicationContext
WARNING: HTTPS is not enabled, specify -Drundeck.ssl.config=/var/lib/rundeck/server/config/ssl.properties to enable.
2016-04-26 20:41:36.423:INFO:oejs.Server:jetty-7.6.0.v20120127
2016-04-26 20:41:38.676:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2016-04-26 20:41:40.008:INFO:/:Initializing Spring root WebApplicationContext

[jjethwa]

Hi @flypenguin

Try commenting this line out in /etc/rundeck/profile

export RDECK_JVM="$RDECK_JVM -Drundeck.ssl.config=/etc/rundeck/ssl/ssl.properties -Dserver.https.port=${RDECK_HTTPS_PORT}"

Then restart the container and see if it works. The server might still be coming up over HTTPS. Let me know if that works for you and I'll make the necessary changes to the container :)

[flypenguin]

oh boy, after 20 minutes of complete silence (!!!) it told me the database could not be reached (AWS security group issue). I did never wait that long before, so it was my fault. but I directly have another issue I'll open right now :)

but yes, the fix would be handy, cause automated deployment of the container is at least difficult otherwise.

[jjethwa]

Hi @flypenguin

Thanks for the information. I'll try to get those changes in for the next release :)

[jjethwa]

Hi @flypenguin and @jolbax

I've made changes so rundeck will default to HTTP. If you want to enable SSL, you just need to uncomment the SSL line in /etc/rundeck/profile. Thanks for reporting the issue. Please re-open it if there is still an issue :)