Problem upgrading to 3.3.4

[sylvaindd]

@jjethwa I encounter this error in logs after upgrading to 3.3.4.

find: '/var/lib/rundeck/cli': No such file or directory

Also there is this warning for this release (I don't know if it impacts this image) :

WARNING

For MySQL users: Starting with Rundeck 3.3.4 the MySQL JDBC driver will no longer be bundled with the distributions. See the Upgrade Guide for instructions to ensure your Rundeck installation can still connect to MySQL after the upgrading.

Could you please look into this ?

[jjethwa]

Hi @sylvaindd

Sure thing, I'll take a look tomorrow as I was just heading to bed 😄

For the jdbc driver issue, the docs state to switch to the MariaDB driver instead

Set the following in rundeck-config.properties

dataSource.driverClassName=org.mariadb.jdbc.Driver

I'll have to see where the error about the /var/lib/rundeck/cli path not being found is coming from. If it's from the rundeck-cli Deb package install, we'll need to report it upstream.

[jjethwa]

Hi @sylvaindd

Please pull the 3.3.4 tag and check it out. DB connectivity works and rd-cli functionality seems OK. Let me know where you ran into the /var/lib/rundeck/cli path error

[jjethwa]

Changes: https://github.com/jjethwa/rundeck/pull/181

[sylvaindd]

Still happening :

09.10.2020 12:11:47.676 - Starting rundeckd2020-10-09 12:11:48,678 INFO success: rundeck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-09 12:11:52,683 INFO exited: rundeck (exit status 0; expected)
2020-10-09 12:11:53,685 INFO spawned: 'rundeck' with pid 1112
find: '/var/lib/rundeck/cli': No such file or directory

Here are my environment variables :

EXTERNAL_SERVER_URL=https://xxxxxxx
RUNDECK_GRAILS_URL=https://xxxx
RUNDECK_SERVER_ADDRESS=0.0.0.0

NO_LOCAL_MYSQL=true
RUNDECK_STORAGE_PROVIDER=db
RUNDECK_PROJECT_STORAGE_TYPE=db
DATABASE_URL=jdbc:mysql://xxxxxxxxxx
#DATABASE_DRIVER=com.mysql.jdbc.Driver
DATABASE_DRIVER=org.mariadb.jdbc.Driver
DATABASE_ADMIN_USER=rundeck
DATABASE_ADMIN_PASSWORD=rundeck
RUNDECK_PASSWORD=rundeck
#SKIP_DATABASE_SETUP=true

RUNDECK_FEATURE_WEBHOOKS_ENABLED=true
RUNDECK_SECURITY_APICOOKIEACCESS_ENABLED=true
RDECK_JVM_SETTINGS=-Drundeck.jetty.connector.forwarded=true

Installation logs :

rundeck.deb: OK
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3613k  100 3613k    0     0  10.0M      0 --:--:-- --:--:-- --:--:-- 10.0M
rundeck-cli.deb: OK
/
Selecting previously unselected package rundeck-cli.
(Reading database ... 14475 files and directories currently installed.)
Preparing to unpack /tmp/rundeck-cli.deb ...
Unpacking rundeck-cli (1.3.4-1) ...
Selecting previously unselected package rundeck.
Preparing to unpack /tmp/rundeck.deb ...
Unpacking rundeck (3.3.4.20201007-1) ...
Setting up rundeck-cli (1.3.4-1) ...
Setting up rundeck (3.3.4.20201007-1) ...
Adding group rundeck....done
Adding system user rundeck....done

Container folders :

[jjethwa]

Hmm, so that error is popping up right after supervisor starts the Rundeck process. Are you using any volumes for /var/lib/rundeck or similar?

I'll poke around a bit more tomorrow and try to reproduce.

[sylvaindd]

Yes it does. No volume for /var/** only for /etc/rundeck

[jjethwa]

Hi @sylvaindd

So far I am not able to reproduce the error locally. Is it actually causing an issue when you're trying to start the container? Do you have any other plugins enabled?

[sylvaindd]

Hi @jjethwa

It keeps restarting :

find: '/var/lib/rundeck/cli': No such file or directory
12.10.2020 06:08:16.474 - Starting rundeckd2020-10-12 06:08:17,476 INFO success: fatalservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-12 06:08:17,476 INFO success: rundeck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-12 06:08:21,481 INFO exited: rundeck (exit status 0; expected)
2020-10-12 06:08:22,483 INFO spawned: 'rundeck' with pid 119
find: '/var/lib/rundeck/cli': No such file or directory
12.10.2020 06:08:22.496 - Starting rundeckd2020-10-12 06:08:23,498 INFO success: rundeck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-12 06:08:27,503 INFO exited: rundeck (exit status 0; expected)
2020-10-12 06:08:28,505 INFO spawned: 'rundeck' with pid 151
find: '/var/lib/rundeck/cli': No such file or directory
12.10.2020 06:08:28.519 - Starting rundeckd2020-10-12 06:08:29,520 INFO success: rundeck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-12 06:08:33,525 INFO exited: rundeck (exit status 0; expected)
2020-10-12 06:08:34,528 INFO spawned: 'rundeck' with pid 183
find: '/var/lib/rundeck/cli': No such file or directory

No plugins enabled. I just realised that there is no /var/lib/rundeck/cli in my 3.3.3 working container either.

The error is coming from /etc/rundeck/profile when it's sourced by supervisor on :

for jar in $(find $RDECK_INSTALL/cli -name '*.jar') ; do
  CLI_CP=${CLI_CP:+$CLI_CP:}$jar
done

I added "mkdir -p /var/lib/rundeck/cli" before sourcing and the error is not appearing anymore. I also copied rundeck-cli in the cli dir : "cp -R rundeck-cli/rd-cli-tool-shadow-1.3.3/ rundeck/cli/" However, it keeps restarting as before :

12.10.2020 06:45:19.476 - Starting rundeckd2020-10-12 06:45:20,479 INFO success: rundeck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-12 06:45:24,483 INFO exited: rundeck (exit status 0; expected)
2020-10-12 06:45:25,485 INFO spawned: 'rundeck' with pid 3704
12.10.2020 06:45:25.500 - Starting rundeckd2020-10-12 06:45:26,502 INFO success: rundeck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Any idea what I could test ?

[jjethwa]

Hi @sylvaindd

Can you docker exec into the container (or check the logs /var/log/rundeck volume if you have it set up) and check for errors in the service.log and rundeck.log? I think that might provide a clue.

On the 3.3.3 and 3.3.4 container I see the path as /var/lib/rundeck-cli

[sylvaindd]

Hi @jjethwa

Well, I didn't think to check these logs :(. It was actually missing /etc/rundeck/log4j2.properties which it couldn't create because of rights issue on my volume. After fixing that, it now starts correctly.

I don't know why it doesn't write in my volume though, it belongs to 104:106, and when I exec in the container I can write file and edit them (as the user rundeck).

The profile file wasn't up to date either, I don't have the /var/rundeck/cli error anymore.

Thank you for your help ;).

I think you can release the 3.3.4 !

[jjethwa]

That's great news, @sylvaindd

The chowns in /opt/run should have fixed the permissions issue, but let's keep an eye of it happens again