search

jjethwa / rundeck

GNU General Public License v3.0
122 stars 137 forks source link

Upgrade from 3.3.0 to 3.3.1 #175

Closed 4Syno closed 4 years ago

4Syno commented 4 years ago

Hi. I lost my rundeck instance during the night. Watchtower updated rundeck to the latest build 3.3.1 [b059aac]. Something went wrong, http port 4440 looks down.

docker usage:

docker run --restart=always -it -d -p 127.0.0.1:4440:4440 \
-e EXTERNAL_SERVER_URL=https://my.domian:9004 \
-e RUNDECK_ADMIN_PASSWORD=superduperPW:D \
-e SMTP_HOST=relay1 -e SMTP_PORT=25 -e SMTP_USERNAME=user \
-e SMTP_PASSWORD= superduperPW:D -e SMTP_DEFAULT_FROM=user@domain \
-e TZ=Europe/Berlin \
-v rundec_data:/etc/rundeck \
-v rundec_var:/var/rundeck \
-v rundec_var_storage:/var/lib/rundeck/var/storage \
-v rundec_var_log:/var/log/rundeck \
-v rundec_mysql:/var/lib/mysql \
-v rundec_ssh:/var/lib/rundeck\.ssh/ \
-v rundec_plugins:/opt/rundeck-plugins \
-v rundec_log:/var/lib/rundeck/logs \
--name rundeck jordan/rundeck:latest

nginx reverse proxy config:

server {
        listen 9004 ssl;
        server_name my.domain;

        ssl_certificate     /etc/letsencrypt/live/my.domain/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/my.domain/privkey.pem;

        error_log  /var/log/nginx/error.log;

    location / {
    proxy_http_version 1.1;
    proxy_set_header    Connection "";
    proxy_set_header       X-Real-IP $remote_addr;
    proxy_set_header       X-Real-IP $remote_addr;  
    proxy_set_header       X-Forwarded-Proto $scheme;   
    # update 12.08.2018
    proxy_set_header    X-Forwarded-Host $host:$server_port;
    proxy_set_header    X-Forwarded-Server $host;   
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_pass              http://127.0.0.1:4440/;
    proxy_headers_hash_bucket_size 128;
    proxy_headers_hash_max_size 1024;

    proxy_read_timeout      600s;
    proxy_send_timeout      600s;

        #auth_basic "RESTRICTED ACCESS";
        #auth_basic_user_file /etc/nginx/.htpasswd;
        }
}

docker logs:

=>Initializing rundeck - This may take a few minutes
=>Generating rundeck key
Generating public/private rsa key pair.
Your identification has been saved in /var/lib/rundeck/.ssh/id_rsa.
Your public key has been saved in /var/lib/rundeck/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa rundeck@12345

The key's randomart image is:
+---[RSA 4096]----+
|    +.o===o.     |
|   o o.*=o+ .    |
|    = +.++ o     |
|   o + .o.. .    |
| .o . . So .     |
|.o.E .    .      |
|. +.+ .          |
| o.. B.+o        |
| .. ++Bo..       |
+----[SHA256]-----+

=>Initializing local MySQL...
[....] Starting MariaDB database server: mysqld[ ok .
[....] Stopping MariaDB database server: mysqld[ ok .

===================================================================
MySQL user 'root' has no password but only allows local connections
MySQL user 'rundeck' password set to du23n--ksj99asd
Rundeck project storage type set to file
Rundeck Storage provider set to file
Rundeck public key:
ssh-rsa asdasasdsasdasd____rundeck@snip
Server URL set to https://my.domain:9004

===================================================================

Generating locales (this might take a while)...

Starting Supervisor...
It can take anywhere from 30 seconds up to a few minutes for the UI
to become available depending on your container/host VM resources.
All further output is stdout/stderr from the supervisor process.

Generation complete.
2020-08-08 14:28:41,120 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2020-08-08 14:28:41,120 WARN For [program:rundeck], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
2020-08-08 14:28:41,120 WARN For [program:mysql], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
2020-08-08 14:28:41,120 INFO Included extra file "/etc/supervisor/conf.d/rundeck.conf" during parsing
2020-08-08 14:28:41,126 INFO RPC interface 'supervisor' initialized
2020-08-08 14:28:41,126 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-08-08 14:28:41,127 INFO supervisord started with pid 426
2020-08-08 14:28:42,129 INFO spawned: 'fatalservice' with pid 429
2020-08-08 14:28:42,131 INFO spawned: 'rundeck' with pid 430
2020-08-08 14:28:42,132 INFO spawned: 'mysql' with pid 431

READY

08.08.2020 14:28:42.138 - Starting mysql
find: '/var/lib/rundeck/cli': No such file or directory
08.08.2020 14:28:42.146 - Starting rundeckdStarting MariaDB database server: mysqld2020-08-08 14:28:43,158 INFO success: fatalservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-08 14:28:43,158 INFO success: rundeck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-08 14:28:43,158 INFO success: mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

4syno:~# netstat -antp |grep 4440
tcp        0      0 127.0.0.1:4440          0.0.0.0:*               LISTEN      9477/docker-proxy
4syno:~# curl -I -L http://localhost:4440/
curl: (56) Recv failure: Connection reset by peer

external browser access:

4syno:~# tail -f /var/log/nginx/error.log
2020/08/08 15:19:07 [error] 9438#9438: *10 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 0.0.0.0, server: my.domain, request: "GET /user/login HTTP/1.1", upstream: "http://127.0.0.1:4440/user/login", host: "my.domain:9004"

Did I miss something for 3.3.1 ? jordan/rundeck:3.3.0 works well with my configuration. I really appreciate any help you can provide.

4Syno commented 4 years ago

One step closer; This is the problem I guess /var/log/rundeck/service.log 


2020-08-08 17:25:46.246 ERROR 483 --- [           main] o.a.t.j.p.ConnectionPool                 : Unable to create initial connections of pool.

java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:906) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:831) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:744) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:676) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:483) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) ~[tomcat-jdbc-9.0.31.jar!/:?]
    at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:412) ~[spring-jdbc-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:385) ~[spring-jdbc-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at com.sun.proxy.$Proxy112.getMetaData(Unknown Source) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
    at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:241) ~[spring-jdbc-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at com.sun.proxy.$Proxy112.getMetaData(Unknown Source) ~[?:?]
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:70) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:101) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:176) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:118) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.build(MetadataBuildingProcess.java:83) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:473) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:84) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:689) ~[hibernate-core-5.4.18.Final.jar!/:5.4.18.Final]
    at org.grails.orm.hibernate.cfg.HibernateMappingContextConfiguration.buildSessionFactory(HibernateMappingContextConfiguration.java:287) ~[grails-datastore-gorm-hibernate5-7.0.3.RELEASE.jar!/:?]
    at org.grails.orm.hibernate.connections.HibernateConnectionSourceFactory.create(HibernateConnectionSourceFactory.java:86) ~[grails-datastore-gorm-hibernate5-7.0.3.RELEASE.jar!/:?]
    at org.grails.orm.hibernate.connections.AbstractHibernateConnectionSourceFactory.create(AbstractHibernateConnectionSourceFactory.java:39) ~[grails-datastore-gorm-hibernate5-7.0.3.RELEASE.jar!/:?]
    at org.grails.orm.hibernate.connections.AbstractHibernateConnectionSourceFactory.create(AbstractHibernateConnectionSourceFactory.java:23) ~[grails-datastore-gorm-hibernate5-7.0.3.RELEASE.jar!/:?]
    at org.grails.datastore.mapping.core.connections.AbstractConnectionSourceFactory.create(AbstractConnectionSourceFactory.java:64) ~[grails-datastore-core-7.0.4.RELEASE.jar!/:?]
    at org.grails.datastore.mapping.core.connections.AbstractConnectionSourceFactory.create(AbstractConnectionSourceFactory.java:52) ~[grails-datastore-core-7.0.4.RELEASE.jar!/:?]
    at org.grails.datastore.mapping.core.connections.ConnectionSourcesInitializer.create(ConnectionSourcesInitializer.groovy:24) ~[grails-datastore-core-7.0.4.RELEASE.jar!/:?]
    at org.grails.orm.hibernate.HibernateDatastore.<init>(HibernateDatastore.java:212) ~[grails-datastore-gorm-hibernate5-7.0.3.RELEASE.jar!/:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_252]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_252]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_252]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_252]
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:300) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:285) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1340) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1186) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:303) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:676) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:188) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1340) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1186) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck(AbstractAutowireCapableBeanFactory.java:990) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:864) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:574) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:520) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:483) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:227) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1419) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1218) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1175) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:595) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:376) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1404) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:592) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:608) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.boot.web.server.WebServerFactoryCustomizerBeanPostProcessor.getWebServerFactoryCustomizerBeans(WebServerFactoryCustomizerBeanPostProcessor.java:87) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.web.server.WebServerFactoryCustomizerBeanPostProcessor.getCustomizers(WebServerFactoryCustomizerBeanPostProcessor.java:78) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.web.server.WebServerFactoryCustomizerBeanPostProcessor.postProcessBeforeInitialization(WebServerFactoryCustomizerBeanPostProcessor.java:70) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.web.server.WebServerFactoryCustomizerBeanPostProcessor.postProcessBeforeInitialization(WebServerFactoryCustomizerBeanPostProcessor.java:58) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) [spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) [spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) [spring-beans-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getWebServerFactory(ServletWebServerApplicationContext.java:210) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:179) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:153) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) [spring-context-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
    at grails.boot.GrailsApp.run(GrailsApp.groovy:96) [grails-core-4.0.3.jar!/:4.0.3]
    at grails.boot.GrailsApp.run(GrailsApp.groovy:456) [grails-core-4.0.3.jar!/:4.0.3]
    at grails.boot.GrailsApp.run(GrailsApp.groovy:443) [grails-core-4.0.3.jar!/:4.0.3]
    at grails.boot.GrailsApp$run.call(Unknown Source) [grails-core-4.0.3.jar!/:4.0.3]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) [groovy-2.5.6.jar!/:2.5.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115) [groovy-2.5.6.jar!/:2.5.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135) [groovy-2.5.6.jar!/:2.5.6]
    at rundeckapp.Application.main(Application.groovy:28) [classes!/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [rundeck-3.3.1-20200727.war:?]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [rundeck-3.3.1-20200727.war:?]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [rundeck-3.3.1-20200727.war:?]
    at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:58) [rundeck-3.3.1-20200727.war:?]
Caused by: com.mysql.cj.exceptions.InvalidConnectionAttributeException: The server time zone value 'CEST' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the 'serverTimezone' configuration property) to use a more specifc time zone value if you want to utilize time zone support.
    at sun.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_252]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_252]
    at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:85) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.util.TimeUtil.getCanonicalTimezone(TimeUtil.java:132) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.protocol.a.NativeProtocol.configureTimezone(NativeProtocol.java:2120) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.protocol.a.NativeProtocol.initServerSession(NativeProtocol.java:2143) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:1310) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:869) ~[mysql-connector-java-8.0.21.jar!/:8.0.21]
    ... 124 more

2020-08-08 17:25:46.249  WARN 483 --- [           main] o.h.e.j.e.i.JdbcEnvironmentInitiator     : HHH000341: Could not obtain connection metadata : Could not create connection to database server. Attempted reconnect 3 times. Giving up.
4Syno commented 4 years ago

I removed the time zone; that fixed the error. But how can I set the right time inside the container now - Any tips? Auswahl_784

jjethwa commented 4 years ago

Hi @4Syno

Sorry about the issue. I would suggest not updating your production instance automatically because there are certain Rundeck releases that might require you to run some upgrade steps manually.

In this case, it looks like the database is complaining about the CEST timezone. Just to make sure, you were setting the server timezone using

-e TZ=Europe/Berlin
4Syno commented 4 years ago

Hey Jordan. You're right! Anyway this wasn't planned. Silly thing that running container can‘t be tagged; so watchtower was faster than I. Not sure if I got you right:

...setting the server timezone using : -e TZ=Europe/Berlin

This cause the error, but it’s the first time since years. I'm using your Image quite a while. Or are there any other option to set the right timezone?

jjethwa commented 4 years ago

Hi @4Syno

Just wanted to make sure you were only setting the timezone using the environment variable for troubleshooting.

From the stacktrace, it looks like the database has an option to have it set via the jdbc driver config:

Caused by: com.mysql.cj.exceptions.InvalidConnectionAttributeException: The server time zone value 'CEST' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the 'serverTimezone' configuration property) to use a more specifc time zone value if you want to utilize time zone support.

I'm betting that property can be added to the jdbc settings in your rundeck-config.properties.

I'll check out the changelog of the MariaDB, OS TZ config, etc and see what was done there. The only change in the container side was updating Rundeck and the Rundeck CLI, so nothing that would affect this in particular 😞

4Syno commented 4 years ago

Got it !! :D Add my own MariaDB configuration file with the following:

[mariadb]
# Set default time zone
default_time_zone = '+02:00'
docker run -it -d \
-p 127.0.0.1:4440:4440 \
-e EXTERNAL_SERVER_URL=http://localhost:4440 \
-e RUNDECK_ADMIN_PASSWORD=admin \
-e TZ=Europe/Berlin \
-v /home/4syno/Apps/rundeck/conf:/etc/mysql/conf.d \
--name rundeck-331 jordan/rundeck:latest
4Syno:~$ docker exec -it rundeck-331 bash
root@fa6dc3473b47:/# date
Sun Aug  9 12:38:38 CEST 2020
root@fa6dc3473b47:/# mysql -u root
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 32
Server version: 10.1.45-MariaDB-0+deb9u1 Debian 9.12

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select now();
+---------------------+
| now()               |
+---------------------+
| 2020-08-09 12:39:03 |
+---------------------+
1 row in set (0.01 sec)

MariaDB [(none)]> exit
Bye
root@fa6dc3473b47:/# tail -f /var/log/rundeck/service.log 
[2020-08-09T12:39:07,658] INFO  rundeckapp.BootStrap - RSS feeds disabled
[2020-08-09T12:39:07,658] INFO  rundeckapp.BootStrap - Using jaas authentication
[2020-08-09T12:39:07,661] INFO  rundeckapp.BootStrap - Preauthentication is disabled
[2020-08-09T12:39:07,764] INFO  rundeckapp.BootStrap - Rundeck is ACTIVE: executions can be run.
[2020-08-09T12:39:07,768] WARN  rundeckapp.BootStrap - The JVM default encoding is not UTF-8: US-ASCII, you may not see output as expected for multibyte locales. Specify -Dfile.encoding=UTF-8 in the JVM options.
[2020-08-09T12:39:07,962] INFO  rundeckapp.BootStrap - workflowConfigFix973: applying... 
[2020-08-09T12:39:07,970] INFO  rundeckapp.BootStrap - workflowConfigFix973: No fix was needed. Storing fix application state.
[2020-08-09T12:39:08,208] INFO  rundeckapp.BootStrap - Rundeck startup finished in 698ms
[2020-08-09T12:39:08,278] INFO  rundeckapp.Application - Started Application in 21.172 seconds (JVM running for 23.128)
Grails application running at http://localhost:4440 in environment: production

Auswahl_789

jjethwa commented 4 years ago

That's great news, @4Syno 😃

4Syno commented 4 years ago

Absolutely – enough lessons learned for this weekend :) Shall I close this case?

jjethwa commented 4 years ago

I think it's good to close out, your solution is really clean 👍