Error 500 - org.jasypt.exceptions.EncryptionOperationNotPossibleException

[zealouspatcat]

Hello,

I'm getting an error when I log into my rundeck. UI is working but I can't access any of my files. I noticed there was a similar issue here, directing me here, but I'm not sure that I want to change the rundeck-config file because I use the passwords for mariadb:

Suggested solution:

  rundeck.storage.converter.1.config.password=default.encryption.password
  rundeck.config.storage.converter.1.config.password=default.encryption.password

Rundeck, Error 500: Internal Server Error

URI /rundeck/menu/home Class org.jasypt.exceptions.EncryptionOperationNotPossibleException Message Decryption failed. Caused by null Trace

        Line | Method
        ->>  452 | getInputStream            in org.rundeck.plugin.encryption.JasyptEncryptionConverterPlugin$DecryptStream
        - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
        |     42 | getInputStream            in com.dtolabs.rundeck.core.storage.BaseStreamResource
        |    462 | loadProjectConfigResource in rundeck.services.ProjectManagerService
        |    700 | $tt__loadProjectConfig    in     ''
        |     94 | doInTransaction . . . . . in grails.gorm.transactions.GrailsTransactionTemplate$2
        |    140 | execute                   in org.springframework.transaction.support.TransactionTemplate
        |     91 | execute . . . . . . . . . in grails.gorm.transactions.GrailsTransactionTemplate
        |    148 | getProjectConfig          in com.dtolabs.rundeck.server.projects.RundeckProject
        |     56 | getProperty . . . . . . . in     ''
        |    248 | doCall                    in rundeck.services.FrameworkService$_projectLabels_closure2
        |    246 | projectLabels . . . . . . in rundeck.services.FrameworkService
        |    326 | refreshSessionProjects    in     ''
        |   2270 | $tt__home . . . . . . . . in rundeck.controllers.MenuController
        |     94 | doInTransaction           in grails.gorm.transactions.GrailsTransactionTemplate$2
        |    140 | execute . . . . . . . . . in org.springframework.transaction.support.TransactionTemplate
        |     91 | execute                   in grails.gorm.transactions.GrailsTransactionTemplate
        |    223 | invoke . . . . . . . . .  in org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker
        |    188 | invoke                    in org.grails.core.DefaultGrailsControllerClass
        |     90 | handle . . . . . . . . .  in org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter
        |   1040 | doDispatch                in org.springframework.web.servlet.DispatcherServlet
        |    943 | doService . . . . . . . . in     ''
        |   1006 | processRequest            in org.springframework.web.servlet.FrameworkServlet
        |    898 | doGet . . . . . . . . . . in     ''
        |    883 | service                   in     ''
        |    755 | handle . . . . . . . . .  in org.eclipse.jetty.servlet.ServletHolder
        |   1617 | doFilter                  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     96 | run . . . . . . . . . . . in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter$1
        |    422 | doAs                      in javax.security.auth.Subject
        |    105 | doFilter . . . . . . . .  in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter
        |   1604 | doFilter                  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     67 | doFilterInternal . . . .  in org.rundeck.grails.plugins.securityheaders.RundeckSecurityHeadersFilter
        |   1604 | doFilter                  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |    320 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    127 | invoke                    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor
        |     91 | doFilter . . . . . . . .  in     ''
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    119 | doFilter . . . . . . . .  in org.springframework.security.web.access.ExceptionTranslationFilter
        |     64 | doFilter                  in grails.plugin.springsecurity.web.UpdateRequestContextHolderExceptionTranslationFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     54 | doFilter                  in grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    158 | doFilter                  in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    170 | doFilter                  in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     96 | run                       in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter$1
        |    422 | doAs . . . . . . . . . .  in javax.security.auth.Subject
        |    105 | doFilter                  in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    200 | doFilter                  in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     64 | doFilter                  in grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    105 | doFilter                  in org.springframework.security.web.context.SecurityContextPersistenceFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     58 | doFilter                  in grails.plugin.springsecurity.web.SecurityRequestHolderFilter
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    215 | doFilterInternal          in org.springframework.security.web.FilterChainProxy
        |    178 | doFilter . . . . . . . .  in     ''
        |   1604 | doFilter                  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     77 | doFilterInternal . . . .  in org.grails.web.servlet.mvc.GrailsWebRequestFilter
        |   1604 | doFilter                  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     67 | doFilterInternal . . . .  in org.grails.web.filters.HiddenHttpMethodFilter
        |   1604 | doFilter                  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |    545 | doHandle . . . . . . . .  in org.eclipse.jetty.servlet.ServletHandler
        |    143 | handle                    in org.eclipse.jetty.server.handler.ScopedHandler
        |    590 | handle . . . . . . . . .  in org.eclipse.jetty.security.SecurityHandler
        |    127 | handle                    in org.eclipse.jetty.server.handler.HandlerWrapper
        |    235 | nextHandle . . . . . . .  in org.eclipse.jetty.server.handler.ScopedHandler
        |   1607 | doHandle                  in org.eclipse.jetty.server.session.SessionHandler
        |    233 | nextHandle . . . . . . .  in org.eclipse.jetty.server.handler.ScopedHandler
        |   1297 | doHandle                  in org.eclipse.jetty.server.handler.ContextHandler
        |    188 | nextScope . . . . . . . . in org.eclipse.jetty.server.handler.ScopedHandler
        |    485 | doScope                   in org.eclipse.jetty.servlet.ServletHandler
        |   1577 | doScope . . . . . . . . . in org.eclipse.jetty.server.session.SessionHandler
        |    186 | nextScope                 in org.eclipse.jetty.server.handler.ScopedHandler
        |   1212 | doScope . . . . . . . . . in org.eclipse.jetty.server.handler.ContextHandler
        |    141 | handle                    in org.eclipse.jetty.server.handler.ScopedHandler
        |    127 | handle . . . . . . . . .  in org.eclipse.jetty.server.handler.HandlerWrapper
        |    500 | handle                    in org.eclipse.jetty.server.Server
        |    383 | lambda$handle$1 . . . . . in org.eclipse.jetty.server.HttpChannel
        |    547 | dispatch                  in     ''
        |    375 | handle . . . . . . . . .  in     ''
        |    270 | onFillable                in org.eclipse.jetty.server.HttpConnection
        |    311 | succeeded . . . . . . . . in org.eclipse.jetty.io.AbstractConnection$ReadCallback
        |    103 | fillable                  in org.eclipse.jetty.io.FillInterest
        |    117 | run . . . . . . . . . . . in org.eclipse.jetty.io.ChannelEndPoint$2
        |    336 | runTask                   in org.eclipse.jetty.util.thread.strategy.EatWhatYouKill
        |    313 | doProduce . . . . . . . . in     ''
        |    171 | tryProduce                in     ''
        |    129 | run . . . . . . . . . . . in     ''
        |    388 | run                       in org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread
        |    806 | runJob . . . . . . . . .  in org.eclipse.jetty.util.thread.QueuedThreadPool
        |    938 | run                       in org.eclipse.jetty.util.thread.QueuedThreadPool$Runner
        ^    748 | run . . . . . . . . . . . in java.lang.Thread

        Caused by EncryptionOperationNotPossibleException: null
        ->> 1169 | decrypt                   in org.jasypt.encryption.pbe.StandardPBEByteEncryptor
        - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
        |    450 | getInputStream            in org.rundeck.plugin.encryption.JasyptEncryptionConverterPlugin$DecryptStream
        |     42 | getInputStream . . . . .  in com.dtolabs.rundeck.core.storage.BaseStreamResource
        |    462 | loadProjectConfigResource in rundeck.services.ProjectManagerService
        |    700 | $tt__loadProjectConfig .  in     ''
        |     94 | doInTransaction           in grails.gorm.transactions.GrailsTransactionTemplate$2
        |    140 | execute . . . . . . . . . in org.springframework.transaction.support.TransactionTemplate
        |     91 | execute                   in grails.gorm.transactions.GrailsTransactionTemplate
        |    148 | getProjectConfig . . . .  in com.dtolabs.rundeck.server.projects.RundeckProject
        |     56 | getProperty               in     ''
        |    248 | doCall . . . . . . . . .  in rundeck.services.FrameworkService$_projectLabels_closure2
        |    246 | projectLabels             in rundeck.services.FrameworkService
        |    326 | refreshSessionProjects .  in     ''
        |   2270 | $tt__home                 in rundeck.controllers.MenuController
        |     94 | doInTransaction . . . . . in grails.gorm.transactions.GrailsTransactionTemplate$2
        |    140 | execute                   in org.springframework.transaction.support.TransactionTemplate
        |     91 | execute . . . . . . . . . in grails.gorm.transactions.GrailsTransactionTemplate
        |    223 | invoke                    in org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker
        |    188 | invoke . . . . . . . . .  in org.grails.core.DefaultGrailsControllerClass
        |     90 | handle                    in org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter
        |   1040 | doDispatch . . . . . . .  in org.springframework.web.servlet.DispatcherServlet
        |    943 | doService                 in     ''
        |   1006 | processRequest . . . . .  in org.springframework.web.servlet.FrameworkServlet
        |    898 | doGet                     in     ''
        |    883 | service . . . . . . . . . in     ''
        |    755 | handle                    in org.eclipse.jetty.servlet.ServletHolder
        |   1617 | doFilter . . . . . . . .  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     96 | run                       in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter$1
        |    422 | doAs . . . . . . . . . .  in javax.security.auth.Subject
        |    105 | doFilter                  in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter
        |   1604 | doFilter . . . . . . . .  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     67 | doFilterInternal          in org.rundeck.grails.plugins.securityheaders.RundeckSecurityHeadersFilter
        |   1604 | doFilter . . . . . . . .  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |    320 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    127 | invoke . . . . . . . . .  in org.springframework.security.web.access.intercept.FilterSecurityInterceptor
        |     91 | doFilter                  in     ''
        |    334 | doFilter . . . . . . . .  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    119 | doFilter                  in org.springframework.security.web.access.ExceptionTranslationFilter
        |     64 | doFilter . . . . . . . .  in grails.plugin.springsecurity.web.UpdateRequestContextHolderExceptionTranslationFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     54 | doFilter . . . . . . . .  in grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    158 | doFilter . . . . . . . .  in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    170 | doFilter . . . . . . . .  in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     96 | run . . . . . . . . . . . in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter$1
        |    422 | doAs                      in javax.security.auth.Subject
        |    105 | doFilter . . . . . . . .  in org.springframework.security.web.jaasapi.JaasApiIntegrationFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    200 | doFilter . . . . . . . .  in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     64 | doFilter . . . . . . . .  in grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    105 | doFilter . . . . . . . .  in org.springframework.security.web.context.SecurityContextPersistenceFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |     58 | doFilter . . . . . . . .  in grails.plugin.springsecurity.web.SecurityRequestHolderFilter
        |    334 | doFilter                  in org.springframework.security.web.FilterChainProxy$VirtualFilterChain
        |    215 | doFilterInternal . . . .  in org.springframework.security.web.FilterChainProxy
        |    178 | doFilter                  in     ''
        |   1604 | doFilter . . . . . . . .  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     77 | doFilterInternal          in org.grails.web.servlet.mvc.GrailsWebRequestFilter
        |   1604 | doFilter . . . . . . . .  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |     67 | doFilterInternal          in org.grails.web.filters.HiddenHttpMethodFilter
        |   1604 | doFilter . . . . . . . .  in org.eclipse.jetty.servlet.ServletHandler$CachedChain
        |    545 | doHandle                  in org.eclipse.jetty.servlet.ServletHandler
        |    143 | handle . . . . . . . . .  in org.eclipse.jetty.server.handler.ScopedHandler
        |    590 | handle                    in org.eclipse.jetty.security.SecurityHandler
        |    127 | handle . . . . . . . . .  in org.eclipse.jetty.server.handler.HandlerWrapper
        |    235 | nextHandle                in org.eclipse.jetty.server.handler.ScopedHandler
        |   1607 | doHandle . . . . . . . .  in org.eclipse.jetty.server.session.SessionHandler
        |    233 | nextHandle                in org.eclipse.jetty.server.handler.ScopedHandler
        |   1297 | doHandle . . . . . . . .  in org.eclipse.jetty.server.handler.ContextHandler
        |    188 | nextScope                 in org.eclipse.jetty.server.handler.ScopedHandler
        |    485 | doScope . . . . . . . . . in org.eclipse.jetty.servlet.ServletHandler
        |   1577 | doScope                   in org.eclipse.jetty.server.session.SessionHandler
        |    186 | nextScope . . . . . . . . in org.eclipse.jetty.server.handler.ScopedHandler
        |   1212 | doScope                   in org.eclipse.jetty.server.handler.ContextHandler
        |    141 | handle . . . . . . . . .  in org.eclipse.jetty.server.handler.ScopedHandler
        |    127 | handle                    in org.eclipse.jetty.server.handler.HandlerWrapper
        |    500 | handle . . . . . . . . .  in org.eclipse.jetty.server.Server
        |    383 | lambda$handle$1           in org.eclipse.jetty.server.HttpChannel
        |    547 | dispatch . . . . . . . .  in     ''
        |    375 | handle                    in     ''
        |    270 | onFillable . . . . . . .  in org.eclipse.jetty.server.HttpConnection
        |    311 | succeeded                 in org.eclipse.jetty.io.AbstractConnection$ReadCallback
        |    103 | fillable . . . . . . . .  in org.eclipse.jetty.io.FillInterest
        |    117 | run                       in org.eclipse.jetty.io.ChannelEndPoint$2
        |    336 | runTask . . . . . . . . . in org.eclipse.jetty.util.thread.strategy.EatWhatYouKill
        |    313 | doProduce                 in     ''
        |    171 | tryProduce . . . . . . .  in     ''
        |    129 | run                       in     ''
        |    388 | run . . . . . . . . . . . in org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread
        |    806 | runJob                    in org.eclipse.jetty.util.thread.QueuedThreadPool
        |    938 | run . . . . . . . . . . . in org.eclipse.jetty.util.thread.QueuedThreadPool$Runner
        ^    748 | run                       in java.lang.Thread

[jjethwa]

Hi @zealouspatcat

When did this occur? You might need to change it as suggested in the second issue you linked to. I believe that should force it to use the right password if it has been set properly.

[zealouspatcat]

This happened today after I created another project/node that ssh'd into my host machine. I made the following changes:

docker-compose.yml

        RUNDECK_STORAGE_CONVERTER_1_CONFIG_PASSWORD: ${RUNDECK_STORAGE_PASSWORD}
        RUNDECK_CONFIG_STORAGE_CONVERTER_1_CONFIG_PASSWORD: ${RUNDECK_STORAGE_PASSWORD}

Main Machine: rundeck-config.properties

  dataSource.url = jdbc:mysql://production_mariadb/rundeckdb?autoReconnect=true

  # Encryption for key storage
  rundeck.storage.provider.1.type=db
  rundeck.storage.provider.1.path=keys

  rundeck.storage.converter.1.type=jasypt-encryption
  rundeck.storage.converter.1.path=keys
  rundeck.storage.converter.1.config.encryptorType=custom
  rundeck.storage.converter.1.config.password=default.encryption.password
  rundeck.storage.converter.1.config.algorithm=PBEWITHSHA256AND128BITAES-CBC-BC
  rundeck.storage.converter.1.config.provider=BC

  # Encryption for project config storage
  rundeck.projectsStorageType=db

  rundeck.config.storage.converter.1.type=jasypt-encryption
  rundeck.config.storage.converter.1.path=projects
  rundeck.config.storage.converter.1.config.password=default.encryption.password
  rundeck.config.storage.converter.1.config.encryptorType=custom
  rundeck.config.storage.converter.1.config.algorithm=PBEWITHSHA256AND128BITAES-CBC-BC
  rundeck.config.storage.converter.1.config.provider=BC

  rundeck.feature.repository.enabled=true

  dataSource.driverClassName = org.mariadb.jdbc.Driver

  dataSource.username = rundeck

  dataSource.password = dca49c6525300392

Container:

  dataSource.url = jdbc:mysql://production_mariadb/rundeckdb?autoReconnect=true

  # Encryption for key storage
  rundeck.storage.provider.1.type=db
  rundeck.storage.provider.1.path=keys

  rundeck.storage.converter.1.type=jasypt-encryption
  rundeck.storage.converter.1.path=keys
  rundeck.storage.converter.1.config.encryptorType=custom
  rundeck.storage.converter.1.config.password=477ad7f7c775a9de
  rundeck.storage.converter.1.config.algorithm=PBEWITHSHA256AND128BITAES-CBC-BC
  rundeck.storage.converter.1.config.provider=BC

  # Encryption for project config storage
  rundeck.projectsStorageType=db

  rundeck.config.storage.converter.1.type=jasypt-encryption
  rundeck.config.storage.converter.1.path=projects
  rundeck.config.storage.converter.1.config.password=477ad7f7c775a9de
  rundeck.config.storage.converter.1.config.encryptorType=custom
  rundeck.config.storage.converter.1.config.algorithm=PBEWITHSHA256AND128BITAES-CBC-BC
  rundeck.config.storage.converter.1.config.provider=BC

  rundeck.feature.repository.enabled=true

  dataSource.driverClassName = org.mariadb.jdbc.Driver

I'm going to check if it has anything to do with my key certs :/

[jjethwa]

Oh, you're setting up encryption for the storage side. I haven't looked at it yet, but looks like the configuration is here: https://docs.rundeck.com/administration/configuration/storage-facility

From the logs, looks like rundeck is having a problem encrypting and decrypting. Is there an error when you try to add a new key?

[zealouspatcat]

Well, I made those adjustments based on the ticket that I linked, the 'solution' was to set those passwords, so I did that.

Doesn't look like I'm having issues adding new keys, but I also don't think it's actually working. I checked my java variables and they all look good

[jjethwa]

Hmm, so looks like it's either not encrypting it correctly when adding or not able to decrypt it if added successfully.

[dev-rowbot]

Hi,

I have managed to isolate the issue to a change in the way the rundeck.storage.converter.1.config.password is generated. I encountered the same issue where updating the container to a newer version caused a failure.

In my testing I created a clean Rundeck 3.3.4 container and uploaded a few jobs, the generated rundeck-properties.config file showed:

rundeck.storage.converter.1.type=jasypt-encryption
rundeck.storage.converter.1.path=keys
rundeck.storage.converter.1.config.encryptorType=custom
rundeck.storage.converter.1.config.password=9d634491ac23aed4

I then created a clean Rundeck 3.4.1 container and uploaded a few jobs, the generated rundeck-properties.config file showed:

rundeck.storage.converter.1.type=jasypt-encryption
rundeck.storage.converter.1.path=keys
rundeck.storage.converter.1.config.encryptorType=custom
rundeck.storage.converter.1.config.password=232a17716fabf3ec

I then ran a clean 3.3.4 container again, uploaded a few jobs, stopped the container and ran Rundeck 3.4.1 reusing the data volumes from 3.3.4. I hit the exact same 500 Internal Server error and I noticed that the generated rundeck-properties.config file showed:

rundeck.storage.converter.1.config.password=232a17716fabf3ec

I changed this to the password from the 3.3.4 instance and rebooted:

rundeck.storage.converter.1.config.password=9d634491ac23aed4

This resolves the issue.

This does of course make upgrading Rundeck and keeping job history a bit of a challenge. Do you know how the config.password is generated? Maybe there is a configuration change that can be applied to generate it the same way as before and avoid this issue

[jjethwa]

Hi @dev-rowbot

Thanks so much for the detailed report. If you're not using a volume for /etc/rundeck, the container will use the default one found here: https://github.com/jjethwa/rundeck/blob/master/content/opt/rundeck-defaults/rundeck-config.properties

The storage section is then modified here: https://github.com/jjethwa/rundeck/blob/master/content/opt/run#L285

Seems that we might need to add a section for modifying rundeck.storage.converter.1.config.password as well? Might need some of the other properties added too. What do you think?

[dev-rowbot]

@jjethwa yes I agree - a section for modifying the config.password would make sense to me. It would have to cater for

rundeck.storage.converter.1.config.password
rundeck.config.storage.converter.1.config.password

At least this way you could guarantee that the password is generated the same way across redeploys. I assume the salt or generation method has changed between Rundeck versions.

I had not thought of using a volume for /etc/rundeck. We've been using Rundeck for years and have managed to not hit any issues without using a volume. I will test again using a volume and let you know how it goes

[dev-rowbot]

@jjethwa - Good news! Using a volume resolves the issue. I can upgrade directly from 3.3.4 to 3.4.1.

For running containers with no volume you I will need to create a volume and copy the data there before upgrading to the new version.

[jjethwa]

That's great news @dev-rowbot

I think using the volume would be the safest bet for everyone as your configs will be stored outside of the container and makes it easier to manage. I'll check out the Rundeck documentation on the storage converter password as we would need to make sure it's a secure and different value for every install for security. Let me see what I can dig up 😄

[dev-rowbot]

@jjethwa - I may have spoken too soon 🙈 I tried this on an active container with history and it got me past the first issue but then when I tried to open a Project I hit this error...

I have not dug into it further but I think I will need to rerun my tests from above and create some actually job history to see if it is then possible to upgrade.

Edit: Adding the searchable version of the screenshot

Error 500: Internal Server Error

URI
    /project/Configuration/job/show/2f1a9b84-50ac-41bd-a95e-3fa4adfda7a8
Class
    java.sql.SQLException
Message
    null
Caused by
    Unknown column 'workflowst1_.child_nodes' in 'field list'

[jjethwa]

Hi @dev-rowbot

That's a new one for me. Are you using a volume for the database or an external database?

[anwareset]

Hi @jjethwa I use external database, and got the same issue.

[jjethwa]

Hi @anwareset

Can you provide the stack trace?

[anwareset]

I'm so sorry, i handled it with a nuke (fresh install). No logs available for it. I suspect that this is due to something when we build a new image and mount it on an existing deployment.

[jjethwa]

Thanks for the info, @anwareset

Really wish I was able to reproduce it locally 😮‍💨

[anwareset]

You can reproduce it by this:

• Step 1: Build rundeck image.

• Step 2: Clean run image with external DB (MySQL), run with kubernetes or just docker. Also use needed env variables.

• Step 3: Create projects and jobs, try to execute it for a test.

• Step 4: Build rundeck image, again. With exactly same Dockerfile.

• Step 5: Run this new image with existing external DB, existing configs.

You'll get something like this when open rundeck UI:

Error 500 - org.jasypt.exceptions.EncryptionOperationNotPossibleException

[jjethwa]

Thanks, @anwareset

I'll run through those steps and see if I can reproduce it, really appreciate it 👍🏽

[jjethwa]

I have reproduced the issue thanks to @anwareset and have identified the issue. Project storage was moved to the database several releases ago, but encryption has also been added. It seems Rundeck will autogenerate the configs (including an encryption password), but this is lost when the container is stopped and new configs generated when a new container is started. This would be mitigated by using a volume for /etc/rundeck so configs persist. I'll need to introduce a new environment variable for the storage password so it works when a volume is not used.

[jjethwa]

Hi @anwareset and @zealouspatcat

I'm pushing latest and re-pushing 4.8.0 with the changes. When starting the rundeck container, you will need to use the new environment variable RUNDECK_STORAGE_PASSWORD. Your current Rundeck container already generated a value for the password, so I would suggest grabbing it by docker exec'ing into your running container and grepping the value of rundeck.config.storage.converter.1.config.password in /etc/rundeck/rundeck-config.properties

Restarting using an updated image should no longer be a problem 🤞🏽

[jjethwa]

Updated the container to default to db storage as it looks like file storage is on the way to deprecation. Also, modified the logic to only set the password when RUNDECK_STORAGE_PASSWORD is set

[anwareset]

I do reproduce step to existing data with new image

1. Without RUNDECK_STORAGE_PASSWORD. The error occurs.

2. With RUNDECK_STORAGE_PASSWORD from existing rundeck.config.storage.converter.1.config.password in /etc/rundeck/rundeck-config.properties. Seems run well.

Well done @jjethwa and thanks @dev-rowbot Guess we need to close this issue.

[jjethwa]

Thanks so much for confirming, @anwareset 😃