Login Screen Looping when scale service in Rancher

[rsmartins78]

Hello, I have the following problem, but I do not know who to turn to. I use Rancher with Cattle to orchestrate my Docker containers, but when I scale my service to 2 containers or more, Rundeck is stay in loop on the login screen.

My schema in Rancher.

[jjethwa]

Hi @rsmartins78

It's a bit hard to tell what the problem could be from the diagram. Is the grails.serverURL set to the LB FQDN or CNAME to the LB? Is SSL terminated at the LB? Do you have the LB configured to send the correct headers? http://rundeck.org/docs/administration/configuring-ssl.html#using-an-ssl-terminated-proxy

[rsmartins78]

Is the grails.serverURL set to the LB FQDN or CNAME to the LB? Since the access URL remains the same (via IP) I have not changed the file.

Is SSL terminated at the LB? No, just a simple LB created in Rancher where I set respective ports and redirects

Do you have the LB configured to send the correct headers? No, I'll try with these parameters and let you know soon.

[jjethwa]

Sounds good! Also, see if there's anything interesting in the rundeck log when trying to login 😃

[rsmartins78]

The parameter -Drundeck.jetty.connector.forwarded=true already is set, but without results.

My service.log

INFO  ExecutionUtilService: Execution successful: 482 in project Test
INFO  ExecutionService: updated scheduled Execution
WARNING: HTTPS is not enabled, specify -Drundeck.ssl.config=/var/lib/rundeck/server/config/ssl.properties to enable.
2018-03-26 16:54:28.344:INFO:oejs.Server:main: jetty-9.0.7.v20131107
2018-03-26 16:54:33.347:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2018-03-26 16:54:36.488:INFO:/:main: Initializing Spring root WebApplicationContext
WARNING: HTTPS is not enabled, specify -Drundeck.ssl.config=/var/lib/rundeck/server/config/ssl.properties to enable.
2018-03-26 16:56:41.853:INFO:oejs.Server:main: jetty-9.0.7.v20131107
2018-03-26 16:56:48.578:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2018-03-26 16:56:51.541:INFO:/:main: Initializing Spring root WebApplicationContext
2018-03-26 16:57:19.576:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-183: Login attempts: 3, Hits: 0, Ratio: 0%.
2018-03-26 16:57:19.577:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-183: Cache Eviction for rsmartins.
2018-03-26 16:57:19.579:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-183: Attempting authentication: CN=RAFAEL DA SILVA MARTINS,CN=Users,DC=indproj,DC=com,DC=br
INFO  ProjectManagerService: Loading project definition for Homologacao...
INFO  ProjectManagerService: Loaded project Homologacao in 19ms
INFO  ProjectManagerService: Loading project definition for Manutencao-Rundeck...
INFO  ProjectManagerService: Loaded project Manutencao-Rundeck in 9ms
INFO  ProjectManagerService: Loading project definition for Test...
INFO  ProjectManagerService: Loaded project Test in 4ms
INFO  BootStrap: Starting Rundeck 2.10.8-1 (2018-03-23) ...
INFO  BootStrap: using rdeck.base config property: /var/lib/rundeck
INFO  BootStrap: loaded configuration: /etc/rundeck/framework.properties
WARN  ProjectManagerService: Discovered filesystem project Homologacao, was previously imported, skipping.
WARN  ProjectManagerService: Discovered filesystem project Test, was previously imported, skipping.
WARN  ProjectManagerService: Discovered filesystem project Manutencao-Rundeck, was previously imported, skipping.
INFO  BootStrap: RSS feeds disabled
INFO  BootStrap: Preauthentication is disabled
INFO  BootStrap: Rundeck is ACTIVE: executions can be run.
WARN  BootStrap: The JVM default encoding is not UTF-8: US-ASCII, you may not see output as expected for multibyte locales. Specify -Dfile.encoding=UTF-8 in the JVM options.
INFO  BootStrap: Rundeck startup finished in 2557ms
2018-03-26 16:57:51.163:INFO:/:main: Initializing Spring FrameworkServlet 'grails'
2018-03-26 16:57:51.296:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@f5b6e78{/,file:/var/lib/rundeck/exp/webapp/,AVAILABLE}{/var/lib/rundeck/exp/webapp}
2018-03-26 16:57:51.420:INFO:oejs.ServerConnector:main: Started ServerConnector@106ac5f4{HTTP/1.1}{0.0.0.0:4440}
INFO  ProjectManagerService: Loading project definition for Manutencao-Rundeck...
INFO  ProjectManagerService: Loaded project Manutencao-Rundeck in 384ms
INFO  ScheduledExecutionService: scheduling new job in project Manutencao-Rundeck 3408ecd9-e04f-474b-8089-95c4b064eadd: 1:Auto-Commit
INFO  ScheduledExecutionService: scheduled job 3408ecd9-e04f-474b-8089-95c4b064eadd. next run: Tue Mar 27 06:10:00 BRT 2018
INFO  ScheduledExecutionService: rescheduled job in project Manutencao-Rundeck: 3408ecd9-e04f-474b-8089-95c4b064eadd
INFO  ScheduledExecutionService: scheduling new job in project Manutencao-Rundeck 5e177bad-17bf-44b3-a324-059638f37c32: 2:Backup
INFO  ScheduledExecutionService: scheduled job 5e177bad-17bf-44b3-a324-059638f37c32. next run: Tue Mar 27 06:00:00 BRT 2018
INFO  ScheduledExecutionService: rescheduled job in project Manutencao-Rundeck: 5e177bad-17bf-44b3-a324-059638f37c32
INFO  ProjectManagerService: Loading project definition for Test...
INFO  ProjectManagerService: Loaded project Test in 35ms
INFO  ScheduledExecutionService: scheduling new job in project Test 9ddf8d94-0b89-4aa8-8434-04eebf8be001: 21:Date
INFO  ScheduledExecutionService: scheduled job 9ddf8d94-0b89-4aa8-8434-04eebf8be001. next run: Tue Mar 27 10:00:00 BRT 2018
INFO  ScheduledExecutionService: rescheduled job in project Test: 9ddf8d94-0b89-4aa8-8434-04eebf8be001
INFO  ScheduledExecutionService: scheduling new job in project Test 75e70953-204b-45a8-ac6c-538c249ec80e: 22:DF -H
INFO  ScheduledExecutionService: scheduled job 75e70953-204b-45a8-ac6c-538c249ec80e. next run: Mon Mar 26 17:00:00 BRT 2018
INFO  ScheduledExecutionService: rescheduled job in project Test: 75e70953-204b-45a8-ac6c-538c249ec80e
2018-03-26 16:58:09.434:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-19: Login attempts: 1, Hits: 0, Ratio: 0%.
2018-03-26 16:58:09.452:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-19: Attempting authentication: CN=MYLOGIN,CN=Users,DC=mydomain,DC=com
2018-03-26 16:58:12.084:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-183: Login attempts: 4, Hits: 0, Ratio: 0%.

After each attempt to login, the last line repeats

[rsmartins78]

I realized that occurs erros in scheduled job executions too

rundeck.services.ExecutionServiceException: Job "DF -H" {{Job 75e70953-204b-45a8-ac6c-538c249ec80e}} is currently being executed {{Execution 483}}
    at rundeck.services.ExecutionService.createExecution(ExecutionService.groovy:2236)
    at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:321)
    at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:114)
    at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:93)
    at com.codahale.metrics.Timer.time(Timer.java:99)
    at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:92)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
ERROR JobRunShell: Job Test:DF -H:group1/subgroup1.22:DF -H threw an unhandled Exception: 
rundeck.services.ExecutionServiceException: Job "DF -H" {{Job 75e70953-204b-45a8-ac6c-538c249ec80e}} is currently being executed {{Execution 483}}
    at rundeck.services.ExecutionService.createExecution(ExecutionService.groovy:2236)
    at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:321)
    at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:114)
    at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:93)
    at com.codahale.metrics.Timer.time(Timer.java:99)
    at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:92)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

[jjethwa]

Hi @rsmartins78

Hmm, maybe an issue with the JAAS configs? com.dtolabs.rundeck.jetty.jaas.JettyCachingLdapLoginModule and org.eclipse.jetty.jaas.spi.PropertyFileLoginModule have debug settings, can you try enabling debug, restarting Rundeck and see if the debug statements provide a clue?

[rsmartins78]

The service.log doesn't change

2018-04-02 09:58:50.801:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-14: Login attempts: 1, Hits: 0, Ratio: 0%.
2018-04-02 09:58:50.819:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-14: Attempting authentication: CN=RAFAEL DA SILVA MARTINS,CN=Users,DC=indproj,DC=com,DC=br
2018-04-02 09:58:56.645:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-18: Login attempts: 2, Hits: 0, Ratio: 0%.
2018-04-02 09:58:59.006:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-19: Login attempts: 2, Hits: 0, Ratio: 0%.
2018-04-02 09:59:02.010:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-18: Login attempts: 3, Hits: 1, Ratio: 33.33%.
2018-04-02 09:59:17.055:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-19: Login attempts: 3, Hits: 1, Ratio: 33.33%.
2018-04-02 09:59:20.068:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-18: Login attempts: 4, Hits: 2, Ratio: 50%.
2018-04-02 09:59:21.565:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-19: Login attempts: 4, Hits: 2, Ratio: 50%.
2018-04-02 09:59:23.449:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-18: Login attempts: 5, Hits: 3, Ratio: 60%.
2018-04-02 09:59:24.021:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-19: Login attempts: 5, Hits: 3, Ratio: 60%.
2018-04-02 09:59:24.445:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-18: Login attempts: 6, Hits: 4, Ratio: 66.67%.
2018-04-02 09:59:24.814:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-19: Login attempts: 6, Hits: 4, Ratio: 66.67%.
2018-04-02 09:59:25.183:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-18: Login attempts: 7, Hits: 5, Ratio: 71.43%.
2018-04-02 09:59:25.914:INFO:cdrjj.JettyCachingLdapLoginModule:qtp189568618-14: Login attempts: 7, Hits: 5, Ratio: 71.43%.

it's as if each click on the login button, tries to log into each container and does not do any of them.

[jjethwa]

Hi @rsmartins78

Are you able to login as as the admin user or another local rundeck account in realm.properties?

[rsmartins78]

I've tried with admin account with no success.

[jjethwa]

Have you tried only running one rundeck container? It seems to be an issue with session persistence. Let's see if one container and one backend works at least 😛

[rsmartins78]

In my default scenario, it's just one container each, everything works perfectly, I was just testing the container scalability part, but I think it's a problem with rundeck and not with this image. I will check other cases in depth.

Thank you for your willingness to help me.

[jjethwa]

NP! Sorry I couldn't help. I'm pretty sure it's just a matter of configuring the rundeck instances so they can maintain persistence in an HA setup. Maybe the main Rundeck project will be able to help. Good luck! 😄

[rsmartins78]

If I get any good results, put it here.

Again, thank you for your commitment.

[jjethwa]

Much appreciated, @rsmartins78 😄