Container is broken on restart

Brozowski commented 5 years ago

I've created a docker with tiredofit/lemonldap:2.0-latest image. On first start, everything is ok. But if i restart the container :

Fail2ban can't start because .pid and .sock already exists
FastCGI Server start & restart & restart & restart ... Infinitly

I disable Fail2ban using env var, but the FastCGI Server is still restarting ...

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3621)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3626)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3631)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3636)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3641)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3646)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3651)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3656)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3661)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3666)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3671)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 3676)

For restart container, i need to remove it and re-run. Any idea ?

tiredofit commented 5 years ago

Hi, sorry about this. I am away out of town right now, but I have a fix for this I just haven't kept the public repo up to date. I'm also going to be removing Fail2ban from the 2.x branch as there is a similar function that is included in the 2.x branches. I'm back in the office on Thursday.

Brozowski commented 5 years ago

Great !

An other question, not related on this problem. How you manage to protect WebApp access with LemonLDAP since it's running in container ? You can't add Vhost in container's Nginx config (on recreate it's lost) ...

tiredofit commented 5 years ago

I use a seperate handler container on each server, which talks to the main portal server via rest (2.x) or SOAP (1.9). What are you trying to protect? I dont make any modifications to the LLNG server or handler once they are up and running, it happens on the other applications/containers nginx config files.

tiredofit commented 5 years ago

New build submitted that matches my private repository. This one also removes MongoDB temporarily as Alpine has removed it from their repos and I'm trying to find an effective way to build and reutilize. Let me know if this works. This is also 2.0.3 which fixes a tonne of issues in the 2.x branch.

tiredofit commented 5 years ago

Also submitted new build that might fix your fail2ban issue? Basically upon each container startup it just wipes the lock and pid files.

Brozowski commented 5 years ago

I use a seperate handler container on each server, which talks to the main portal server via rest (2.x) or SOAP (1.9). What are you trying to protect? I dont make any modifications to the LLNG server or handler once they are up and running, it happens on the other applications/containers nginx config files.

What I want (but don't know if it's possible) :

Internet --> Server Plesk + Docker (listening port 80 and 443) --> Web App (docker or hosted by plesk) --> SSO Auth (docker LemonLDAP, if no session)

Via reverse proxy (by Plesk VHost) I have portal and manager working. But I don't know how configure handler on my Plesk Vhost for protecting it by SSO.

Also submitted new build that might fix your fail2ban issue? Basically upon each container startup it just wipes the lock and pid files.

I'll try ! Thx

Brozowski commented 5 years ago

New build submitted that matches my private repository. This one also removes MongoDB temporarily as Alpine has removed it from their repos and I'm trying to find an effective way to build and reutilize. Let me know if this works. This is also 2.0.3 which fixes a tonne of issues in the 2.x branch.

Still not working. After a Docker restart, FastCGI server is starting (and exit, obviously) infinitly

** [lemonldap-hander-socket] Enabling Handler TCP expose functionality. Socket listens at port 2884
[cont-init.d] 20-llng-fastcgi-server: exited 0.
[cont-init.d] 99-container-init: executing... 
[cont-init.d] 99-container-init: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 976)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 981)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 986)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 991)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 996)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1001)

Also submitted new build that might fix your fail2ban issue? Basically upon each container startup it just wipes the lock and pid files.

Working !

Brozowski commented 5 years ago

Fun fact, coudot/lemonldap-ng image has almost same error ... It's a LemonLDAP error ?

tiredofit commented 5 years ago

Perhaps. Can you take a peek in any of the llng logs, or even try to get into the container and execute the /etc/s6/services/20-llng-fastcgi-server and see what the output reveals?

It may very well be a configuration file issue that might have crept in.

Brozowski commented 5 years ago

Perhaps. Can you take a peek in any of the llng logs, or even try to get into the container and execute the /etc/s6/services/20-llng-fastcgi-server and see what the output reveals?

Manual execution does not log more than docker logs ...

It may very well be a configuration file issue that might have crept in.

Maybe, but i don't see why. I just follow instructions. But if i'm the only one whith this error, indeed, my config is probably the cause.

Brozowski commented 5 years ago

Full start sequence :

[s6-init] making user provided files available at /var/run/s6/etc...

** [zabbix] Starting Zabbix Agent

** [lemonldap] Starting LemonLDAP FastCGI Server
** [fail2ban] Starting fail2ban
** [rsyslog] Starting Syslog

** [lemonldap] Starting nginx
** [cron] Starting cron
exited 0.
rsyslogd: ID for user 'nginx' could not be found or error [v8.40.0 try https://www.rsyslog.com/e/3003 ]
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 01-s6: applying... 
[fix-attrs.d] 01-s6: exited 0.
[fix-attrs.d] 02-zabbix: applying... 
[fix-attrs.d] 02-zabbix: exited 0.
[fix-attrs.d] 03-logrotate: applying... 
[fix-attrs.d] 03-logrotate: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-hosts: executing... 
[cont-init.d] 00-hosts: exited 0.
[cont-init.d] 01-permissions: executing... 
[cont-init.d] 01-permissions: exited 0.
[cont-init.d] 02-zabbix: executing... 
**** [zabbix] Disabling Zabbix
[cont-init.d] 02-zabbix: exited 0.
[cont-init.d] 03-cron: executing... 
**** [cron] Enabling Cron
[cont-init.d] 03-cron: exited 0.
[cont-init.d] 04-smtp: executing... 
**** [smtp] Disabling SMTP Features
[cont-init.d] 04-smtp: exited 0.
[cont-init.d] 05-fail2ban: executing... 
[cont-init.d] 05-fail2ban: exited 0.
[cont-init.d] 06-rsyslog: executing... 
** [rsyslog] Disabling Syslog
[cont-init.d] 06-rsyslog: exited 0.
[cont-init.d] 09-webserver: executing... 
[cont-init.d] 09-webserver: exited 0.
[cont-init.d] 10-lemonldap: executing... 
** [lemonldap] Custom Files Found, inserting into image overtop of sourcecode..
cp: can't stat '/assets/custom/*': No such file or directory
Server ready
[cont-init.d] 10-lemonldap: exited 0.
[cont-init.d] 20-llng-fastcgi-server: executing... 
** [lemonldap-hander-socket] Enabling Handler TCP expose functionality. Socket listens at port 2884
[cont-init.d] 20-llng-fastcgi-server: exited 0.
[cont-init.d] 99-container-init: executing... 
[cont-init.d] 99-container-init: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
FastCGI daemon started (pid 971)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 976)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 981)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 986)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 991)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 996)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1001)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1006)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1011)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1016)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1021)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1026)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1031)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1036)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1041)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1046)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1051)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1056)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1061)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1066)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1071)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1076)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1081)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1086)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1091)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1096)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1101)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1106)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1111)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1116)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1121)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1126)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1131)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1136)

** [lemonldap] Starting LemonLDAP FastCGI Server
FastCGI daemon started (pid 1141)

tiredofit commented 5 years ago

You are right, that doesn't tell us anything. This is with a brand new install?

Lets try this: Change your environment variables to the following

DEBUG_MODE=TRUE SETUP_TYPE=MANUAL

Then edit your ./data/etc/lemonldap-ng/lemonldap-ng.ini to this:

[all]

logger     = Lemonldap::NG::Common::Logger::Std
userLogger = Lemonldap::NG::Common::Logger::Std
logLevel = warn
globalSessionStorage=Cache::FileCache
globalSessionStorageOptions={'namespace' => 'lemonldap-ng-sessions','default_expires_in' => 600, 'directory_umask' => '007', 'cache_root' => '/tmp', 'cache_depth' => 3, }
globalStorage = Apache::Session::File
globalStorageOptions = { 'Directory' => '/var/lib/lemonldap-ng/sessions/', 'LockDirectory' => '/var/lib/lemonldap-ng/sessions/lock/', }

[configuration]
type = File
dirName = /var/lib/lemonldap-ng/conf

localStorage=Cache::FileCache
localStorageOptions={'namespace' => 'lemonldap-ng-config','default_expires_in' => 600, 'directory_umask' => '007', 'cache_root' => '/tmp', 'cache_depth' => 0, }

[portal]
staticPrefix = /static
templateDir = /usr/share/lemonldap-ng/portal/templates
languages = en

[handler]
useRedirectOnError = 1
status = 1

[manager]
protection = manager
staticPrefix = /static
templateDir = /usr/share/lemonldap-ng/manager/templates
languages = en
enabledModules = conf, sessions, 2ndFA

and restart, that's a stripped down configuration I use.

Brozowski commented 5 years ago

You are right, that doesn't tell us anything. This is with a brand new install?

Lets try this: Change your environment variables to the following

DEBUG_MODE=TRUE SETUP_TYPE=MANUAL

Then edit your ./data/etc/lemonldap-ng/lemonldap-ng.ini to this:

[all]

logger     = Lemonldap::NG::Common::Logger::Std
userLogger = Lemonldap::NG::Common::Logger::Std
logLevel = warn
globalSessionStorage=Cache::FileCache
globalSessionStorageOptions={'namespace' => 'lemonldap-ng-sessions','default_expires_in' => 600, 'directory_umask' => '007', 'cache_root' => '/tmp', 'cache_depth' => 3, }
globalStorage = Apache::Session::File
globalStorageOptions = { 'Directory' => '/var/lib/lemonldap-ng/sessions/', 'LockDirectory' => '/var/lib/lemonldap-ng/sessions/lock/', }

[configuration]
type = File
dirName = /var/lib/lemonldap-ng/conf

localStorage=Cache::FileCache
localStorageOptions={'namespace' => 'lemonldap-ng-config','default_expires_in' => 600, 'directory_umask' => '007', 'cache_root' => '/tmp', 'cache_depth' => 0, }

[portal]
staticPrefix = /static
templateDir = /usr/share/lemonldap-ng/portal/templates
languages = en

[handler]
useRedirectOnError = 1
status = 1

[manager]
protection = manager
staticPrefix = /static
templateDir = /usr/share/lemonldap-ng/manager/templates
languages = en
enabledModules = conf, sessions, 2ndFA

and restart, that's a stripped down configuration I use.

On each test, I remove container and prune volumes. The first start is OK. But even without editing any config, after a container restart, FastCGI server won't start. Same result with your config file and a manual editing. Am I cursed ?

tiredofit commented 5 years ago

I wouldn't say you are cursed, I'm actually going to raise my hand here and bet I don't fully understand 2.0 myself. I had it running in production for approximately 2 weeks and had to pull back to 1.9 due to some bugs in the application code, and some quirks I wasn't able to fully figure out. I would be willing to get together and poke around with you on your server to a) satisfy my curiosity b) fix the image if its related to me c) get you up and running.

Brozowski commented 5 years ago

This is the way I create my container :

docker run -d --name="lemonldap" -p 8080:80 -p 2884:2884 \
--restart=always \
--env SETUP_TYPE="AUTO" \
--env MODE="MASTER" \
--env CONFIG_TYPE="FILE" \
--env DOMAIN_NAME="exemple.fr" \
--env MANAGER_HOSTNAME="manager.exemple.fr" \
--env PORTAL_HOSTNAME="portail.exemple.fr" \
--env HANDLER_HOSTNAME="handler.exemple.fr" \
--env TEST_HOSTNAME="test.exemple.fr" \
--env ENABLE_FAIL2BAN="FALSE" \
--env ENABLE_SMTP="FALSE" \
--env ENABLE_ZABBIX="FALSE" \
-v lemonldap_core:/etc/lemonldap-ng \
-v lemonldap_conf:/var/lib/lemonldap-ng/conf \
-v lemonldap_sessions:/var/lib/lemonldap-ng/sessions \
-v lemonldap_psessions:/var/lib/lemonldap-ng/psessions \
-v lemonldap_assets:/assets/custom \
-v lemonldap_logs:/www/logs \
 tiredofit/lemonldap:2.0-latest

If you try yourself with this, can you reproduce my problem ?

coudot commented 5 years ago

Hello, @maxbes has fixed it on the official docker image: https://github.com/LemonLDAPNG/lemonldap-ng-docker/pull/25

You can try this patch on your side

tiredofit / docker-lemonldap

Container is broken on restart #4