PHP Fatal error: Uncaught mysqli_sql_exception: Too many connections

[aleks001]

Steps to reproduce

This happens in the background there is not a set of of individual steps I can complete. What I can see is it is generating a tone of session in the mysql database from the teampass host till it hits the max_connections limit, it doesn't seem to matter how many users there are on the system and it will generate an infinite amount of sessions it seems, increase the cap doesn't help it just reaches it again. My guess is that it's something to do with the cron system regarding the users.

It seems to just be spawning sessions that do nothing. This is definetly linked to the cron scheduler, if I disable the cron job it stops.

Expected behaviour

Shouldn't be creating so many sessions.

Actual behaviour

The application crashes periodically.

Server configuration

Operating system: docker php-apache 8.1 image

Web server: docker php-apache 8.1 image

Database: mysql 8.0.33

PHP version: 8.1

Teampass version: 3.0.10.72

Teampass configuration file: tp.config.zip

Client configuration Browser: Chrome

Operating system: Windows 11

Web server error log

[Tue Oct 31 09:02:48.248876 2023] [php:error] [pid 2649675] [client 10.89.0.21:48906] PHP Fatal error:  Uncaught mysqli_sql_exception: Too many connections in /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php:192\nStack trace:\n#0 /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php(192): mysqli->real_connect()\n#1 /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php(648): MeekroDB->get()\n#2 /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php(637): MeekroDB->queryHelper()\n#3 /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php(638): MeekroDB->prependCall()\n#4 /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php(753): MeekroDB->query()\n#5 /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php(66): MeekroDB->queryFirstRow()\n#6 /var/www/html/teampass/sources/checks.php(178): DB::queryFirstRow()\n#7 /var/www/html/teampass/sources/items.queries.php(43): checkUser()\n#8 {main}\n  thrown in /var/www/html/teampass/includes/libraries/Database/Meekrodb/db.class.php on line 192, referer: https://host/teampass/index.php?page=items

#### Log from the web-browser developer console (CTRL + SHIFT + i)

None Insert the log here and especially the answer of the query that failed.

None, something in the background is doing this

[whosht]

Affects me as well. In mariadb I also see that teampass has a lot of open connections when doing "show processlist;" I am at 64c0768c92c8bb19f2a308ca318b37f699b62ef1 (3.0.10.81).

[whosht]

This seems to have something to do with the background Tasks. I see a lot of processes when they run (100 - as many as limitted by the database):

~# ps -Af | grep teampass
www-data 1535004       1  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___user_keys_creation.php
www-data 1535012       1  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___items_handler.php
www-data 1535021 1535004  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___user_keys_creation.php
www-data 1535022 1535012  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___items_handler.php
www-data 1535023 1535021  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___user_keys_creation.php
www-data 1535024 1535022  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___items_handler.php
www-data 1535030 1535023  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___user_keys_creation.php
www-data 1535031 1535024  0 17:00 ?        00:00:00 /usr/bin/php8.2 /var/www/html/teampass/scripts/background_tasks___items_handler.php

and so on.

[Arash9703]

Linked with the background crontab !?

The grep u are doing is it in mysql ?

[whosht]

@Arash9703: The background tasks are running through the cron job - but this is not important as I can reproduce the same behaviour by running it manually e.g. su -mc "/usr/bin/php sources/scheduler.php" www-data or by calling key management only with su -mc "/usr/bin/php scripts/background_tasks___user_keys_creation.php" www-data (The latter was used a few weeks ago to check whats wrong and to finish user creation due to a misbehaviour which results in an entry in the database being left with a wrong value, when timeout for background task execution exceeded - I haven't elaborated that good enough yet to open a new issue).

ps -Af | grep teampass is run in a shell (list Linux processes and filter everything with teampass string in it)

[aleks001]

This most likely affects everyone and most people are just not noticing it. Probably linked with the amount of passwords you have or users. It is quite a serious issue that needs addressing as it will cause other processes to error out randomly when the database limit is hit.

[Arash9703]

I will check this out in mine , and get back to you.

[aleks001]

I should have mentioned but they are not constantly there. You have to keep running those commands around the time the scheduler runs to catch it. I usually just keep running it every 10-20 seconds for a minute or two to catch it happening.

[whosht]

@aleks001: the schedular has a granularity of minutes (from cron). The time the tasks run is configurable. The default is to run some tasks every minute, some every 2 minutes, one every 5 minutes, 5 Tasks are undefined (e.g. personal folder creation). How long they run is configurable too (default 300 seconds, x items per run).

@Arash9703: Did you check right after cron startet e.g. user creation? This is best reproducible for me if user creation process is configured to run every minute.

[aleks001]

@whosht I've tried messing with a lot of those settings and nothing seems to help or change what I'm seeing. So I'm not sure exactly what it is that's causing this.

[Arash9703]

@whosht , I did created a test local user and loged in in order to synchronise vault clés to the serveur, "in the meantime i kept running the shell grep teampass" and didn't faced it neither in MySQL neither on debian server.

[whosht]

@Arash9703: Do you have any hints where I can start debugging in the scripts? E.g. in scripts/background_tasks___user_keys_creation.php

When I look at the process list excerpt, I posted, I see the first process of scripts/background_tasks_user_keys_creation.php has a parent PID of 1 which can be explained because it is started by the scheduler.php itself which has already been ended/terminated. The next scripts/backgroundtasksuser_keys_creation.php process is started by the first process, the next one by the second and so on. So there must be some place that calls (or forks or whatever) scripts/background_tasks___user_keys_creation.php either iterative or recursively.

[Arash9703]

No clue :( maybe php/usr/bin !

check my task manager config , maybe changing this might help the background task !

[whosht]

@whosht , I did created a test local user and loged in in order to synchronise vault clés to the serveur, "in the meantime i kept running the shell grep teampass" and didn't faced it neither in MySQL neither on debian server.

You have to watch at the right time. You have set user creation task to be executed every 2 minutes. So I guess you have to wait for e.g. 15:22 (even minute) - if you start at 15:21 (odd minute) nothing is run. Regardless if there is a new user or not, you should see at least the number of processes when the scheduler.php is run at the right time (if you do manually you have to wait for the right time as well). This is because scripts/background_tasks___user_keys_creation.php calls itself recursively (line 127 with 64c0768c92c8bb19f2a308ca318b37f699b62ef1) until the timeout (php = 60s or as configured). One more note: you will probably not see that if you are on a newer version than @aleks001 and me.

In b207d0047fcdea02d842ea92af36adaca4fed9c9 there is fix that there is no endless spawning of processes. I haven't updated to a newer version because the last time I tried, the upgrade.php-Process failed (with 958ca2bd41ab3011b8bc91d8626bbcfa2e34e977).

So @aleks001 maybe you should try to update beyond b207d0047fcdea02d842ea92af36adaca4fed9c9 and check if there is still a problem. I would say not because the number of connections is limitted to the number of tasks - not number of spawned processes within the defined time (although I don't know if there is still a bug, keeping a connection to the databases open after the task has been handled and before a new process is spawned).

[Arash9703]

This most likely affects everyone and most people are just not noticing it. Probably linked with the amount of passwords you have or users. It is quite a serious issue that needs addressing as it will cause other processes to error out randomly when the database limit is hit.

It took me a little time, indeed I agree with @whosht and @aleks001 . The issue does exist & as @aleks001 mentionned this could be à little serious as its relie with mysql !

I guess @nilsteampassnet needs to check this out before the latest 3.0.10 release.

@nilsteampassnet pour info :

[nilsteampassnet]

Hi The issue is solved but not yet committed. The next commit is a huge one and I'm testing it. The fix is quite simple, I will post tomorrow morning how to do it manually.

[Arash9703]

@nilsteampassnet I've noticed that disabled Active Directory users can log in to their vault. Is this expected behavior? If so, it might be necessary for Teampass administrators to perform additional checks when employees leave the team.

Teampass currently checks for accounts existence but doesn't verify the account's active status. Should we create an issue to discuss and address this?

Your input would be appreciated.

[cgkelleher88]

@nilsteampassnet did you manage to post the manual fix for this issue - i have the issue on version 3.0.10.67 and still after upgrade to latest. should it be fixed with upgrade?