glpi-project / glpi

GLPI is a Free Asset and IT Management Software package, Data center management, ITIL Service Desk, licenses tracking and software auditing.
https://glpi-project.org
GNU General Public License v3.0
4.26k stars 1.29k forks source link

Mailgate permanently stuck in "Running" state #17812

Closed NoxWorld2660 closed 1 month ago

NoxWorld2660 commented 1 month ago

Code of Conduct

Is there an existing issue for this?

Version

10.0.16

Bug description

Mailgate is almost constantly stuck in "running" state.

image

I can't find any relevant log in GLPI or PHP despite being in "log all" mode everywhere (php.ini & glpi). It occurs both in CLI and GLPI mode.

I tried to change the execution timing cycle a few times ( 1 min - 5 min - 10 min ) , and i also adapted the crontask (trying to run this in CLI under Windows Server 2019). The bug remains in every configuration i've tried.

I also made a Cron every 5 min to unlock that specific task through glpi console, it doesn't work either. image

Note that my mail server is On-Prem, and i am gathering email through IMAP. The server sometimes seems slow though.

Relevant log output

No response

Page URL

No response

Steps To reproduce

No response

Your GLPI setup information

Informations sur le système, l'installation et la configuration
GLPI 10.0.16 ( => D:\www\glpi)
Installation mode: TARBALL
Current language:fr_FR

Server
 
Operating system: Windows NT SP6004V 10.0 build 17763 (Windows Server 2019) AMD64
PHP 8.2.10 apache2handler (Core, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, apache2handler, bcmath, bz2, calendar,
    ctype, curl, date, dom, exif, fileinfo, filter, gd, hash, iconv, imap, intl, json, ldap, libxml, mbstring, mysqli, mysqlnd,
    openssl, pcre, pdo_mysql, pdo_sqlsrv, random, readline, session, sodium, sqlsrv, standard, tokenizer, xml, xmlreader, xmlwriter,
    zip, zlib)
Setup: max_execution_time="300" memory_limit="512M" post_max_size="40M" safe_mode="" session.save_handler="files"
    upload_max_filesize="20M" disable_functions="" 
Software: Apache/2.4.62 (Win64) PHP/8.2.10 OpenSSL/3.1.6 ()
    Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0
Server Software: MySQL Community Server - GPL
    Server Version: 8.0.35
    Server SQL Mode: STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
    Parameters: glpi_prd@servername123.xxx.xxxxxxxx.xx/glpi_prd
    Host info: servername123.xxx.xxxxxxxx.xx via TCP/IP

PHP version (8.2.10) is supported.
Sessions configuration is OK.
Allocated memory is sufficient.
mysqli extension is installed.
Following extensions are installed: dom, fileinfo, filter, libxml, json, simplexml, xmlreader, xmlwriter.
curl extension is installed.
gd extension is installed.
intl extension is installed.
zlib extension is installed.
The constant SODIUM_CRYPTO_AEAD_XCHACHA20POLY1305_IETF_NPUBBYTES is present.
Database engine version (8.0.35) is supported.
No files from previous GLPI version detected.
The log file has been created successfully.
Write access to D:\www\glpi/files/_cache has been validated.
Write access to D:\www\glpi/files/_cron has been validated.
Write access to D:\www\glpi/files has been validated.
Write access to D:\www\glpi/files/_dumps has been validated.
Write access to D:\www\glpi/files/_graphs has been validated.
Write access to D:\www\glpi/files/_lock has been validated.
Write access to D:\www\glpi/files/_pictures has been validated.
Write access to D:\www\glpi/files/_plugins has been validated.
Write access to D:\www\glpi/files/_rss has been validated.
Write access to D:\www\glpi/files/_sessions has been validated.
Write access to D:\www\glpi/files/_tmp has been validated.
Write access to D:\www\glpi/files/_uploads has been validated.

Web server root directory configuration seems safe.
Sessions configuration is secured.
OS and PHP are relying on 64 bits integers.
exif extension is installed.
ldap extension is installed.
openssl extension is installed.
Following extensions are installed: bz2, Phar, zip.
Zend OPcache extension is installed.
Following extensions are installed: ctype, iconv, mbstring, sodium.
Write access to D:\www\glpi/marketplace has been validated.
Access to timezone database (mysql) is not allowed.

GLPI constants
 
GLPI_ROOT: "D:\\www\\glpi"
GLPI_CONFIG_DIR: "D:\\www\\glpi/config"
GLPI_VAR_DIR: "D:\\www\\glpi/files"
GLPI_MARKETPLACE_DIR: "D:\\www\\glpi/marketplace"
GLPI_USE_CSRF_CHECK: "1"
GLPI_CSRF_EXPIRES: "7200"
GLPI_CSRF_MAX_TOKENS: "100"
GLPI_USE_IDOR_CHECK: "1"
GLPI_IDOR_EXPIRES: "7200"
GLPI_ALLOW_IFRAME_IN_RICH_TEXT: false
GLPI_SERVERSIDE_URL_ALLOWLIST: ["/^(https?|feed):\\/\\/[^@:]+(\\/.*)?$/"]
GLPI_TELEMETRY_URI: "https://telemetry.glpi-project.org"
GLPI_INSTALL_MODE: "TARBALL"
GLPI_NETWORK_MAIL: "glpi@teclib.com"
GLPI_NETWORK_SERVICES: "https://services.glpi-network.com"
GLPI_MARKETPLACE_ALLOW_OVERRIDE: true
GLPI_MARKETPLACE_MANUAL_DOWNLOADS: true
GLPI_USER_AGENT_EXTRA_COMMENTS: ""
GLPI_DISABLE_ONLY_FULL_GROUP_BY_SQL_MODE: "1"
GLPI_AJAX_DASHBOARD: "1"
GLPI_CALDAV_IMPORT_STATE: 0
GLPI_DEMO_MODE: "0"
GLPI_CENTRAL_WARNINGS: "1"
GLPI_TEXT_MAXSIZE: "4000"
GLPI_DOC_DIR: "D:\\www\\glpi/files"
GLPI_CACHE_DIR: "D:\\www\\glpi/files/_cache"
GLPI_CRON_DIR: "D:\\www\\glpi/files/_cron"
GLPI_DUMP_DIR: "D:\\www\\glpi/files/_dumps"
GLPI_GRAPH_DIR: "D:\\www\\glpi/files/_graphs"
GLPI_LOCAL_I18N_DIR: "D:\\www\\glpi/files/_locales"
GLPI_LOCK_DIR: "D:\\www\\glpi/files/_lock"
GLPI_LOG_DIR: "D:\\www\\glpi/files/_log"
GLPI_PICTURE_DIR: "D:\\www\\glpi/files/_pictures"
GLPI_PLUGIN_DOC_DIR: "D:\\www\\glpi/files/_plugins"
GLPI_RSS_DIR: "D:\\www\\glpi/files/_rss"
GLPI_SESSION_DIR: "D:\\www\\glpi/files/_sessions"
GLPI_TMP_DIR: "D:\\www\\glpi/files/_tmp"
GLPI_UPLOAD_DIR: "D:\\www\\glpi/files/_uploads"
GLPI_INVENTORY_DIR: "D:\\www\\glpi/files/_inventories"
GLPI_NETWORK_REGISTRATION_API_URL: "https://services.glpi-network.com/api/registration/"
GLPI_MARKETPLACE_PLUGINS_API_URI: "https://services.glpi-network.com/api/marketplace/"
GLPI_I18N_DIR: "D:\\www\\glpi/locales"
GLPI_VERSION: "10.0.16"
GLPI_SCHEMA_VERSION: "10.0.16@b13256c443dd4fdb27b4a0d3b8fea8caba4dfaa9"
GLPI_MARKETPLACE_PRERELEASES: false
GLPI_MIN_PHP: "7.4.0"
GLPI_MAX_PHP: "8.4.0"
GLPI_YEAR: "2024"

Libraries
 
htmlawed/htmlawed version 1.2.14 in (D:\www\glpi\vendor\htmlawed\htmlawed)
phpmailer/phpmailer version 6.8.0 in (D:\www\glpi\vendor\phpmailer\phpmailer\src)
simplepie/simplepie version 1.5.8 in (D:\www\glpi\vendor\simplepie\simplepie\library)
tecnickcom/tcpdf version 6.7.5 in (D:\www\glpi\vendor\tecnickcom\tcpdf)
michelf/php-markdown in (D:\www\glpi\vendor\michelf\php-markdown\Michelf)
true/punycode in (D:\www\glpi\vendor\true\punycode\src)
iamcal/lib_autolink in (D:\www\glpi\vendor\iamcal\lib_autolink)
sabre/dav in (D:\www\glpi\vendor\sabre\dav\lib\DAV)
sabre/http in (D:\www\glpi\vendor\sabre\http\lib)
sabre/uri in (D:\www\glpi\vendor\sabre\uri\lib)
sabre/vobject in (D:\www\glpi\vendor\sabre\vobject\lib)
laminas/laminas-i18n in (D:\www\glpi\vendor\laminas\laminas-i18n\src)
laminas/laminas-servicemanager in (D:\www\glpi\vendor\laminas\laminas-servicemanager\src)
monolog/monolog in (D:\www\glpi\vendor\monolog\monolog\src\Monolog)
sebastian/diff in (D:\www\glpi\vendor\sebastian\diff\src)
donatj/phpuseragentparser in (D:\www\glpi\vendor\donatj\phpuseragentparser\src\UserAgent)
elvanto/litemoji in (D:\www\glpi\vendor\elvanto\litemoji\src)
symfony/console in (D:\www\glpi\vendor\symfony\console)
scssphp/scssphp in (D:\www\glpi\vendor\scssphp\scssphp\src)
laminas/laminas-mail in (D:\www\glpi\vendor\laminas\laminas-mail\src\Protocol)
laminas/laminas-mime in (D:\www\glpi\vendor\laminas\laminas-mime\src)
rlanvin/php-rrule in (D:\www\glpi\vendor\rlanvin\php-rrule\src)
ramsey/uuid in (D:\www\glpi\vendor\ramsey\uuid\src)
psr/log in (D:\www\glpi\vendor\psr\log\Psr\Log)
psr/simple-cache in (D:\www\glpi\vendor\psr\simple-cache\src)
psr/cache in (D:\www\glpi\vendor\psr\cache\src)
league/csv in (D:\www\glpi\vendor\league\csv\src)
mexitek/phpcolors in (D:\www\glpi\vendor\mexitek\phpcolors\src\Mexitek\PHPColors)
guzzlehttp/guzzle in (D:\www\glpi\vendor\guzzlehttp\guzzle\src)
guzzlehttp/psr7 in (D:\www\glpi\vendor\guzzlehttp\psr7\src)
glpi-project/inventory_format in (D:\www\glpi\vendor\glpi-project\inventory_format\lib\php)
wapmorgan/unified-archive in (D:\www\glpi\vendor\wapmorgan\unified-archive\src)
paragonie/sodium_compat in (D:\www\glpi\vendor\paragonie\sodium_compat\src)
symfony/cache in (D:\www\glpi\vendor\symfony\cache)
html2text/html2text in (D:\www\glpi\vendor\html2text\html2text\src)
symfony/css-selector in (D:\www\glpi\vendor\symfony\css-selector)
symfony/dom-crawler in (D:\www\glpi\vendor\symfony\dom-crawler)
twig/twig in (D:\www\glpi\vendor\twig\twig\src)
twig/string-extra in (D:\www\glpi\vendor\twig\string-extra)
symfony/polyfill-ctype not found
symfony/polyfill-iconv not found
symfony/polyfill-mbstring not found
symfony/polyfill-php80 not found
symfony/polyfill-php81 not found
symfony/polyfill-php82 in (D:\www\glpi\vendor\symfony\polyfill-php82)
league/oauth2-client in (D:\www\glpi\vendor\league\oauth2-client\src\Provider)
league/oauth2-google in (D:\www\glpi\vendor\league\oauth2-google\src\Provider)
thenetworg/oauth2-azure in (D:\www\glpi\vendor\thenetworg\oauth2-azure\src\Provider)

Not active

Notifications
 
Way of sending emails: SMTP (anonymous@smtp-appl.XXXXX.XXXXX.XX)

Plugins list
 
    fields               Name: Champs supplémentaires         Version: 1.20.7     State: Enabled                                 
        Install Method: Manual
    datainjection        Name: Data Injection                 Version: 2.13.1     State: Enabled                                 
        Install Method: Manual
    formcreator          Name: Form Creator                   Version: 2.13.9     State: Enabled                                 
        Install Method: Manual
    tag                  Name: Gestion des tags               Version: 2.11.7     State: Enabled                                 
        Install Method: Manual
    mreporting           Name: Plus de rapports               Version: 1.8.2      State: Enabled                                 
        Install Method: Manual

Anything else?

No response

NoxWorld2660 commented 1 month ago

After checking the database, i noticed the "lastrun" field is updated correctly no matter what even if the task is failing. However the the "state" remains with the value "2" , which according to what i see in the UI, seems to be the value for "Running".

State of cronstasks seems to be :

I therefore digged into the code and tried to find a workaround or solution that would fix the remaining "Running" state issue without impacting anything else and found this : https://github.com/glpi-project/glpi/blob/10.0.16/src/MailCollector.php#L720

The following attempt is a wrong turn, doesn't work. A try-catch block already exists on the Cron , see next comment.

I just added 'state' => 1, in the database update part. After checking, this is not a proper or suitable fix : the collect() method seems to be called for each mailcollector configured. If you have multiple mailcollector it will probably mess things up. But i only have 1 mail collector for now, so i will try if this solves my issue temporarily.

If it works, a slightly cleaner solution (just enough to support multiple mail collectors) would be to put a Try-Catch and that fix in the cronMailgate() foreach loop, and execute it only in case of errors, just to make sure the task status is correct for the next run before you exit the script in case of crashes.

Hope this will help. Il'l keep updating if relevant.

cedric-anne commented 1 month ago

https://github.com/glpi-project/glpi/blob/10.0.16/src/MailCollector.php#L720

I just added 'state' => 1, in the database update part.

This update referes to the MailCollector itself that has no state property. Changing this should not have any impact on the crontask state itself.

If it works, a slightly cleaner solution (just enough to support multiple mail collectors) would be to put a Try-Catch and that fix in the cronMailgate() foreach loop, and execute it only in case of errors, just to make sure the task status is correct for the next run before you exit the script in case of crashes.

There is already a try/catch block to surround every crontask execution, see https://github.com/glpi-project/glpi/blob/308d15fad7f4fa09936e96048ed1206528f01722/src/CronTask.php#L1026-L1048

It may be possible that this try/catch block fails if the error is not recoverable. I am not sure, but I think it could be the case when the error is due to an execution that consume more memory than the configured limit to when the execution time reach the configured timeout. It may also happen if you are facing a fatal error of the PHP engine itself.

Check the cron.log file to see if any issue is reported. Also, you could try to upgrade your PHP version from 8.2.10 to 8.2.23. If your problem is related to a PHP bug, then it may be fixed by this upgrade.

NoxWorld2660 commented 1 month ago

Thank you for your reply.

I still can't see any log about the mailgate anywhere, but i restarted the server and deleted old logs yesterday so, i might just not have crashed yet. I suspect Windows is not helping things to run smoothly either (already had an opcache bug related to windows).

Some part of my php.ini , maybe you can suggest improvements regarding logs or glpi cron :

max_execution_time = 300
memory_limit = 512M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT

I looked in glpi_cronstasklogs , and i can confirm sometimes it's missing the "Action Completed" part. The maximum elapsed time for the cronstasks_id=9 (the only mailgate collection i have configured) is 4.88s for 6 mails at once.

I will try to see if there is a PHP Bug somewhere, but i find it weird that a potential fatal error is not logged anywhere. Do you have any suggestion for further investigation ?

cedric-anne commented 1 month ago

Is there anything in the cron.log file? If this file is empty, then it may means that is is not writable by the webserver. Also, check if logging in files is allowed in Setup > General > System. image

NoxWorld2660 commented 1 month ago

image

I do have a few logs in cron.log, but just " XXXX start " + a timestamp. Example :

2024-09-11 14:58:01 [@servername]
Externe #1 : Démarrage mailgate
cedric-anne commented 1 month ago

I do have a few logs in cron.log, but just " XXXX start " + a timestamp.

So it probably means that the PHP script is literally killed and cannot event log the failure.

NoxWorld2660 commented 1 month ago

Tips : Spacing the running schedule of the Cron / Scheduled task seems to help. I suspect sometimes the Windows Scheduled Task was triggered before the cron was able to finish , either killing the actual running script or queuing a lot of tasks and causing crash after a little while.

@cedric-anne , i only have 3 or 4 GLPI enabled tasks and using CLI , and changed the time frame of some tasks to be run at night. Is there a way to control which task is scheduled by the Cron ?

For example, i would like to do something like this, on the OS level with parameters given to cron.php :

cedric-anne commented 1 month ago

Is there a way to control which task is scheduled by the Cron ?

See https://glpi-user-documentation.readthedocs.io/fr/master/modules/configuration/crontasks.html#force-execution-of-action-from-cli

mailgate corresponds to the name displayed in the Setup > Automatic actions page.

Also, please check if there is something, at Windows level, used to kill your cron script. I already see documentations in the wild that advice to use some tricks to automatically kill the PHP process of the GLPI cron when its execution time is too long. I do not remember if it was done by another scheduled task, an option, ... but anyway, killing the GLPI cron is a bad idea.

I close this issue as it is not a GLPI bug. Indeed, the GLPI cron script has some kind of semaphore logic that is used to prevent parallel executions and has an internal error management that exists to prevent your initial issue (as long as it is not killed by someone else).

github-actions[bot] commented 1 month ago

This issue has been closed as we only track bugs here.

You can get community support on forums or you can consider taking a subscription to get professional support. You can also contact GLPI editor team directly.