search

glpi-project / glpi

GLPI is a Free Asset and IT Management Software package, Data center management, ITIL Service Desk, licenses tracking and software auditing.
https://glpi-project.org
GNU General Public License v3.0
4.36k stars 1.3k forks source link

When ticket created by mailgate, occasionally assigns a already used Ticket ID, and does not actually create ticket or modify the existing one it takes the ID from. #17849

Closed Terrokkar closed 1 week ago

Terrokkar commented 2 months ago

Code of Conduct

Is there an existing issue for this?

Version

10.0.15

Bug description

When ticket is created from mailgate, it sometimes takes a random ticket ID from the past, sends notification with that ID inside (also link to that ticket), but doesnt create the new ticket. Occurence is not tied to specific user, mail content, timeframe or anything else i checked.

Proposed work around of having a cronjob doesnt work either:

This is currently happening in about 5-10% of our tickets, spamming users with bogus mails with links to some random old ticket and having us to manually recheck every email in our receiver.

As a sidenote, our Linux server user and system-wide crontabs were cleared of the cronjob "php /var/www/glpi/front/cron.php".

Only copy of this cronjob exists in www-data users crontab now.

Installed plugins: Mailanalyzer ( tested with it disabled, issue is the same) GLPI Inventory

Relevant log output

No logs refer to these mails, when everything works okay, then the logs appear.

Page URL

n/a

Steps To reproduce

Send mail to mailgate.

Your GLPI setup information

GLPI 10.0.15 ( => /var/www/glpi) Installation mode: TARBALL Current language:en_GB -- Operating system: Linux glpi.bogus.com 6.8.0-41-generic #​41-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 2 20:41:06 UTC 2024 x86_64 PHP 8.3.6 apache2handler (Core, FFI, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, apache2handler, apcu, bcmath, bz2, calendar, ctype, curl, date, dom, exif, fileinfo, filter, ftp, gd, gettext, hash, iconv, igbinary, imap, intl, json, ldap, libxml, mbstring, mysqli, mysqlnd, openssl, pcre, pdo_mysql, posix, random, readline, redis, session, shmop, soap, sockets, sodium, standard, sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlreader, xmlrpc, xmlwriter, xsl, zip, zlib) Setup: max_execution_time="30" memory_limit="128M" post_max_size="8M" safe_mode="" session.save_handler="files" upload_max_filesize="2M" disable_functions="" Software: Apache/2.4.58 (Ubuntu) (Apache/2.4.58 (Ubuntu) Server at glpi.bogus.com Port 80 ) Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0 Server Software: Ubuntu 24.04 Server Version: 10.11.8-MariaDB-0ubuntu0.24.04.1 Server SQL Mode: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION Parameters: glpi@127.0.0.1/glpi Host info: 127.0.0.1 via TCP/IP PHP version (8.3.6) is supported. Sessions configuration is OK. Allocated memory is sufficient. mysqli extension is installed. Following extensions are installed: dom, fileinfo, filter, libxml, json, simplexml, xmlreader, xmlwriter. curl extension is installed. gd extension is installed. intl extension is installed. zlib extension is installed. The constant SODIUM_CRYPTO_AEAD_XCHACHA20POLY1305_IETF_NPUBBYTES is present. Database engine version (10.11.8) is supported. No files from previous GLPI version detected. The log file has been created successfully. Write access to /var/lib/glpi/_cache has been validated. Write access to /var/lib/glpi/_cron has been validated. Write access to /var/lib/glpi has been validated. Write access to /var/lib/glpi/_dumps has been validated. Write access to /var/lib/glpi/_graphs has been validated. Write access to /var/lib/glpi/_lock has been validated. Write access to /var/lib/glpi/_pictures has been validated. Write access to /var/lib/glpi/_plugins has been validated. Write access to /var/lib/glpi/_rss has been validated. Write access to /var/lib/glpi/_sessions has been validated. Write access to /var/lib/glpi/_tmp has been validated. Write access to /var/lib/glpi/_uploads has been validated. Web server root directory configuration seems safe. PHP directive "session.cookie_httponly" should be set to "on" to prevent client-side script to access cookie values. OS and PHP are relying on 64 bits integers. exif extension is installed. ldap extension is installed. openssl extension is installed. Following extensions are installed: bz2, Phar, zip. Zend OPcache extension is installed. Following extensions are installed: ctype, iconv, mbstring, sodium. Write access to /var/www/glpi/marketplace has been validated. Timezones seems loaded in database. GLPI_ROOT: "/var/www/glpi" GLPI_CONFIG_DIR: "/etc/glpi/" GLPI_VAR_DIR: "/var/lib/glpi" GLPI_LOG_DIR: "/var/log/glpi" GLPI_MARKETPLACE_DIR: "/var/www/glpi/marketplace" GLPI_USE_CSRF_CHECK: "1" GLPI_CSRF_EXPIRES: "7200" GLPI_CSRF_MAX_TOKENS: "100" GLPI_USE_IDOR_CHECK: "1" GLPI_IDOR_EXPIRES: "7200" GLPI_ALLOW_IFRAME_IN_RICH_TEXT: false GLPI_SERVERSIDE_URL_ALLOWLIST: ["/^(https?\|feed):\\/\\/[^@:]+(\\/.*)?$/"] GLPI_TELEMETRY_URI: "https://telemetry.glpi-project.org" GLPI_INSTALL_MODE: "TARBALL" GLPI_NETWORK_MAIL: "glpi@teclib.com" GLPI_NETWORK_SERVICES: "https://services.glpi-network.com" GLPI_MARKETPLACE_ALLOW_OVERRIDE: true GLPI_MARKETPLACE_MANUAL_DOWNLOADS: true GLPI_USER_AGENT_EXTRA_COMMENTS: "" GLPI_DISABLE_ONLY_FULL_GROUP_BY_SQL_MODE: "1" GLPI_AJAX_DASHBOARD: "1" GLPI_CALDAV_IMPORT_STATE: 0 GLPI_DEMO_MODE: "0" GLPI_CENTRAL_WARNINGS: "1" GLPI_TEXT_MAXSIZE: "4000" GLPI_DOC_DIR: "/var/lib/glpi" GLPI_CACHE_DIR: "/var/lib/glpi/_cache" GLPI_CRON_DIR: "/var/lib/glpi/_cron" GLPI_DUMP_DIR: "/var/lib/glpi/_dumps" GLPI_GRAPH_DIR: "/var/lib/glpi/_graphs" GLPI_LOCAL_I18N_DIR: "/var/lib/glpi/_locales" GLPI_LOCK_DIR: "/var/lib/glpi/_lock" GLPI_PICTURE_DIR: "/var/lib/glpi/_pictures" GLPI_PLUGIN_DOC_DIR: "/var/lib/glpi/_plugins" GLPI_RSS_DIR: "/var/lib/glpi/_rss" GLPI_SESSION_DIR: "/var/lib/glpi/_sessions" GLPI_TMP_DIR: "/var/lib/glpi/_tmp" GLPI_UPLOAD_DIR: "/var/lib/glpi/_uploads" GLPI_INVENTORY_DIR: "/var/lib/glpi/_inventories" GLPI_NETWORK_REGISTRATION_API_URL: "https://services.glpi-network.com/api/registration/" GLPI_MARKETPLACE_PLUGINS_API_URI: "https://services.glpi-network.com/api/marketplace/" GLPI_I18N_DIR: "/var/www/glpi/locales" GLPI_VERSION: "10.0.15" GLPI_SCHEMA_VERSION: "10.0.15@2eed74704cb07e0bac48b933cbd5c1c356f09629" GLPI_MARKETPLACE_PRERELEASES: false GLPI_MIN_PHP: "7.4.0" GLPI_MAX_PHP: "8.4.0" GLPI_YEAR: "2024" PHP User deprecated function (16384): phpCAS autoloader is deprecated. Install phpCAS using composer instead. in /usr/share/php/CAS/source/CAS/Autoload.php at line 86PHP User deprecated function (16384): Including CAS.php is deprecated. Install phpCAS using composer instead. in /usr/share/php/CAS/CAS.php at line 32htmlawed/htmlawed version 1.2.14 in (/var/www/glpi/vendor/htmlawed/htmlawed) phpmailer/phpmailer version 6.8.0 in (/var/www/glpi/vendor/phpmailer/phpmailer/src) simplepie/simplepie version 1.5.8 in (/var/www/glpi/vendor/simplepie/simplepie/library) tecnickcom/tcpdf version 6.7.5 in (/var/www/glpi/vendor/tecnickcom/tcpdf) michelf/php-markdown in (/var/www/glpi/vendor/michelf/php-markdown/Michelf) true/punycode in (/var/www/glpi/vendor/true/punycode/src) iamcal/lib_autolink in (/var/www/glpi/vendor/iamcal/lib_autolink) sabre/dav in (/var/www/glpi/vendor/sabre/dav/lib/DAV) sabre/http in (/var/www/glpi/vendor/sabre/http/lib) sabre/uri in (/var/www/glpi/vendor/sabre/uri/lib) sabre/vobject in (/var/www/glpi/vendor/sabre/vobject/lib) laminas/laminas-i18n in (/var/www/glpi/vendor/laminas/laminas-i18n/src) laminas/laminas-servicemanager in (/var/www/glpi/vendor/laminas/laminas-servicemanager/src) monolog/monolog in (/var/www/glpi/vendor/monolog/monolog/src/Monolog) sebastian/diff in (/var/www/glpi/vendor/sebastian/diff/src) donatj/phpuseragentparser in (/var/www/glpi/vendor/donatj/phpuseragentparser/src/UserAgent) elvanto/litemoji in (/var/www/glpi/vendor/elvanto/litemoji/src) symfony/console in (/var/www/glpi/vendor/symfony/console) scssphp/scssphp in (/var/www/glpi/vendor/scssphp/scssphp/src) laminas/laminas-mail in (/var/www/glpi/vendor/laminas/laminas-mail/src/Protocol) laminas/laminas-mime in (/var/www/glpi/vendor/laminas/laminas-mime/src) rlanvin/php-rrule in (/var/www/glpi/vendor/rlanvin/php-rrule/src) ramsey/uuid in (/var/www/glpi/vendor/ramsey/uuid/src) psr/log in (/var/www/glpi/vendor/psr/log/Psr/Log) psr/simple-cache in (/var/www/glpi/vendor/psr/simple-cache/src) psr/cache in (/var/www/glpi/vendor/psr/cache/src) league/csv in (/var/www/glpi/vendor/league/csv/src) mexitek/phpcolors in (/var/www/glpi/vendor/mexitek/phpcolors/src/Mexitek/PHPColors) guzzlehttp/guzzle in (/var/www/glpi/vendor/guzzlehttp/guzzle/src) guzzlehttp/psr7 in (/var/www/glpi/vendor/guzzlehttp/psr7/src) glpi-project/inventory_format in (/var/www/glpi/vendor/glpi-project/inventory_format/lib/php) wapmorgan/unified-archive in (/var/www/glpi/vendor/wapmorgan/unified-archive/src) paragonie/sodium_compat in (/var/www/glpi/vendor/paragonie/sodium_compat/src) symfony/cache in (/var/www/glpi/vendor/symfony/cache) html2text/html2text in (/var/www/glpi/vendor/html2text/html2text/src) symfony/css-selector in (/var/www/glpi/vendor/symfony/css-selector) symfony/dom-crawler in (/var/www/glpi/vendor/symfony/dom-crawler) twig/twig in (/var/www/glpi/vendor/twig/twig/src) twig/string-extra in (/var/www/glpi/vendor/twig/string-extra) symfony/polyfill-ctype not found symfony/polyfill-iconv not found symfony/polyfill-mbstring not found symfony/polyfill-php80 not found symfony/polyfill-php81 not found symfony/polyfill-php82 in (/var/www/glpi/vendor/symfony/polyfill-php82) league/oauth2-client in (/var/www/glpi/vendor/league/oauth2-client/src/Provider) league/oauth2-google in (/var/www/glpi/vendor/league/oauth2-google/src/Provider) thenetworg/oauth2-azure in (/var/www/glpi/vendor/thenetworg/oauth2-azure/src/Provider) phpCas version 1.6.0 in (/usr/share/php/CAS/source) Not active Way of sending emails: SMTP+TLS (IT@bogus.com@exch.bogus.com) Name: 'IT@bogus.com' Active: Yes Server: '{exch.bogus.com/imap/ssl/novalidate-cert}INBOX' Login: 'IT@bogus.com' Password: Yes fusioninventory Name: FusionInventory Version: 10.0.6+1.1 State: Replaced Install Method: Manual glpiinventory Name: GLPI Inventory Version: 1.3.5 State: Enabled Install Method: Marketplace mailanalyzer Name: Mail Analyzer Version: 3.2.0 State: Enabled Install Method: Manual onetimesecret Name: OneTimeSecret Version: 2.1.3 State: Not installed Install Method: Marketplace ticketfilter Name: Ticketfilter Version: 1.3.2 State: Not installed Install Method: Marketplace

Anything else?

This was already reported in https://github.com/glpi-project/glpi/issues/8448, it was not solved however.

cedric-anne commented 2 months ago

Hi,

When this happens, could you paste here the mail headers and the corresponding ticket ID that is sent in notifications ?

Terrokkar commented 2 months ago

I have done a bunch of further troubleshooting, pattern seems to point towards "ghost" mailgate runs. They dont appear in cron.log or in any other log. But they still run once a minute.

I cannot turn off this automatic mailgate in any way, i dont know if this is a bug or lack of knowledge, mailgate is set to disabled and crontab for all users is emptied.

Server rebooted and crontab service reloaded and restarted.

Now, these mailgate runs are the problem, whenever these run, it never succeeds. Always sends notification with some old ticket number and doesnt create the ticket. With cron emptied, error rate is 100%. This also holds true for followups, however it does not send any notifications, just doesnt create them.

Is there a way to see whats triggering the running of those .php files that handle mail collection?

Workaround i have found is, that i essentially dilute this issue with running a cron job every 2 seconds. This makes the error rate somewhere in the neighbourhood of 1/31 (note that mailgate is still disabled in automatic actions). php /var/www/glpi/front/cron.php --force mailgate

In most cases if i just move the miscreated mail back to the Inbox folder, the ticket/followup gets created when next php /var/www/glpi/front/cron.php --force mailgate is ran. Sometimes however they get refused with receiver error log showing "Failed operation".

I have tried to also run: /usr/bin/php -cli /var/www/glpi/bin/console glpi:task:unlock --all But this doesnt seem to help, even when ran every 1 second to catch the "ghost" mailgate, if its getting stuck.

There is no correlation in mails that fail to properly process, same mail from same user, same header, same title, only difference is changed number in email body from 654.10 to 654.11. One fails, one doesnt. This particular user opens ~10 tickets a day like this with only changed numbers, sometimes they fail, most times they work. If php /var/www/glpi/front/cron.php --force mailgate is removed from crontab, they all fail.

I will add the header when one of them fails again.

Best regards, Rok

cedric-anne commented 1 month ago

I have done a bunch of further troubleshooting, pattern seems to point towards "ghost" mailgate runs. They dont appear in cron.log or in any other log. But they still run once a minute.

I cannot turn off this automatic mailgate in any way, i dont know if this is a bug or lack of knowledge, mailgate is set to disabled and crontab for all users is emptied.

Server rebooted and crontab service reloaded and restarted.

Is it possible that you have somewhere in the wild another server (for instance a preproduction server) connected to the same database than your production server ? It is the only logical explaination to me.

Terrokkar commented 1 month ago

There is a test server, but its been offline since before this one was put up. No data was brought over, no sync, no connection with API. It was not even running on same VLAN or same Cluster. Also this one is using its own local database.

One thing i thought of is the plugin for checking MailID and match it to tickets. Its called MailAnalyzer, is this maybe able to mess with the TicketIDs in some way?

But still the mailcollector runs that run on their own are the problem. Is there another process in the background that would run mailgates besides cron or internal glpi automatic actions? He does grab the mails and read them, so it has the be the mailcollector.

Terrokkar commented 1 month ago

A little update, i have set the Receiver for the mail to active=no, cleared the crontabs again and result is the same. It still runs every minute or so. I will try to completely delete it as well.

Is there any other reason besides automatic actions and cron, that would make mailgate run?

cedric-anne commented 1 month ago

IMHO, your test server is up and it collects emails on its own side. As it has its own database, and is not used by anyone, it has probably fewer tickets in its database. So when it collects an email, it takes the next available ID, and that ID correspoond to an old ticket on the production server.

Terrokkar commented 1 week ago

Someone has seen the GLPI-Test VM offline and powered it on. Issue disappeared after VM was again turned off.

Sorry for inconvenience.

Best regards, Terro