glpi-project / glpi

GLPI is a Free Asset and IT Management Software package, Data center management, ITIL Service Desk, licenses tracking and software auditing.
https://glpi-project.org
GNU General Public License v3.0
4.35k stars 1.3k forks source link

Hebrew Characters turn to Gibberish when ticket is received via 0Auth with Office365 #17396

Open NegroniBlast opened 4 months ago

NegroniBlast commented 4 months ago

Code of Conduct

Is there an existing issue for this?

Version

10.0.12

Bug description

I installed GLPI 10.0.12 and after connecting our ticketing office365 tenant mailbox via 0auth, I tried opening tickets via email with Hebrew characters and I get gibberish instead in the subject and the body.

When I open it manually on the platform there's no problem and Hebrew is shown fine.

Relevant log output

No response

Page URL

No response

Steps To reproduce

  1. Connect receiver office365 mailbox with 0Auth to GLPI
  2. Setup automatic mailgate to retrieve the new emails as tickets
  3. Send email with Hebrew characters to this mailbox
  4. The new ticket is created, but with Hebrew content in Gibberish

Your GLPI setup information

Information about system installation & configuration
GLPI 10.0.12 ( => /var/www/glpi)
Installation mode: TARBALL
Current language:en_US

Server
 
Operating system: Linux REDACTED WEBSITE ADDRESS 5.15.0-113-generic #​123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64
PHP 8.1.2-1ubuntu2.18 fpm-fcgi (Core, FFI, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, apcu, bz2, calendar, cgi-fcgi,
    ctype, curl, date, dom, exif, fileinfo, filter, ftp, gd, gettext, hash, iconv, imagick, imap, intl, json, ldap, libxml,
    mbstring, memcache, mysqli, mysqlnd, openssl, pcre, pdo_mysql, posix, pspell, readline, session, shmop, sockets, sodium,
    standard, sysvmsg, sysvsem, sysvshm, tidy, tokenizer, xml, xmlreader, xmlrpc, xmlwriter, xsl, zip, zlib)
Setup: max_execution_time="30" memory_limit="128M" post_max_size="8M" safe_mode="" session.save_handler="files"
    upload_max_filesize="2M" disable_functions="" 
Software: nginx/1.18.0
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Server Software: Ubuntu 22.04
    Server Version: 10.6.18-MariaDB-0ubuntu0.22.04.1
    Server SQL Mode: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
    Parameters: glpi_admin@localhost/glpi_db
    Host info: Localhost via UNIX socket

PHP version (8.1.2-1ubuntu2.18) is supported.
Sessions configuration is OK.
Allocated memory is sufficient.
mysqli extension is installed.
Following extensions are installed: dom, fileinfo, filter, libxml, json, simplexml, xmlreader, xmlwriter.
curl extension is installed.
gd extension is installed.
intl extension is installed.
zlib extension is installed.
The constant SODIUM_CRYPTO_AEAD_XCHACHA20POLY1305_IETF_NPUBBYTES is present.
Database engine version (10.6.18) is supported.
No files from previous GLPI version detected.
The log file has been created successfully.
Write access to /var/www/glpi/files/_cache has been validated.
Write access to /var/www/glpi/files/_cron has been validated.
Write access to /var/www/glpi/files has been validated.
Write access to /var/www/glpi/files/_dumps has been validated.
Write access to /var/www/glpi/files/_graphs has been validated.
Write access to /var/www/glpi/files/_lock has been validated.
Write access to /var/www/glpi/files/_pictures has been validated.
Write access to /var/www/glpi/files/_plugins has been validated.
Write access to /var/www/glpi/files/_rss has been validated.
Write access to /var/www/glpi/files/_sessions has been validated.
Write access to /var/www/glpi/files/_tmp has been validated.
Write access to /var/www/glpi/files/_uploads has been validated.

Web server root directory configuration seems safe.
Sessions configuration is secured.
OS and PHP are relying on 64 bits integers.
exif extension is installed.
ldap extension is installed.
openssl extension is installed.
Following extensions are installed: bz2, Phar, zip.
Zend OPcache extension is installed.
Following extensions are installed: ctype, iconv, mbstring, sodium.
Write access to /var/www/glpi/marketplace has been validated.
Timezones seems loaded in database.

GLPI constants
 
GLPI_ROOT: "/var/www/glpi"
GLPI_CONFIG_DIR: "/var/www/glpi/config"
GLPI_VAR_DIR: "/var/www/glpi/files"
GLPI_MARKETPLACE_DIR: "/var/www/glpi/marketplace"
GLPI_USE_CSRF_CHECK: "1"
GLPI_CSRF_EXPIRES: "7200"
GLPI_CSRF_MAX_TOKENS: "100"
GLPI_USE_IDOR_CHECK: "1"
GLPI_IDOR_EXPIRES: "7200"
GLPI_ALLOW_IFRAME_IN_RICH_TEXT: false
GLPI_SERVERSIDE_URL_ALLOWLIST: ["/^(https?|feed):\\/\\/[^@:]+(\\/.*)?$/"]
GLPI_TELEMETRY_URI: "https://telemetry.glpi-project.org"
GLPI_INSTALL_MODE: "TARBALL"
GLPI_NETWORK_MAIL: "glpi@teclib.com"
GLPI_NETWORK_SERVICES: "https://services.glpi-network.com"
GLPI_MARKETPLACE_ALLOW_OVERRIDE: true
GLPI_MARKETPLACE_MANUAL_DOWNLOADS: true
GLPI_USER_AGENT_EXTRA_COMMENTS: ""
GLPI_DISABLE_ONLY_FULL_GROUP_BY_SQL_MODE: "1"
GLPI_AJAX_DASHBOARD: "1"
GLPI_CALDAV_IMPORT_STATE: 0
GLPI_DEMO_MODE: "0"
GLPI_CENTRAL_WARNINGS: "1"
GLPI_TEXT_MAXSIZE: "4000"
GLPI_DOC_DIR: "/var/www/glpi/files"
GLPI_CACHE_DIR: "/var/www/glpi/files/_cache"
GLPI_CRON_DIR: "/var/www/glpi/files/_cron"
GLPI_DUMP_DIR: "/var/www/glpi/files/_dumps"
GLPI_GRAPH_DIR: "/var/www/glpi/files/_graphs"
GLPI_LOCAL_I18N_DIR: "/var/www/glpi/files/_locales"
GLPI_LOCK_DIR: "/var/www/glpi/files/_lock"
GLPI_LOG_DIR: "/var/www/glpi/files/_log"
GLPI_PICTURE_DIR: "/var/www/glpi/files/_pictures"
GLPI_PLUGIN_DOC_DIR: "/var/www/glpi/files/_plugins"
GLPI_RSS_DIR: "/var/www/glpi/files/_rss"
GLPI_SESSION_DIR: "/var/www/glpi/files/_sessions"
GLPI_TMP_DIR: "/var/www/glpi/files/_tmp"
GLPI_UPLOAD_DIR: "/var/www/glpi/files/_uploads"
GLPI_INVENTORY_DIR: "/var/www/glpi/files/_inventories"
GLPI_NETWORK_REGISTRATION_API_URL: "https://services.glpi-network.com/api/registration/"
GLPI_MARKETPLACE_PLUGINS_API_URI: "https://services.glpi-network.com/api/marketplace/"
GLPI_I18N_DIR: "/var/www/glpi/locales"
GLPI_VERSION: "10.0.12"
GLPI_SCHEMA_VERSION: "10.0.12@b902ef5f5da967c1a9d0826ee8a7dce07d57cf3d"
GLPI_MARKETPLACE_PRERELEASES: false
GLPI_MIN_PHP: "7.4.0"
GLPI_MAX_PHP: "8.4.0"
GLPI_YEAR: "2024"

Libraries
 
htmlawed/htmlawed version 1.2.14 in (/var/www/glpi/vendor/htmlawed/htmlawed)
phpmailer/phpmailer version 6.8.0 in (/var/www/glpi/vendor/phpmailer/phpmailer/src)
simplepie/simplepie version 1.5.8 in (/var/www/glpi/vendor/simplepie/simplepie/library)
tecnickcom/tcpdf version 6.6.2 in (/var/www/glpi/vendor/tecnickcom/tcpdf)
michelf/php-markdown in (/var/www/glpi/vendor/michelf/php-markdown/Michelf)
true/punycode in (/var/www/glpi/vendor/true/punycode/src)
iamcal/lib_autolink in (/var/www/glpi/vendor/iamcal/lib_autolink)
sabre/dav in (/var/www/glpi/vendor/sabre/dav/lib/DAV)
sabre/http in (/var/www/glpi/vendor/sabre/http/lib)
sabre/uri in (/var/www/glpi/vendor/sabre/uri/lib)
sabre/vobject in (/var/www/glpi/vendor/sabre/vobject/lib)
laminas/laminas-i18n in (/var/www/glpi/vendor/laminas/laminas-i18n/src)
laminas/laminas-servicemanager in (/var/www/glpi/vendor/laminas/laminas-servicemanager/src)
monolog/monolog in (/var/www/glpi/vendor/monolog/monolog/src/Monolog)
sebastian/diff in (/var/www/glpi/vendor/sebastian/diff/src)
donatj/phpuseragentparser in (/var/www/glpi/vendor/donatj/phpuseragentparser/src/UserAgent)
elvanto/litemoji in (/var/www/glpi/vendor/elvanto/litemoji/src)
symfony/console in (/var/www/glpi/vendor/symfony/console)
scssphp/scssphp in (/var/www/glpi/vendor/scssphp/scssphp/src)
laminas/laminas-mail in (/var/www/glpi/vendor/laminas/laminas-mail/src/Protocol)
laminas/laminas-mime in (/var/www/glpi/vendor/laminas/laminas-mime/src)
rlanvin/php-rrule in (/var/www/glpi/vendor/rlanvin/php-rrule/src)
ramsey/uuid in (/var/www/glpi/vendor/ramsey/uuid/src)
psr/log in (/var/www/glpi/vendor/psr/log/Psr/Log)
psr/simple-cache in (/var/www/glpi/vendor/psr/simple-cache/src)
psr/cache in (/var/www/glpi/vendor/psr/cache/src)
league/csv in (/var/www/glpi/vendor/league/csv/src)
mexitek/phpcolors in (/var/www/glpi/vendor/mexitek/phpcolors/src/Mexitek/PHPColors)
guzzlehttp/guzzle in (/var/www/glpi/vendor/guzzlehttp/guzzle/src)
guzzlehttp/psr7 in (/var/www/glpi/vendor/guzzlehttp/psr7/src)
glpi-project/inventory_format in (/var/www/glpi/vendor/glpi-project/inventory_format/lib/php)
wapmorgan/unified-archive in (/var/www/glpi/vendor/wapmorgan/unified-archive/src)
paragonie/sodium_compat in (/var/www/glpi/vendor/paragonie/sodium_compat/src)
symfony/cache in (/var/www/glpi/vendor/symfony/cache)
html2text/html2text in (/var/www/glpi/vendor/html2text/html2text/src)
symfony/css-selector in (/var/www/glpi/vendor/symfony/css-selector)
symfony/dom-crawler in (/var/www/glpi/vendor/symfony/dom-crawler)
twig/twig in (/var/www/glpi/vendor/twig/twig/src)
twig/string-extra in (/var/www/glpi/vendor/twig/string-extra)
symfony/polyfill-ctype not found
symfony/polyfill-iconv not found
symfony/polyfill-mbstring not found
symfony/polyfill-php80 not found
symfony/polyfill-php81 not found
symfony/polyfill-php82 in (/var/www/glpi/vendor/symfony/polyfill-php82)
league/oauth2-client in (/var/www/glpi/vendor/league/oauth2-client/src/Provider)
league/oauth2-google in (/var/www/glpi/vendor/league/oauth2-google/src/Provider)
thenetworg/oauth2-azure in (/var/www/glpi/vendor/thenetworg/oauth2-azure/src/Provider)
phpCas version 1.3.8 in (/usr/share/php/CAS/source)

SQL replicas
 
Not active

Notifications
 
Way of sending emails: SMTP+TLS (REDACTED@octopuscs.com@smtp.office365.com)

Plugins list
 
    news                 Name: Alerts                         Version: 1.12.2     State: Installed / not activated               
        Install Method: Marketplace
    behaviors            Name: Behaviours                     Version: 2.7.3      State: Installed / not activated               
        Install Method: Marketplace
    datainjection        Name: Data injection                 Version: 2.13.5     State: Installed / not activated               
        Install Method: Marketplace
    archimap             Name: Diagrams                       Version: 3.3.4      State: Installed / not activated               
        Install Method: Marketplace
    financialreports     Name: Financial reports              Version: 3.0.0      State: Installed / not activated               
        Install Method: Marketplace
    addressing           Name: IP Adressing                   Version: 3.0.2      State: Installed / not activated               
        Install Method: Marketplace
    mfa                  Name: MFA                            Version: 1.0.2      State: Installed / not activated               
        Install Method: Marketplace
    mreporting           Name: More Reporting                 Version: 1.8.6      State: Installed / not activated               
        Install Method: Marketplace
    oauthimap            Name: Oauth IMAP                     Version: 1.4.3      State: Enabled                                 
        Install Method: Marketplace
    onetimesecret        Name: OneTimeSecret                  Version: 2.1.3      State: Installed / not activated               
        Install Method: Marketplace
    pdf                  Name: Print to pdf                   Version: 3.0.0      State: Installed / not activated               
        Install Method: Marketplace
    reports              Name: Reports                        Version: 1.16.0     State: Installed / not activated               
        Install Method: Marketplace
    screenshot           Name: Screenshot                     Version: 2.0.3      State: Installed / not activated               
        Install Method: Marketplace
    ticketfilter         Name: Ticketfilter                   Version: 1.3.2      State: Installed / not activated               
        Install Method: Marketplace
    timelineticket       Name: Timeline of tickets            Version: 10.0+1.2   State: Installed / not activated               
        Install Method: Marketplace
    vip                  Name: VIP                            Version: 1.8.2      State: Installed / not activated               
        Install Method: Marketplace

Anything else?

Screenshot 2024-07-01 at 14 07 51

NegroniBlast commented 4 months ago

these might be relevant but i dont know if these fixes were implemented into 10.0.12 so i decided to open a new issue here https://github.com/glpi-project/glpi/pull/7748 https://github.com/glpi-project/glpi/pull/4844

NegroniBlast commented 4 months ago

this is the closest issue I found on the subject https://github.com/glpi-project/glpi/issues/7709

cedric-anne commented 4 months ago

Hi,

Please check if the problem persists in GLPI 10.0.15. If problem persists, please attache the message into .eml format so we could add it to our test suite and see the results.

NegroniBlast commented 4 months ago

Hi Cedric thanks for the quick reply. we upgrade to 10.0.15 and now the gibberish is different haha, still an encoding issue. Screenshot 2024-07-01 at 16 23 32 any recommendations?

NegroniBlast commented 4 months ago

Testing after 10.0.15 בדיקה נוספת אחרי מעבר לגרסה חדשה.eml.zip

it was blocking the .eml file so it's attached inside this compressed zip file.

cedric-anne commented 4 months ago

I validate the issue. It sees that the problem is related to usage of iso-8859-8-i encoding.

*** PHP Warning (2): iconv(): Wrong encoding, conversion from "iso-8859-8-i" to "UTF-8//IGNORE" is not allowed in /var/www/glpi/src/MailCollector.php at line 2470
  Backtrace :
  src/MailCollector.php:2470                         iconv()
  src/MailCollector.php:1808                         MailCollector->getDecodedContent()
  src/MailCollector.php:1150                         MailCollector->getBody()
  src/MailCollector.php:811                          MailCollector->buildTicket()
  front/mailcollector.form.php:106                   MailCollector->collect()
  public/index.php:82                                require()
NegroniBlast commented 3 months ago

Issue solved.

By editing Mailcollector.php I was able to edit the functions decodeMimeHeader and getDecodedContent to be able to handle Hebrew characters. the decoding function that works for incoming emails from M365 is mb_convert_encoding.

Attaching my code snippets of Mailcollector.php for your consideration. I will test some more next week but you can close this thread if you want.

  1. Subject wasn't using the subject decoding function - it was going to 'cleanSubject' instead. I replace cleanSubject with my decodeMimeHeader function.
  2. getDecodedContent had too many conditions at first, then i replaced it with a leaner version which is suitable for our needs. GLPI Mailcollector.php updated functions and code for Hebrew Encoding email messages.zip
trasher commented 3 months ago

Could you please open a pull request with your changes? Test suite will run with your changes; and it's easiest for us to review changes.