Duplicated entries everywhere

tetsumaki commented 2 months ago

Code of Conduct

[x] I agree to follow this project's Code of Conduct

Is there an existing issue for this?

[X] I have searched the existing issues

Version

10.0.16

Bug description

Hello, when agents on a number of relatively similar machines send inventories, and at the same time, this creates duplicates if the name doesn't yet exist. with different IDs of course.

I don't know if it's possible to manage this in the PHP code, given that the PHP pools work independently, but if so, so much the better.

But I think you'd also need to add UNIQUE CONSTRAINT constraints to MariaDB tables on a lot of entries.

I've included several screenshots, but this kind of duplication can be seen almost everywhere.

2024-09-10 10_32_06-Logiciels - GLPI - Vivaldi

2024-09-10 10_32_27-Type d'ordinateur - GLPI - Vivaldi

2024-09-10 10_32_40-Fabricant - GLPI - Vivaldi

2024-09-10 10_32_56-Modèle d'ordinateur - GLPI - Vivaldi

2024-09-10 10_33_08-Système d'exploitation - GLPI - Vivaldi

2024-09-10 10_33_27-Version de noyau - GLPI - Vivaldi

Relevant log output

No response

Page URL

No response

Steps To reproduce

Send multiple inventories at the same time (second, or millisecond)

Your GLPI setup information

No response

Anything else?

No response

Nour-Ivory commented 2 months ago

Experiencing the same issue in Assets-->Software (again) as I have reported this issue last month. Went ahead and spun a brand new LAMP server with a clean and fresh copy of GLPI server, and at the beginning all worked smoothly, as Software versions were grouped correctly, however after a couple of weeks of using it, the same behaviour started showing up again, with the same application showing multiple entries whereas they should all be grouped under the same heading.

The screenshots below all show Applications that all have the same version but listed multiple times

trasher commented 2 months ago

I've tried to reproduce, and the only way for me to do it is to send several inventories at the same time. From my local computer, it sends with several version of the computer, and also several versions of each software - among others.

You can try https://github.com/glpi-project/glpi/pull/17924 - that will prevent concurrent inventories. It requires the sysvsem PHP extension to be present and loaded (it will silently fail if extension is not present).

It's quite difficult to know the implications on hundreds of inventories - I just have my local instance to make tests. If you run agent as a service (the recommended way); all should work fine. If you rely on a crontab or similar - maybe that will cause some issues (and anyway, using agent from a crontab is strongly discouraged).

cmeh commented 1 month ago

As already mentioned (but not referenced) by @Nour-Ivory , this might be related to #17319 reported by @troloff.

tetsumaki commented 1 month ago

I've tried to reproduce, and the only way for me to do it is to send several inventories at the same time. From my local computer, it sends with several version of the computer, and also several versions of each software - among others.

You can try #17924 - that will prevent concurrent inventories. It requires the sysvsem PHP extension to be present and loaded (it will silently fail if extension is not present).

It's quite difficult to know the implications on hundreds of inventories - I just have my local instance to make tests. If you run agent as a service (the recommended way); all should work fine. If you rely on a crontab or similar - maybe that will cause some issues (and anyway, using agent from a crontab is strongly discouraged).

On a clean installation. It's possible to use ansible and send (POST) a many test inventory (.json) list with ansible.builtin.uri and a loop. Or simply with curl.

trasher commented 1 month ago

Hi,

You can try #17924 - that will prevent concurrent inventories. It requires the sysvsem PHP extension to be present and loaded (it will silently fail if extension is not present).

I still have no feedback on that fix proposal.

Nour-Ivory commented 1 month ago

Hi there

The extension is present and loaded (screenshot of loaded extensions below). ( See System Information below)

This is on a new Ubuntu server, and something that I have noticed was the fact that initially the inventories were grouped correctly , it is only after 2 to 3 weeks of updated inventories that the issue becomes active, almost as if an Entry exists, when an updated version of that software is installed on a client system, it is considered as a new piece of software and has a single entry in the database.

Information about system installation and configuration

GLPI 10.0.16 ( => /var/www/html/glpi)
Installation mode: TARBALL
Current language:en_GB

Server

Operating system: Linux GLPI 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29
13:45:52 UTC 2024 x86_64
PHP 8.2.24 apache2handler (Core, FFI, PDO, Phar, Reflection, SPL,
SimpleXML, Zend OPcache, apache2handler, apcu, bcmath, bz2,
calendar, ctype, curl, date, dom, exif, fileinfo, filter, ftp, gd, gettext,
hash, iconv, intl, json, ldap, libxml, mbstring,
mysqli, mysqlnd, openssl, pcre, pdo_mysql, posix, random, readline,
session, shmop, sockets, sodium, standard, sysvmsg, sysvsem,
sysvshm, tokenizer, xml, xmlreader, xmlwriter, xsl, zip, zlib)
Setup: max_execution_time="60" memory_limit="256M" post_max_size="20M"
safe_mode="" session.save_handler="files"
upload_max_filesize="20M" disable_functions=""
Software: Apache/2.4.52 (Ubuntu) (Apache/2.4.52 (Ubuntu) Server at
ivorytalks.com Port 443)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/129.0.0.0 Safari/537.36
Server Software: (Ubuntu)
Server Version: 8.0.39-0ubuntu0.22.04.1
Server SQL Mode:
STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
Parameters: ***@***.***/glpi
Host info: Localhost via UNIX socket

PHP version (8.2.24) is supported.
Sessions configuration is OK.
Allocated memory is sufficient.
mysqli extension is installed.
Following extensions are installed: dom, fileinfo, filter, libxml, json,
simplexml, xmlreader, xmlwriter.
curl extension is installed.
gd extension is installed.
intl extension is installed.
zlib extension is installed.
The constant SODIUM_CRYPTO_AEAD_XCHACHA20POLY1305_IETF_NPUBBYTES is present.
Database engine version (8.0.39) is supported.
No files from previous GLPI version detected.
The log file has been created successfully.
Write access to /var/lib/glpi/_cache has been validated.
Write access to /var/lib/glpi/_cron has been validated.
Write access to /var/lib/glpi has been validated.
Write access to /var/lib/glpi/_dumps has been validated.
Write access to /var/lib/glpi/_graphs has been validated.
Write access to /var/lib/glpi/_lock has been validated.
Write access to /var/lib/glpi/_pictures has been validated.
Write access to /var/lib/glpi/_plugins has been validated.
Write access to /var/lib/glpi/_rss has been validated.
Write access to /var/lib/glpi/_sessions has been validated.
Write access to /var/lib/glpi/_tmp has been validated.
Write access to /var/lib/glpi/_uploads has been validated.

Web server root directory configuration seems safe.
Sessions configuration is secured.
OS and PHP are relying on 64 bits integers.
exif extension is installed.
ldap extension is installed.
openssl extension is installed.
Following extensions are installed: bz2, Phar, zip.
Zend OPcache extension is installed.
Following extensions are installed: ctype, iconv, mbstring, sodium.
Write access to /var/www/html/glpi/marketplace has been validated.
Timezones seems loaded in database.

GLPI constants

GLPI_ROOT: "/var/www/html/glpi"
GLPI_CONFIG_DIR: "/etc/glpi/"
GLPI_VAR_DIR: "/var/lib/glpi"
GLPI_DOC_DIR: "/var/lib/glpi"
GLPI_CRON_DIR: "/var/lib/glpi/_cron"
GLPI_DUMP_DIR: "/var/lib/glpi/_dumps"
GLPI_GRAPH_DIR: "/var/lib/glpi/_graphs"
GLPI_LOCK_DIR: "/var/lib/glpi/_lock"
GLPI_PICTURE_DIR: "/var/lib/glpi/_pictures"
GLPI_PLUGIN_DOC_DIR: "/var/lib/glpi/_plugins"
GLPI_RSS_DIR: "/var/lib/glpi/_rss"
GLPI_SESSION_DIR: "/var/lib/glpi/_sessions"
GLPI_TMP_DIR: "/var/lib/glpi/_tmp"
GLPI_UPLOAD_DIR: "/var/lib/glpi/_uploads"
GLPI_CACHE_DIR: "/var/lib/glpi/_cache"
GLPI_LOG_DIR: "/var/log/glpi"
GLPI_MARKETPLACE_DIR: "/var/www/html/glpi/marketplace"
GLPI_USE_CSRF_CHECK: "1"
GLPI_CSRF_EXPIRES: "7200"
GLPI_CSRF_MAX_TOKENS: "100"
GLPI_USE_IDOR_CHECK: "1"
GLPI_IDOR_EXPIRES: "7200"
GLPI_ALLOW_IFRAME_IN_RICH_TEXT: false
GLPI_SERVERSIDE_URL_ALLOWLIST: ["/^(https?|feed):\\/\\/[^@:]+(\\/.*)?$/"]
GLPI_TELEMETRY_URI: "https://telemetry.glpi-project.org"
GLPI_INSTALL_MODE: "TARBALL"
GLPI_NETWORK_MAIL: ***@***.***"
GLPI_NETWORK_SERVICES: "https://services.glpi-network.com"
GLPI_MARKETPLACE_ALLOW_OVERRIDE: true
GLPI_MARKETPLACE_MANUAL_DOWNLOADS: true
GLPI_USER_AGENT_EXTRA_COMMENTS: ""
GLPI_DISABLE_ONLY_FULL_GROUP_BY_SQL_MODE: "1"
GLPI_AJAX_DASHBOARD: "1"
GLPI_CALDAV_IMPORT_STATE: 0
GLPI_DEMO_MODE: "0"
GLPI_CENTRAL_WARNINGS: "1"
GLPI_TEXT_MAXSIZE: "4000"
GLPI_LOCAL_I18N_DIR: "/var/lib/glpi/_locales"
GLPI_INVENTORY_DIR: "/var/lib/glpi/_inventories"
GLPI_NETWORK_REGISTRATION_API_URL: "
https://services.glpi-network.com/api/registration/"
GLPI_MARKETPLACE_PLUGINS_API_URI: "
https://services.glpi-network.com/api/marketplace/"
GLPI_I18N_DIR: "/var/www/html/glpi/locales"
GLPI_VERSION: "10.0.16"
GLPI_SCHEMA_VERSION: ***@***.***"
GLPI_MARKETPLACE_PRERELEASES: false
GLPI_MIN_PHP: "7.4.0"
GLPI_MAX_PHP: "8.4.0"
GLPI_YEAR: "2024"

Libraries

htmlawed/htmlawed version 1.2.14 in
(/var/www/html/glpi/vendor/htmlawed/htmlawed)
phpmailer/phpmailer version 6.8.0 in
(/var/www/html/glpi/vendor/phpmailer/phpmailer/src)
simplepie/simplepie version 1.5.8 in
(/var/www/html/glpi/vendor/simplepie/simplepie/library)
tecnickcom/tcpdf version 6.7.5 in
(/var/www/html/glpi/vendor/tecnickcom/tcpdf)
michelf/php-markdown in
(/var/www/html/glpi/vendor/michelf/php-markdown/Michelf)
true/punycode in (/var/www/html/glpi/vendor/true/punycode/src)
iamcal/lib_autolink in (/var/www/html/glpi/vendor/iamcal/lib_autolink)
sabre/dav in (/var/www/html/glpi/vendor/sabre/dav/lib/DAV)
sabre/http in (/var/www/html/glpi/vendor/sabre/http/lib)
sabre/uri in (/var/www/html/glpi/vendor/sabre/uri/lib)
sabre/vobject in (/var/www/html/glpi/vendor/sabre/vobject/lib)
laminas/laminas-i18n in (/var/www/html/glpi/vendor/laminas/laminas-i18n/src)
laminas/laminas-servicemanager in
(/var/www/html/glpi/vendor/laminas/laminas-servicemanager/src)
monolog/monolog in (/var/www/html/glpi/vendor/monolog/monolog/src/Monolog)
sebastian/diff in (/var/www/html/glpi/vendor/sebastian/diff/src)
donatj/phpuseragentparser in
(/var/www/html/glpi/vendor/donatj/phpuseragentparser/src/UserAgent)
elvanto/litemoji in (/var/www/html/glpi/vendor/elvanto/litemoji/src)
symfony/console in (/var/www/html/glpi/vendor/symfony/console)
scssphp/scssphp in (/var/www/html/glpi/vendor/scssphp/scssphp/src)
laminas/laminas-mail in
(/var/www/html/glpi/vendor/laminas/laminas-mail/src/Protocol)
laminas/laminas-mime in (/var/www/html/glpi/vendor/laminas/laminas-mime/src)
rlanvin/php-rrule in (/var/www/html/glpi/vendor/rlanvin/php-rrule/src)
ramsey/uuid in (/var/www/html/glpi/vendor/ramsey/uuid/src)
psr/log in (/var/www/html/glpi/vendor/psr/log/Psr/Log)
psr/simple-cache in (/var/www/html/glpi/vendor/psr/simple-cache/src)
psr/cache in (/var/www/html/glpi/vendor/psr/cache/src)
league/csv in (/var/www/html/glpi/vendor/league/csv/src)
mexitek/phpcolors in
(/var/www/html/glpi/vendor/mexitek/phpcolors/src/Mexitek/PHPColors)
guzzlehttp/guzzle in (/var/www/html/glpi/vendor/guzzlehttp/guzzle/src)
guzzlehttp/psr7 in (/var/www/html/glpi/vendor/guzzlehttp/psr7/src)
glpi-project/inventory_format in
(/var/www/html/glpi/vendor/glpi-project/inventory_format/lib/php)
wapmorgan/unified-archive in
(/var/www/html/glpi/vendor/wapmorgan/unified-archive/src)
paragonie/sodium_compat in
(/var/www/html/glpi/vendor/paragonie/sodium_compat/src)
symfony/cache in (/var/www/html/glpi/vendor/symfony/cache)
html2text/html2text in (/var/www/html/glpi/vendor/html2text/html2text/src)
symfony/css-selector in (/var/www/html/glpi/vendor/symfony/css-selector)
symfony/dom-crawler in (/var/www/html/glpi/vendor/symfony/dom-crawler)
twig/twig in (/var/www/html/glpi/vendor/twig/twig/src)
twig/string-extra in (/var/www/html/glpi/vendor/twig/string-extra)
symfony/polyfill-ctype not found
symfony/polyfill-iconv not found
symfony/polyfill-mbstring not found
symfony/polyfill-php80 not found
symfony/polyfill-php81 not found
symfony/polyfill-php82 in (/var/www/html/glpi/vendor/symfony/polyfill-php82)
league/oauth2-client in
(/var/www/html/glpi/vendor/league/oauth2-client/src/Provider)
league/oauth2-google in
(/var/www/html/glpi/vendor/league/oauth2-google/src/Provider)
thenetworg/oauth2-azure in
(/var/www/html/glpi/vendor/thenetworg/oauth2-azure/src/Provider)
phpCas version 1.3.8 in (/usr/share/php/CAS/source)

SQL replicas

Not active

Notifications

Way of sending emails: PHP

Plugins list

[image: Screenshot 2024-10-15 at 13.49.21.png]

On Tue, 15 Oct 2024 at 13:27, Johan Cwiklinski @.***> wrote:

Hi,

You can try #17924 https://github.com/glpi-project/glpi/pull/17924 - that will prevent concurrent inventories. It requires the sysvsem PHP extension to be present and loaded (it will silently fail if extension is not present).

I still have no feedback on that fix proposal.

— Reply to this email directly, view it on GitHub https://github.com/glpi-project/glpi/issues/17821#issuecomment-2413774692, or unsubscribe https://github.com/notifications/unsubscribe-auth/BKMF4SPKREP7A6T2PKPFNXLZ3UCZNAVCNFSM6AAAAABN6H4EWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJTG43TINRZGI . You are receiving this because you were mentioned.Message ID: @.***>

troloff commented 1 week ago

We tried the patch and it did not make any difference. Also, new version 10.0.17 does not seem to change anything.

Another thing that came to my mind: Before the issue occured first, we did a switch from apache mod_php to php-fpm. Will try to switch back and see if that makes any difference.

trasher commented 1 week ago

I get no feedback, so it's not included in any release.

The proposed fix relies on a new PHP extension, please double check it is properly installed.

trasher commented 1 week ago

Also, the fix would prevent new duplicates, it won't cleanup anything.

troloff commented 1 week ago

Sorry for being unclear:

after upgrade to 10.0.17, I cleaned tables as suggested in https://github.com/glpi-project/glpi/issues/17319#issuecomment-2222180686
GLPI setup says sysvsem is there PHP 8.2.25 fpm-fcgi (Core, FFI, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, bz2, calendar, cgi-fcgi, ctype, curl, date, dom, exif, fileinfo, filter, ftp, gd, gettext, hash, iconv, imap, intl, json, ldap, libxml, mbstring, mysqli, mysqlnd, openssl, pcre, pdo_mysql, posix, random, readline, session, shmop, sockets, sodium, standard, sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlreader, xmlwriter, xsl, zip, zlib)

trasher commented 1 week ago

10.0.17 does not include the fix. You still have to apply #17924. If that does not prevent duplicates, that's because the issue is probably not concurrent inventories - then I have absolutely no idea what the problem is or how to reproduce.

tetsumaki commented 1 week ago

Would you like bash code to reproduce the problem via a mass insert ?

glpi-project / glpi