glpi-project / glpi

GLPI is a Free Asset and IT Management Software package, Data center management, ITIL Service Desk, licenses tracking and software auditing.
https://glpi-project.org
GNU General Public License v3.0
4.16k stars 1.28k forks source link

Duplicate Software Versions #17319

Closed troloff closed 1 month ago

troloff commented 3 months ago

Code of Conduct

Is there an existing issue for this?

Version

10.0.15 / GLPI Agent 1.9

Bug description

After update to 10.0.15, we are seeing multiple (duplicate) entries of the same software version. Example screenshot:

software_versions

Relevant log output

No response

Page URL

No response

Steps To reproduce

  1. deploy software using glpi agent (optional)
  2. run glpi inventory agent on different machines
  3. same software version shows up multiple times under assets->software->versions

Your GLPI setup information

Informationen über das System, die Installation und die Konfiguration
GLPI 10.0.15 ( => /XXX
Installation mode: TARBALL
Current language:de_DE

Server
 
Operating system: Linux XXX 6.1.0-13-amd64 #​1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64
PHP 8.2.18 fpm-fcgi (Core, FFI, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, bz2, calendar, cgi-fcgi, ctype, curl, date,
    dom, exif, fileinfo, filter, ftp, gd, gettext, hash, iconv, imap, intl, json, ldap, libxml, mbstring, mysqli, mysqlnd, openssl,
    pcre, pdo_mysql, posix, random, readline, session, shmop, sockets, sodium, standard, sysvmsg, sysvsem, sysvshm, tokenizer, xml,
    xmlreader, xmlwriter, xsl, zip, zlib)
Setup: max_execution_time="300" memory_limit="1024M" post_max_size="512M" safe_mode="" session.save_handler="files"
    upload_max_filesize="512M" disable_functions="" 
Software: Apache/2.4.59 ()
    Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0
Server Software: Debian 12
    Server Version: 10.11.6-MariaDB-0+deb12u1
    Server SQL Mode: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
    Parameters: glpi@localhost/glpi
    Host info: Localhost via UNIX socket

PHP version (8.2.18) is supported.
Sessions configuration is OK.
Allocated memory is sufficient.
mysqli extension is installed.
Following extensions are installed: dom, fileinfo, filter, libxml, json, simplexml, xmlreader, xmlwriter.
curl extension is installed.
gd extension is installed.
intl extension is installed.
zlib extension is installed.
The constant SODIUM_CRYPTO_AEAD_XCHACHA20POLY1305_IETF_NPUBBYTES is present.
Database engine version (10.11.6) is supported.
No files from previous GLPI version detected.
The log file has been created successfully.
Write access to /var/lib/glpi/_cache has been validated.
Write access to /var/lib/glpi/_cron has been validated.
Write access to /var/lib/glpi has been validated.
Write access to /var/lib/glpi/_dumps has been validated.
Write access to /var/lib/glpi/_graphs has been validated.
Write access to /var/lib/glpi/_lock has been validated.
Write access to /var/lib/glpi/_pictures has been validated.
Write access to /var/lib/glpi/_plugins has been validated.
Write access to /var/lib/glpi/_rss has been validated.
Write access to /var/lib/glpi/_sessions has been validated.
Write access to /var/lib/glpi/_tmp has been validated.
Write access to /var/lib/glpi/_uploads has been validated.

Web server root directory configuration seems safe.
Sessions configuration is secured.
OS and PHP are relying on 64 bits integers.
exif extension is installed.
ldap extension is installed.
openssl extension is installed.
Following extensions are installed: bz2, Phar, zip.
Zend OPcache extension is installed.
Following extensions are installed: ctype, iconv, mbstring, sodium.
Write access to /var/www/inventory.net.adk.de/marketplace has been validated.
Timezones seems loaded in database.

GLPI constants
 
GLPI_ROOT: ""
GLPI_CONFIG_DIR: "/etc/glpi"
GLPI_VAR_DIR: "/var/lib/glpi"
GLPI_LOG_DIR: "/var/log/glpi"
GLPI_MARKETPLACE_DIR: "XXX"
GLPI_USE_CSRF_CHECK: "1"
GLPI_CSRF_EXPIRES: "7200"
GLPI_CSRF_MAX_TOKENS: "100"
GLPI_USE_IDOR_CHECK: "1"
GLPI_IDOR_EXPIRES: "7200"
GLPI_ALLOW_IFRAME_IN_RICH_TEXT: false
GLPI_SERVERSIDE_URL_ALLOWLIST: ["/^(https?|feed):\\/\\/[^@:]+(\\/.*)?$/"]
GLPI_TELEMETRY_URI: "https://telemetry.glpi-project.org"
GLPI_INSTALL_MODE: "TARBALL"
GLPI_NETWORK_MAIL: "glpi@teclib.com"
GLPI_NETWORK_SERVICES: "https://services.glpi-network.com"
GLPI_MARKETPLACE_ALLOW_OVERRIDE: true
GLPI_MARKETPLACE_MANUAL_DOWNLOADS: true
GLPI_USER_AGENT_EXTRA_COMMENTS: ""
GLPI_DISABLE_ONLY_FULL_GROUP_BY_SQL_MODE: "1"
GLPI_AJAX_DASHBOARD: "1"
GLPI_CALDAV_IMPORT_STATE: 0
GLPI_DEMO_MODE: "0"
GLPI_CENTRAL_WARNINGS: "1"
GLPI_TEXT_MAXSIZE: "4000"
GLPI_DOC_DIR: "/var/lib/glpi"
GLPI_CACHE_DIR: "/var/lib/glpi/_cache"
GLPI_CRON_DIR: "/var/lib/glpi/_cron"
GLPI_DUMP_DIR: "/var/lib/glpi/_dumps"
GLPI_GRAPH_DIR: "/var/lib/glpi/_graphs"
GLPI_LOCAL_I18N_DIR: "/var/lib/glpi/_locales"
GLPI_LOCK_DIR: "/var/lib/glpi/_lock"
GLPI_PICTURE_DIR: "/var/lib/glpi/_pictures"
GLPI_PLUGIN_DOC_DIR: "/var/lib/glpi/_plugins"
GLPI_RSS_DIR: "/var/lib/glpi/_rss"
GLPI_SESSION_DIR: "/var/lib/glpi/_sessions"
GLPI_TMP_DIR: "/var/lib/glpi/_tmp"
GLPI_UPLOAD_DIR: "/var/lib/glpi/_uploads"
GLPI_INVENTORY_DIR: "/var/lib/glpi/_inventories"
GLPI_NETWORK_REGISTRATION_API_URL: "https://services.glpi-network.com/api/registration/"
GLPI_MARKETPLACE_PLUGINS_API_URI: "https://services.glpi-network.com/api/marketplace/"
GLPI_I18N_DIR: "XXX"
GLPI_VERSION: "10.0.15"
GLPI_SCHEMA_VERSION: "10.0.15@2eed74704cb07e0bac48b933cbd5c1c356f09629"
GLPI_MARKETPLACE_PRERELEASES: false
GLPI_MIN_PHP: "7.4.0"
GLPI_MAX_PHP: "8.4.0"
GLPI_YEAR: "2024"

Libraries
 
htmlawed/htmlawed version 1.2.14 in (/var/www/inventory.net.adk.de/vendor/htmlawed/htmlawed)
phpmailer/phpmailer version 6.8.0 in (/var/www/inventory.net.adk.de/vendor/phpmailer/phpmailer/src)
simplepie/simplepie version 1.5.8 in (/var/www/inventory.net.adk.de/vendor/simplepie/simplepie/library)
tecnickcom/tcpdf version 6.7.5 in (/var/www/inventory.net.adk.de/vendor/tecnickcom/tcpdf)
michelf/php-markdown in (/var/www/inventory.net.adk.de/vendor/michelf/php-markdown/Michelf)
true/punycode in (/var/www/inventory.net.adk.de/vendor/true/punycode/src)
iamcal/lib_autolink in (/var/www/inventory.net.adk.de/vendor/iamcal/lib_autolink)
sabre/dav in (/var/www/inventory.net.adk.de/vendor/sabre/dav/lib/DAV)
sabre/http in (/var/www/inventory.net.adk.de/vendor/sabre/http/lib)
sabre/uri in (/var/www/inventory.net.adk.de/vendor/sabre/uri/lib)
sabre/vobject in (/var/www/inventory.net.adk.de/vendor/sabre/vobject/lib)
laminas/laminas-i18n in (/var/www/inventory.net.adk.de/vendor/laminas/laminas-i18n/src)
laminas/laminas-servicemanager in (/var/www/inventory.net.adk.de/vendor/laminas/laminas-servicemanager/src)
monolog/monolog in (/var/www/inventory.net.adk.de/vendor/monolog/monolog/src/Monolog)
sebastian/diff in (/var/www/inventory.net.adk.de/vendor/sebastian/diff/src)
donatj/phpuseragentparser in (/var/www/inventory.net.adk.de/vendor/donatj/phpuseragentparser/src/UserAgent)
elvanto/litemoji in (/var/www/inventory.net.adk.de/vendor/elvanto/litemoji/src)
symfony/console in (/var/www/inventory.net.adk.de/vendor/symfony/console)
scssphp/scssphp in (/var/www/inventory.net.adk.de/vendor/scssphp/scssphp/src)
laminas/laminas-mail in (/var/www/inventory.net.adk.de/vendor/laminas/laminas-mail/src/Protocol)
laminas/laminas-mime in (/var/www/inventory.net.adk.de/vendor/laminas/laminas-mime/src)
rlanvin/php-rrule in (/var/www/inventory.net.adk.de/vendor/rlanvin/php-rrule/src)
ramsey/uuid in (/var/www/inventory.net.adk.de/vendor/ramsey/uuid/src)
psr/log in (/var/www/inventory.net.adk.de/vendor/psr/log/Psr/Log)
psr/simple-cache in (/var/www/inventory.net.adk.de/vendor/psr/simple-cache/src)
psr/cache in (/var/www/inventory.net.adk.de/vendor/psr/cache/src)
league/csv in (/var/www/inventory.net.adk.de/vendor/league/csv/src)
mexitek/phpcolors in (/var/www/inventory.net.adk.de/vendor/mexitek/phpcolors/src/Mexitek/PHPColors)
guzzlehttp/guzzle in (/var/www/inventory.net.adk.de/vendor/guzzlehttp/guzzle/src)
guzzlehttp/psr7 in (/var/www/inventory.net.adk.de/vendor/guzzlehttp/psr7/src)
glpi-project/inventory_format in (/var/www/inventory.net.adk.de/vendor/glpi-project/inventory_format/lib/php)
wapmorgan/unified-archive in (/var/www/inventory.net.adk.de/vendor/wapmorgan/unified-archive/src)
paragonie/sodium_compat in (/var/www/inventory.net.adk.de/vendor/paragonie/sodium_compat/src)
symfony/cache in (/var/www/inventory.net.adk.de/vendor/symfony/cache)
html2text/html2text in (/var/www/inventory.net.adk.de/vendor/html2text/html2text/src)
symfony/css-selector in (/var/www/inventory.net.adk.de/vendor/symfony/css-selector)
symfony/dom-crawler in (/var/www/inventory.net.adk.de/vendor/symfony/dom-crawler)
twig/twig in (/var/www/inventory.net.adk.de/vendor/twig/twig/src)
twig/string-extra in (/var/www/inventory.net.adk.de/vendor/twig/string-extra)
symfony/polyfill-ctype not found
symfony/polyfill-iconv not found
symfony/polyfill-mbstring not found
symfony/polyfill-php80 not found
symfony/polyfill-php81 not found
symfony/polyfill-php82 in (/var/www/inventory.net.adk.de/vendor/symfony/polyfill-php82)
league/oauth2-client in (/var/www/inventory.net.adk.de/vendor/league/oauth2-client/src/Provider)
league/oauth2-google in (/var/www/inventory.net.adk.de/vendor/league/oauth2-google/src/Provider)
thenetworg/oauth2-azure in (/var/www/inventory.net.adk.de/vendor/thenetworg/oauth2-azure/src/Provider)
phpCas version 1.6.0 in (/usr/share/php/CAS/source)

SQL replicas
 
Not active

Notifications
 
Way of sending emails: PHP

Plugins list
 
    datainjection        Name: Data injection                 Version: 2.13.5     State: Enabled                                 
        Install Method: Manual
    formcreator          Name: Form Creator                   Version: 2.13.9     State: Enabled                                 
        Install Method: Manual
    glpiinventory        Name: GLPI Inventory                 Version: 1.3.5      State: Enabled                                 
        Install Method: Manual

Anything else?

No response

troloff commented 3 months ago

I did an inventory run on two machines which show up as different versions. This is the output of the example software above:

Agent 1:

   <SOFTWARES>
      <ARCH>x86_64</ARCH>
      <FROM>registry</FROM>
      <GUID>Camunda Modeler</GUID>
      <INSTALLDATE>15/06/2024</INSTALLDATE>
      <NAME>Camunda Modeler</NAME>
      <PUBLISHER>Camunda</PUBLISHER>
      <SYSTEM_CATEGORY>application</SYSTEM_CATEGORY>
      <UNINSTALL_STRING>C:\Program Files\Camunda Modeler\uninstall.bat</UNINSTALL_STRING>
      <URL_INFO_ABOUT>https://camunda.com/de/download/modeler/</URL_INFO_ABOUT>
      <VERSION>5.24.0</VERSION>
    </SOFTWARES>

Agent 2:

    <SOFTWARES>
      <ARCH>x86_64</ARCH>
      <FROM>registry</FROM>
      <GUID>Camunda Modeler</GUID>
      <INSTALLDATE>14/06/2024</INSTALLDATE>
      <NAME>Camunda Modeler</NAME>
      <PUBLISHER>Camunda</PUBLISHER>
      <SYSTEM_CATEGORY>application</SYSTEM_CATEGORY>
      <UNINSTALL_STRING>C:\Program Files\Camunda Modeler\uninstall.bat</UNINSTALL_STRING>
      <URL_INFO_ABOUT>https://camunda.com/de/download/modeler/</URL_INFO_ABOUT>
      <VERSION>5.24.0</VERSION>
    </SOFTWARES>

As far as I can see, the only difference is the install date.

troloff commented 3 months ago

Maybe related: https://forum.glpi-project.org/viewtopic.php?id=290215

troloff commented 3 months ago

Could this https://github.com/glpi-project/glpi/pull/16598 be the cause? Maybe, another "version" (in terms of glpi) gets added, if installation dates differ? (no developer here, so just a guess...)

troloff commented 3 months ago

CSV export of the glpi_softwareversions table (filtered according to the example above): glpi_softwareversions_202406180753.csv

trasher commented 3 months ago

16598 targets next major version, it's not included in 10.x series.

We need inventory file(s) in order to reproduce from a fresh install.

troloff commented 3 months ago

Do you mean the complete xml produced on clients after installation of software? What do you mean by "fresh install" - glpiinventory agent?

troloff commented 2 months ago

Just upgraded to glpi 10.0.16, and we are still seeing duplicate software versions. I even purged glpi_softwareversion and glpi_item_softwareversion tables to make sure there is no conflict because of already existing duplicates. Please find attached the software inventories of two hosts in question. roloffvm2020-2024-07-01-12-57-04.xml.txt ernstvm2022-2024-07-10-21-16-50.xml.txt

stonebuzz commented 2 months ago

Hi @troloff

After importing (several times) your files, I do not reproduce

Camunda Modeler software does not have a duplicated version

image

When importing XML files, do you see errors in GLPI logs (php-error.log / sql-error.log)?

One clue could be an inconsistent state in your database data.

The presence of several identical versions (name, arch, os, linked software) should not be possible.

GLPI cannot find a single version and therefore adds it by default (1 +1 +1 +1 ....)

You can try to clean up the software with the following SQL queries

DELETE FROM glpi_softwares;
DELETE FROM glpi_softwareversion;
DELETE FROM glpi_items_softwareversions;

Software information will be updated as and when the inventory is updated.

troloff commented 2 months ago

Deleted the three tables as suggested. Now, software inventory rebuilds, but we have duplicate versions for every software/version installed on every computer.

software_versions software

There are no errors in sql-error.log

Only error in php-errors.log (but only <10 times a day) is

[2024-06-28 16:27:45] glpiphplog.WARNING:   *** PHP Warning (2): Undefined array key "HTTP_REFERER" in /var/www/inventory.net.adk.de/plugins/glpiinventory/inc/deploypackage.class.php at lin>
  Backtrace :
  src/MassiveAction.php:842                          PluginGlpiinventoryDeployPackage->getForbiddenStandardMassiveAction()
  src/Search.php:1835                                MassiveAction::getAllMassiveActions()
  src/Search.php:213                                 Search::displayData()
  src/Search.php:177                                 Search::showList()
  ...s/glpiinventory/inc/deploypackage.class.php:454 Search::show()
  plugins/glpiinventory/front/deploypackage.php:48   PluginGlpiinventoryDeployPackage->showList()
  public/index.php:82                                require()

We did not see this behaviour before update to GLPI 10.0.15 / GLPI-Agent 1.9 (which we unfortunately did in the same run...).

troloff commented 2 months ago

Maybe, I can add some debugging output to /src/Inventory/Asset/Software.php (line 410 ff?) to see what is happening, if new software "arrives" from the GLPI agents?

troloff commented 2 months ago

Just to be sure:

root@glpi:/var/www/glpi # php bin/console database:check_schema_integrity
Das Datenbankschema ist in Ordnung.

(which reads "database schema is ok")

stonebuzz commented 2 months ago

Do you have several entities?

Are the PCs in different entities?

How is software management configured in the entity?

image

troloff commented 2 months ago

Software management looks the same to me: entities_assets

There is only one "root" entity, entities

and all computers belong to this entity (did a search for "entity is not root" and received an empty result). computers_not_in_root_entity

troloff commented 2 months ago

Are inventories processed in parallel? If so, might there be a race condition in https://github.com/glpi-project/glpi/blob/d721c1f0734f979eea24d024153269429fa82363/src/Inventory/Asset/Software.php#L247 ?

Like:

squirellpl commented 1 month ago

Same problem with two different installations. GLPI 10.0.16, Agent 1.10, GLPI Inventory plugin 1.3.5. Some versions are without OS, some are I guess from different OS version. But this is the same software and version i.e. 7-zip 23.01 x86_64

trasher commented 1 month ago

Operating System is part of the dedup key; as it already was in the plugin. As already said, we need a way to reproduce on a fresh GLPI instance, so please provide related inventories files and detailed steps to do so.

brunosrodrigues commented 1 month ago

Same problem here. Any ideas?

Nour-Ivory commented 1 month ago

Same issue here, Server version 10.0.16 and clients on version 1.10 (both Windows and MacOS) . I have even deleted the software tables from the glpi database and allowed the agents to repopulate the tables and I am getting single entries for every installed application despite the fact that most of them are the same version See screenshots below, in some cases, it is the same version of the software and yet it is listed 20 times

Any ideas of what the issue might be?

Screenshot 2024-08-07 at 15 20 32 Screenshot 2024-08-07 at 15 21 03 Screenshot 2024-08-07 at 15 21 50 Screenshot 2024-08-07 at 15 22 08
trasher commented 1 month ago

Not reproducible on our side. Feel free to reopen with a reproducble use case on a fresh GLPI latest stable release installation.