Closed laszlokertesz closed 3 months ago
I found the log entries. It does seem to point to the glpi inventory plugin?
/var/www/html/glpi/files/_log/sql-errors.log: SQL: DELETE `glpi_logs` FROM `glpi_logs` WHERE `itemtype` = 'Agent' AND `items_id` = '1051'
/var/www/html/glpi/files/_log/sql-errors.log- Error: Lock wait timeout exceeded; try restarting transaction
/var/www/html/glpi/files/_log/sql-errors.log- Backtrace :
/var/www/html/glpi/files/_log/sql-errors.log- src/DBmysql.php:1579
/var/www/html/glpi/files/_log/sql-errors.log- src/CommonDBTM.php:841 DBmysql->delete()
/var/www/html/glpi/files/_log/sql-errors.log- src/CommonDBTM.php:787 CommonDBTM->cleanHistory()
/var/www/html/glpi/files/_log/sql-errors.log- src/CommonDBTM.php:2068 CommonDBTM->deleteFromDB()
/var/www/html/glpi/files/_log/sql-errors.log- src/CommonDBTM.php:4627 CommonDBTM->delete()
/var/www/html/glpi/files/_log/sql-errors.log- src/Inventory/Asset/MainAsset.php:786 CommonDBTM->deleteByCriteria()
/var/www/html/glpi/files/_log/sql-errors.log- src/RuleImportAsset.php:969 Glpi\Inventory\Asset\MainAsset->rulepassed()
/var/www/html/glpi/files/_log/sql-errors.log- src/Rule.php:1525 RuleImportAsset->executeActions()
/var/www/html/glpi/files/_log/sql-errors.log- src/RuleCollection.php:1640 Rule->process()
/var/www/html/glpi/files/_log/sql-errors.log- src/Inventory/Asset/MainAsset.php:573 RuleCollection->processAllRules()
/var/www/html/glpi/files/_log/sql-errors.log- src/Inventory/Inventory.php:709 Glpi\Inventory\Asset\MainAsset->handle()
/var/www/html/glpi/files/_log/sql-errors.log- src/Inventory/Inventory.php:340 Glpi\Inventory\Inventory->handleItem()
/var/www/html/glpi/files/_log/sql-errors.log- src/Inventory/Request.php:360 Glpi\Inventory\Inventory->doInventory()
/var/www/html/glpi/files/_log/sql-errors.log- src/Inventory/Request.php:90 Glpi\Inventory\Request->inventory()
/var/www/html/glpi/files/_log/sql-errors.log- src/Agent/Communication/AbstractRequest.php:359 Glpi\Inventory\Request->handleAction()
/var/www/html/glpi/files/_log/sql-errors.log- src/Agent/Communication/AbstractRequest.php:271 Glpi\Agent\Communication\AbstractRequest->handleJSONRequest()
/var/www/html/glpi/files/_log/sql-errors.log- front/inventory.php:92 Glpi\Agent\Communication\AbstractRequest->handleRequest()
/var/www/html/glpi/files/_log/sql-errors.log- plugins/glpiinventory/front/communication.php:72 include_once()
/var/www/html/glpi/files/_log/sql-errors.log- plugins/glpiinventory/index.php:45 include_once()
/var/www/html/glpi/files/_log/sql-errors.log- {"user":"inventory@inventory.3pillarglobal.com","mem_usage":"87.900\", 6.13Mio)"}
I traced it to src/Inventory/Asset/MainAsset.php, part of Glpi proper.
Line 834
//check for any old agent to remove
$agent = new \Agent();
$agent->deleteByCriteria([
'itemtype' => $this->item->getType(),
'items_id' => $items_id,
'NOT' => [
'id' => $this->agent->fields['id']
]
]);
If i comment these lines out, everything returns to normal. Is this section really necessary?
This code deletes the agent from the asset currently being inventoried if it has changed (for example, you switch from Fusion to GLPI-agent).
This will lead to a cascade of database cleanup, particularly in the glpi_logs table, where all entries relating to this itemtype
and item_id
must be deleted.
What does the following SQL query return?
SELECT count(*) FROM `glpi_logs` WHERE `itemtype` = 'Agent' AND `items_id` = '1051'
Best regards
This code deletes the agent from the asset currently being inventoried if it has changed (for example, you switch from Fusion to GLPI-agent).
This will lead to a cascade of database cleanup, particularly in the glpi_logs table, where all entries relating to this
itemtype
anditem_id
must be deleted.What does the following SQL query return?
SELECT count(*) FROM `glpi_logs` WHERE `itemtype` = 'Agent' AND `items_id` = '1051'
Best regards
About 50 000 000 lines. Obviously this is our issue here. Actualy the issue is that this is trigered for hundreds of agents in a few minutes. I let them run in the weekend but none finished even after 12 hours and the server was mostly unresponsive for Web UI management, the logs were full of timed ot transactions.
And we switched to glpi agent a while ago. But only after we rebooted the server this was triggered for some reason.
Are all your tables migrated to using InnoDB instead of MyISAM? There are a lot of performance benefits including smarter locking. With MyISAM, the entire table is locked (made read-only) during a DELETE operation.
Are all your tables migrated to using InnoDB instead of MyISAM? There are a lot of performance benefits including smarter locking. With MyISAM, the entire table is locked (made read-only) during a DELETE operation.
I think so. The logs table definitely is.
@laszlokertesz
Can you view the agent's history with the ID 1051
Do you see anything unusual (an untimely change of linked item, something else)?
There has been no activity on this issue for some time and therefore it is considered stale and will be closed automatically in 10 days.
If this issue is related to a bug, please try to reproduce on latest release. If the problem persist, feel free to add a comment to revive this issue. If it is related to a new feature, please open a topic to discuss with community about this enhancement on suggestion website.
You may also consider taking a subscription to get professionnal support or contact GLPI editor team directly.
Code of Conduct
Is there an existing issue for this?
Version
10.0.12
Bug description
After reboot of the server and start of apache and mariadb (this happened both with 10.0.9 and 10.0.12) the database becomes flooded with queries like
I counted 350 at a certain point. This is a big issue since our logs table is really big (around 800 000 000 records) and these queries bog the machine down until the service basically becomes unresponsive. These are related to incoming inventories (if i disable it, after a while the server becomes serviceable), i presume every agent that reports in gets a query like this. But these take very long time, sometimes they get killed because of timeouts and result in rollbacks.
I don't know why these are triggered, the agents that appear on the list are both new and old. Can someone give some pointers where can i disable this, i can do some background cleanup if needed but tying these queries to inventory requests that result in the web server being overwhelmed is not really working out.
Relevant log output
No response
Page URL
No response
Steps To reproduce
I just restarted the server (had some 70 days uptime) and this behavior started.
Your GLPI setup information
Information about system installation and configuration
Server
GLPI constants
Libraries
LDAP directories
SQL replicas
Notifications
Plugins list
Anything else?
We have around 3000 endpoints reporting. The behavior started after upgrading to 10.0.12, but even after a rollback to a backup with 10.0.9 and reboot it did the same so this is not tied to version 10.0.12, but to something else, some cleanup routine i guess but i haven't found it's source.