Add verbose / detailed output for Indexers into a separate file

[pmonosolo]

Description

Its been 2.5 years since I started to work on M2 and all this time, the indexers are the most complex, convoluted part of Magento to test and debug.

Example

^^^No errors are showing up whenever there is an issue with an indexer. We dont know what time it started or finished or if it ever finished. Error logs also do not show anything.

The errors usually show up in terms of general store errors - so when the error happens, you dont know that this error is related to an indexer or its just a random error from customer / admin actions.

I would like to output ALL the information the indexers showing when the Cron Tasks indexer_update_all_views and indexer_reindex_all_invalid.

It doesnt have to show which entities / products / prices it's working on, just that it:

• started the index
• index finished
• error happened (show details of which indexer was running)
• timeout issues
• over memory issues

SOLUTION It could be as simple as creating the indexer tasks with Cron with an output file:

php bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1 >> /var/www/php/var/log/indexer.log 2>&1

Expected behavior

Output verbose information on indexers running in the background into a separate file.

Benefits

Improve testing, debugging, understanding of the indexing processes.

Additional information

No response

Release note

No response

[m2-assistant[bot]]

Hi @pmonosolo. Thank you for your report. To speed up processing of this issue, make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, review the Magento Contributor Assistant documentation.

Add a comment to assign the issue: @magento I am working on this

To learn more about issue processing workflow, refer to the Code Contributions.

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

  :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[pmonosolo]

As a small workaround you can add to your: /vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php

Line 265:

$this->_shell->execute( $phpPath . ' %s cron:run --group=' . $groupId . ' --' . Cli::INPUT_KEY_BOOTSTRAP . '=' . self::STANDALONE_PROCESS_STARTED . '=1 >> /var/www/php/var/log/indexer-process-errors.log 2>&1', [ BP . '/bin/magento' ] );

I'm running some tests on this now.

UPDATE: Looks like there's no ouput when you run a process like this. Log file is created, but there's no report of which indexer is being processed. No wonder there's no output of indexer errors anywhere!

If you turn off cron and just run this via CLI: php bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1 >> /var/www/php//var/log/indexer-test.log 2>&1

The output is shown in the file.

I assume this is some sort of limitation due to Parent / Child processes running separately?