Removing files from the index after updating asset indexes may fail silently

OscarBarrett commented 3 years ago

Description

If you have a large number of removed assets and go to update the asset indexes, attempting to remove missing folders and files from the index can fail however the UI shows it as being successful.

From the error log on request to admin/actions/utilities/asset-index-perform-action:

yii\base\ErrorException: Unknown: Input variables exceeded 1000. To increase the limit change max_input_vars in php.ini.
#2 /app/vendor/yiisoft/yii2/base/ErrorException.php(43): require
#1 /app/vendor/craftcms/cms/bootstrap/web.php(51): require
#0 index.php(20): null

Steps to reproduce

Something like this I imagine:

Create an S3 volume where the bucket has 1000 assets and sync it up. (When we ran into this we had 799 files to be deleted)
Change the S3 bucket to one that doesn't have any files.
Go to Utilities -> Asset Indexes, click Update Asset Indexes then choose to remove all. A tick will be shown but the above error appears in the log.

Additional info

Craft version: 3.5.15.1
PHP version: 7.4
Database driver & version: MySQL
Plugins & versions: AWS S3 1.2.11

andris-sevcenko commented 3 years ago

We'll be tweaking the indexing process in Craft 4.0 a bit. Meanwhile, this can be considered as a rather rare edge case and you can work around this by increasing the mentioned PHP setting.

benjaminkohl commented 3 years ago

For the record, increasing the max_input_vars does not fix the issue because Craft isn't accounting for the zero index. We are trying to clear an entire Asset source folder in Craft 3.6.12. We got this error when max_input_vars was 1,000. Craft tried to delete 1,001. We increased max_input_vars to 10,000 and Craft tried to delete 10,001 so we got the error again.

I hope this gets fixed before Craft 4.

craftcms / cms