Pending image transforms jobs can hang forever

OscarBarrett commented 4 years ago

Description

Sometimes we run into an issue where an asset associated to an entry gets into a bad state and no longer exists, even though Craft thinks it does. I'm not sure exactly when this happens, but my guess is that it's when using an Amazon S3 volume and the image is missing from the S3 bucket. I believe this happened to us when we changed a large amount of entry slugs and certain asset folders included the slug in the path, and then some associated jobs didn't get to run to their entirety.

When this happens, pending image transform jobs hang on generating the transform and never complete.

Going to the edit page of an entry that is using the asset throws an error:

yii\web\ServerErrorHttpException: Image transform cannot be created.
#10 /app/vendor/craftcms/cms/src/controllers/AssetsController.php(1185): actionGenerateTransform
#9 /app/vendor/craftcms/cms/src/controllers/AssetsController.php(0): call_user_func_array
#8 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): runWithParams
#7 /app/vendor/yiisoft/yii2/base/Controller.php(157): runAction
#6 /app/vendor/craftcms/cms/src/web/Controller.php(178): runAction
#5 /app/vendor/yiisoft/yii2/base/Module.php(528): runAction
#4 /app/vendor/craftcms/cms/src/web/Application.php(291): runAction
#3 /app/vendor/craftcms/cms/src/web/Application.php(559): _processActionRequest
#2 /app/vendor/craftcms/cms/src/web/Application.php(270): handleRequest
#1 /app/vendor/yiisoft/yii2/base/Application.php(386): run
#0 index.php(21): null

GraphQL queries that would result in this asset being requested will fail to complete and appear to just time out.

The web log includes an error about not being able to create a transform.

Ideally the job should fail, or mark the asset as failed. GraphQL queries should not also hang on this.

Steps to reproduce

I haven't come up with a reproduction yet. I'll continue to work on a reproduction.

Additional info

Craft version: 3.4.29.1
PHP version: 7.2
Database driver & version: MySQL 5.7
Plugins & versions: Amazon S3 1.2.8

andris-sevcenko commented 4 years ago

As of Craft 3.4.28, the transforms should fail a soon as it's clear the file is not available and Craft shouldn't be hanging on this.

If you could come up with reproduction steps, I'd be very grateful!

OscarBarrett commented 4 years ago

I've spent some time trying to reproduce the issue, and have gone through the relevant Craft code to look for any signs of where we were getting stuck (it does look a lot more robust with the recent changes :+1:), but haven't been able to get the pending transforms job (or GraphQL) to stall again.

I've looked into the asset in question and can confirm that it was missing from our bucket (not due to Craft though).

I don't think there's much more to be done at this point without any additional information, so I'll close this for now. If we do run into this again on the same site I'll take a snapshot of the database and debug further. Unfortunately at the time the priority was to get the site working again 😅

craftcms / cms