Jobs get stuck at 99% or 100% and never clear queue

[justinholtweb]

Description

Jobs get stuck at 99% or 100% and never clear queue, causing other queue jobs to back up behind them. The jobs remain in an active state but appear to have finished according to the Feed Me log, so I'm not sure if this is due to my queue or Feed Me.

Notes: The site uses a custom value of 900 seconds for Feed Me TTR to allow larger jobs to finish. The site also has Async Queue in use.

Steps to reproduce

1. Configure Feed Me to use 900 seconds for TTR.
2. Start an import job (either via URL by cron job OR manually starting a job from the Feed Me interface.)
3. Run a job and watch it hit 99% or 100% but never clear queue.

Additional info

• Craft version: Craft Pro 3.8.7
• PHP version: 7.4.30
• Database driver & version: MariaDB 5.5.5
• Plugins & versions: Algolia | 3.1.0, Amazon S3 1.3.2, AsyncQueue 2.3.0, Cookies 1.1.15, CP Filters 1.3.1, Dumper 2.0.0, Empty Coalesce 1.0.6, Entry Instructions 1.0.7, Feed Me 4.6.2, Freeform 3.13.23, Google Maps 4.1.1, No-Cache 2.0.8, Redactor 2.10.12, Redirect Manager 1.1.1, Scout 2.7.2, SEOmatic 3.4.52, SuperSort 3.1.0, Super Table 2.7.5.1, Twig Perversion 2.2.0, Videos 2.1.0

[justinholtweb]

Turns out this appears to be an issue with Async Queue. When it is disabled and I process queue runners with a cron job every minute the jobs clear off with no problem. Closing this issue as unrelated to Feed Me.

[justinholtweb]

If anyone else runs into this I ended up hitting this bug again even with Async Queue disabled. The feed data I am having to work with is extremely large, the JSON is almost 198k lines long, and is provided by a third party so we don't have scope to reduce it (it is uploaded to the site via FTP by the provider 2 times a day). I ended up tweaking some SQL settings (I set BUFFER POOL SIZE to 1024MB, LOCK WAIT TIMEOUT to 120 seconds and WAIT TIMEOUT to 48800 seconds (default for that is 28800 seconds) and I finally got feeds to finish and clear. If you run into this yourself try editing those SQL settings to hopefully resolve.