[Bug]: [C] Sitemap Generation issue

[migbash]

Is there an existing issue for this?

• [x] I have searched the existing issues

🐞 Describe the bug

It appears that a case for an error occurs when the queue is triggered, in cases where:

1. (column) category :: fixture is present.
2. Many (workflow) permalink_queue process are triggered in succession.

🤔 Expected Behavior

Should work correctly and on-time.

(optional) 🖥️ Console Logs (or similar)

^ Has been accumulating processes (143 active accululated) for the sitemap-check process over the last day.

(optional) 📌 Additional Information

📆 First detected on:

25/06/24 at 10:00 CET during testing of links that were sent to the queue.

📌 Potential initial theorized cause(s)/solution(s):

Cause(s)

Believed (defintive) is that the issue is caused by the:

• large size of the temporary (intermediary) file for category :: fixture, and
• the execessive amount of queue workflow triggers sent to the cache-worker to process for sitemap-check, which contained the category :: fixture.

Potential solution

1. Throttle worker process in cases of which category :: fixture is the process that will be executed. For example: disable the queue workflows for sitemap update when a workflow is already active for the update of sitemap for category :: fixture.

2. In ddition, offset the worker process for the sitemap to omit and/or defer the category :: fixture for after the other more important categories (articles_articles, competitions, etc.) have been processed first.

🩹 Workarounds:

No workarounds found, other than rebooting the docker container, to restart process, as it was overwhelemed with jobs and stuck in processing.

[migbash]

🟧 UPDATE

Data in the intermediary sitemaps was corrupted (again), and was caused by the use of the manual regeneration attempt of brand new fixtures (not picked up by the workflow).

✅ Was corrected manually.

Correction of the outlined issue still needs to be corrected as soon as possible.