Apache Superset 3.0.2 Scheduled reports didn't run as expected as per the Report Schedule

Bug description

We have a set of Daily Scheduled Reports at a particular time, sometimes we get the reports over email on schedule time and sometimes we don't, we don't see any kind of failure in execution logs of the reports on the superset Console.

To test this in detail, i had setup a report to be triggered every 4 min's, the behavior remains the same. Sometimes we get Reports Every 4 mins and suddenly gap of 30 to 40 mins average is seen in between, Really Surprised with this Behavior of the tool.

We Verified on Superset Flower UI, we don't see any big queue in active tasks section (1 or 2 tasks are running usually) and no failure is seen on the stats.

When debugged on Pod level logs: superset celerybeat pod doesn't show any error,

But Superset worker node throws an error Everyday @ 07:45:00,

Getting this error on superset worker node: [2024-03-13 07:45:00,710: WARNING/ForkPoolWorker-1] A downstream warning occurred while generating a report: 99a3dcff-6c19-493a-123546. Report Schedule not found. raise ReportScheduleNotFoundError() superset.reports.commands.exceptions.ReportScheduleNotFoundError: Report Schedule not found

How to reproduce the bug

We have deployed Apache Superset 3.0.2 using Helm Charts in GKE Autopilot Cluster in gcp. No major changes are made to helm configurations.

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

I don't know

Browser

Chrome

Additional context

Some guy on stackoverflow also posted for similar error: https://stackoverflow.com/questions/77787440/superset-celery-beat-daily-scheduled-reports-failed-in-some-days-need-assistanc/78160765#78160765

Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

apache / superset