Closed padbk closed 1 month ago
Hey there, @padbk! 👋 I'm here to assist with any bugs, questions, or contributions you have regarding the repository. Let's tackle this together!
To resolve the issue with Celery 4.1.0rc1 on Kubernetes where you receive an error about an unregistered task 'reports.scheduler' every minute on the worker node, you need to ensure that the task is properly registered in your Celery configuration. The error indicates that the task 'reports.scheduler' is not recognized by the worker, which suggests that it might not be included in the imports
section of your Celery configuration.
Here is an example of how you can configure your Celery settings to include the 'reports.scheduler' task:
configOverrides:
celery_conf: |
from celery.schedules import crontab
class CeleryConfig:
broker_url = f"redis://{env('REDIS_HOST')}:{env('REDIS_PORT')}/0"
imports = (
"superset.sql_lab",
"superset.tasks.cache",
"superset.tasks.scheduler", # Ensure this line is included
)
result_backend = f"redis://{env('REDIS_HOST')}:{env('REDIS_PORT')}/0"
task_annotations = {
"sql_lab.get_sql_results": {
"rate_limit": "100/s",
},
}
beat_schedule = {
"reports.scheduler": {
"task": "reports.scheduler",
"schedule": crontab(minute="*", hour="*"),
},
"reports.prune_log": {
"task": "reports.prune_log",
'schedule': crontab(minute=0, hour=0),
},
'cache-warmup-hourly': {
"task": "cache-warmup",
"schedule": crontab(minute="*/30", hour="*"),
"kwargs": {
"strategy_name": "top_n_dashboards",
"top_n": 10,
"since": "7 days ago",
},
}
}
CELERY_CONFIG = CeleryConfig
Make sure that the superset.tasks.scheduler
module is correctly specified in the imports
section. This should resolve the issue of the unregistered task error. Additionally, ensure that the reports.scheduler
task is defined in your codebase as shown below:
from superset.extensions import celery_app
@celery_app.task(name="reports.scheduler")
def scheduler() -> None:
"""
Celery beat main scheduler for reports
"""
# Task implementation
This should ensure that the task is recognized and properly registered by the Celery worker [1][2].
To continue talking to Dosu, mention @dosu.
Please feel free to mention this on the official vote thread if you'd like to draw more attention to it, and see if others are experiencing the same. The release is currently in testing/voting, so it'd be appreciated :) Let me know if you're not on the developer mailing list and I can provide more instructions.
Turns out the imports line I was using was out of date and it needs to be changed. Now mine reads:
imports = ("superset.sql_lab", "superset.tasks.scheduler", "superset.tasks.thumbnails", "superset.tasks.cache", )
Not sure what has changed to make "superset.tasks" no longer work in 4.1.0, but the docs seem up to date.
I had the same issue and it went away after making that one-line change you suggest. Thanks! I have been using this same config without issues since 2.0.0, I guess something changed for 4.1.0 that made it finally not work.
I noticed warnings around this in docker-compose last week, and connected this issue when looking at 4.1 blockers.
I think I should be fixing the root cause here -> https://github.com/apache/superset/pull/29862
Bug description
Getting the following every minute on the worker node:
And no reports run at all.
I don't have this issue in 4.0.2 with the same config
How to reproduce the bug
Installed 4.1.0rc1-py310 on k8s
Screenshots/recordings
No response
Superset version
master / latest-dev
Python version
3.10
Node version
Not applicable
Browser
Not applicable
Additional context
No response
Checklist