Open fzhan opened 3 months ago
To address the issue of worker instances continuously restarting after upgrading to Superset version 4.0.0, consider the following steps:
Compatibility Check: Ensure all components, including Python 3.11 and Node 16, are compatible with Superset 4.0.0. While Python 3.11 is compatible, Node.js version 18.20.1 is recommended for the best development experience with Superset's frontend.
Review Custom Bootstrap Script: Inspect your custom bootstrap script for any conflicts or deprecated packages that might not be compatible with Superset 4.0.0.
Worker Configuration: Verify the command
used to start worker instances in your helm chart is correct for Superset 4.0.0, focusing on the path to superset_bootstrap.sh
and the Celery command.
Liveness and Readiness Probes: Ensure the configuration for these probes in the supersetWorker
section is correctly set up for Superset 4.0.0, as incorrect configurations can lead to continuous restarts.
Environment Variables and Configurations: Check that all necessary environment variables and configurations are correctly set up, especially those derived from secrets or config maps.
Log Analysis: Review the logs of the restarting worker instances for errors or warnings that could indicate the cause of the restarts.
Dependencies and Package Versions: Make sure all dependencies specified in your bootstrap script or helm chart are compatible with Superset 4.0.0.
Resource Limits: Review and adjust the resource limits and requests for the worker instances if they are being terminated due to resource constraints.
Review Helm Chart Annotations and Checksums: Update checksums for configurations if changes are made, ensuring Kubernetes recognizes and applies these changes.
Security Concerns with runAsUser
: Running containers as root (UID 0) is discouraged due to security risks. Consider running containers with a non-root user.
These steps should help identify and resolve the issue with worker instances restarting. If the problem persists, further investigation into Superset's change logs for version 4.0.0 may be necessary to identify any breaking changes or additional migration steps required.
To continue the conversation, mention @dosu.
It's an issue with Liveness probe :
Liveness probe failed: ['/usr/local/bin', '/app/pythonpath', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app', '/app/configs/'] Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py] logging was configured successfully 2024-05-13 02:49:48,445:INFO:superset.utils.logging_configurator:logging was configured successfully 2024-05-13 02:49:48,447:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'> Error: No nodes replied within time constraint
After running 'celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME' manually in worker pod, response was Error:
root@superset-worker-54fcfd4b-bzpzd:/app# celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME
['/usr/local/bin', '/app/pythonpath', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app', '/app/configs/']
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
logging was configured successfully
2024-05-13 02:58:06,359:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-05-13 02:58:06,361:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Error: No nodes replied within time constraint
After done some digging, it appears to be related to the Celery version issue with Redis. https://github.com/apache/superset/pull/28471
After upgrade to celery 5.4, flower can see workers listed, but the ping for liveness within worker still fails with no nodes replied.
LivenessProbe still fails with 'Error: No nodes replied within time constraint' as result from "celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME"
@dosu-bot
Bug description
After upgrade to 4.0.0, the workers keeps restarting
How to reproduce the bug
Install helm with this value.yaml
Screenshots/recordings
Logs from the worker pod
Superset version
4.0.0
Python version
3.10
Node version
16
Browser
Chrome
Additional context
No response
Checklist