polyaxon / polyaxon

MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
https://polyaxon.com
Apache License 2.0
3.57k stars 314 forks source link

CrashLoopBackoff when deploying Polyaxon CE with async workers broker #1533

Closed ClementMaliet closed 9 months ago

ClementMaliet commented 9 months ago

Unclear documentation

The CE setup documentation indicates the the scheduler and broker services may be enabled to allow for increased concurrency (https://polyaxon.com/docs/setup/platform/community-edition/#enable-extra-services).

The relevant configuration values I use are as follow:

gateway:
  replicas: 2
  concurrency: 2

scheduler:
  enabled: true
  replicas: 1
  concurrency: 2

redis:
  enabled: true

broker: redis

However, the scheduler service never starts and falls into a CrashLoopBackoff with the following error in the logs:

Running...                                                                                                                                                                                                                                 
Usage: haupt queues [OPTIONS]                                                                                                                                                                                                              
Try 'haupt queues --help' for help.                                                                                                                                                                                                        

Error: Got unexpected extra argument (scheduler)

Are any additional steps necessary on my part to enable the scheduler service in the Polyaxon CE ?

Thanks in advance for your continued support.

bastianerh commented 9 months ago

This is a regression in the deployment CLI, we will fix it in v2.1.2.

I would suggest to disable the scheduler, the instance without scheduler has all features and should work as expected. You can enable the scheduler when you upgrade to future versions.

The scheduler is only necessary if you have large user base and a large amount of jobs scheduled, otherwise it's better not to add the additional complexity to your deployment.

polyaxon-team commented 9 months ago

To be released as part of v2.1.2