airflow celery flower - airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'celery' ...

kapuza commented 1 year ago

Apache Airflow version

2.7.2

What happened

I am using this docs. System: Ubuntu 22.04.3 LTS Python 3.10.12

Install like:

pip install -U pip; \
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"; \
AIRFLOW_VERSION='2.7.2'; \
pip install wheel; \
pip install psycopg2-binary; \
pip install virtualenv; \
pip install "apache-airflow[celery,redis,postgres]==${AIRFLOW_VERSION}" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt";

Installed ok, and work fine with LocalExecutor.

But try:

$ airflow celery flower
Usage: airflow [-h] GROUP_OR_COMMAND ...

Positional Arguments:
  GROUP_OR_COMMAND

    Groups
      config         View configuration
      connections    Manage connections
      dags           Manage DAGs
      db             Database operations
      jobs           Manage jobs
      pools          Manage pools
      providers      Display providers
      roles          Manage roles
      tasks          Manage tasks
      users          Manage users
      variables      Manage variables

    Commands:
      cheat-sheet    Display cheat sheet
      dag-processor  Start a standalone Dag Processor instance
      info           Show information about current Airflow and environment
      kerberos       Start a kerberos ticket renewer
      plugins        Dump information about loaded plugins
      rotate-fernet-key
                     Rotate encrypted connection credentials and variables
      scheduler      Start a scheduler instance
      standalone     Run an all-in-one copy of Airflow
      sync-perm      Update permissions for existing roles and optionally DAGs
      triggerer      Start a triggerer instance
      version        Show the version
      webserver      Start a Airflow webserver instance

Options:
  -h, --help         show this help message and exit

airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'celery' (choose from 'cheat-sheet', 'config', 'connections', 'dag-processor', 'dags', 'db', 'info', 'jobs', 'kerberos', 'plugins', 'pools', 'providers', 'roles', 'rotate-fernet-key', 'scheduler', 'standalone', 'sync-perm', 'tasks', 'triggerer', 'users', 'variables', 'version', 'webserver'), see help above.

Try:

$ celery 
Usage: celery [OPTIONS] COMMAND [ARGS]...

  Celery command entrypoint.

Options:
  -A, --app APPLICATION
  -b, --broker TEXT
  --result-backend TEXT
  --loader TEXT
  --config TEXT
  --workdir PATH
  -C, --no-color
  -q, --quiet
  --version
  --skip-checks          Skip Django core checks on startup. Setting the
                         SKIP_CHECKS environment variable to any non-empty
                         string will have the same effect.
  --help                 Show this message and exit.

Commands:
  amqp     AMQP Administration Shell.
  beat     Start the beat periodic task scheduler.
  call     Call a task by name.
  control  Workers remote control.
  events   Event-stream utilities.
  flower   Web based tool for monitoring and administrating Celery clusters.
  graph    The ``celery graph`` command.
  inspect  Inspect the worker at runtime.
  list     Get info from broker.
  logtool  The ``celery logtool`` command.
  migrate  Migrate tasks from one broker to another.
  multi    Start multiple worker instances.
  purge    Erase all messages from all known task queues.
  report   Shows information useful to include in bug-reports.
  result   Print the return value for a given task id.
  shell    Start shell session with convenient access to celery symbols.
  status   Show list of workers that are online.
  upgrade  Perform upgrade between versions.
  worker   Start worker instance.

How i can use celery in airflow?

What you think should happen instead

Documentation not work for this version.

How to reproduce

Just install not in docker and try run airflow celery flower

Operating System

Ubuntu 22.04.3 LTS

Versions of Apache Airflow Providers

aiohttp==3.8.6 aiosignal==1.3.1 alembic==1.12.0 amqp==5.1.1 annotated-types==0.6.0 anyio==4.0.0 apache-airflow==2.7.2 apache-airflow-providers-celery==3.3.4 apache-airflow-providers-common-sql==1.7.2 apache-airflow-providers-ftp==3.5.2 apache-airflow-providers-http==4.5.2 apache-airflow-providers-imap==3.3.2 apache-airflow-providers-sqlite==3.4.3 apispec==6.3.0 argcomplete==3.1.2 asgiref==3.7.2 async-timeout==4.0.3 attrs==23.1.0 Babel==2.13.0 backoff==1.10.0 billiard==4.1.0 blinker==1.6.3 cachelib==0.9.0 cattrs==23.1.2 celery==5.3.4 certifi==2023.7.22 cffi==1.16.0 charset-normalizer==3.3.0 click==8.1.7 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.3.0 clickclick==20.10.2 colorama==0.4.6 colorlog==4.8.0 ConfigUpdater==3.1.1 connexion==2.14.2 cron-descriptor==1.4.0 croniter==1.4.1 cryptography==41.0.4 Deprecated==1.2.14 dill==0.3.1.1 distlib==0.3.7 dnspython==2.4.2 docutils==0.20.1 email-validator==1.3.1 exceptiongroup==1.1.3 filelock==3.13.0 Flask==2.2.5 Flask-AppBuilder==4.3.6 Flask-Babel==2.0.0 Flask-Caching==2.0.2 Flask-JWT-Extended==4.5.3 Flask-Limiter==3.5.0 Flask-Login==0.6.2 Flask-Session==0.5.0 Flask-SQLAlchemy==2.5.1 Flask-WTF==1.2.1 flower==2.0.1 frozenlist==1.4.0 google-re2==1.1 googleapis-common-protos==1.60.0 graphviz==0.20.1 greenlet==3.0.0 grpcio==1.59.0 gunicorn==21.2.0 h11==0.14.0 httpcore==0.16.3 httpx==0.23.3 humanize==4.8.0 idna==3.4 importlib-metadata==6.8.0 importlib-resources==6.1.0 inflection==0.5.1 itsdangerous==2.1.2 Jinja2==3.1.2 jsonschema==4.19.1 jsonschema-specifications==2023.7.1 kombu==5.3.2 lazy-object-proxy==1.9.0 limits==3.6.0 linkify-it-py==2.0.2 lockfile==0.12.2 Mako==1.2.4 Markdown==3.5 markdown-it-py==3.0.0 MarkupSafe==2.1.3 marshmallow==3.20.1 marshmallow-oneofschema==3.0.1 marshmallow-sqlalchemy==0.26.1 mdit-py-plugins==0.4.0 mdurl==0.1.2 multidict==6.0.4 opentelemetry-api==1.20.0 opentelemetry-exporter-otlp==1.20.0 opentelemetry-exporter-otlp-proto-common==1.20.0 opentelemetry-exporter-otlp-proto-grpc==1.20.0 opentelemetry-exporter-otlp-proto-http==1.20.0 opentelemetry-proto==1.20.0 opentelemetry-sdk==1.20.0 opentelemetry-semantic-conventions==0.41b0 ordered-set==4.1.0 packaging==23.2 pathspec==0.11.2 pendulum==2.1.2 platformdirs==3.11.0 pluggy==1.3.0 prison==0.2.1 prometheus-client==0.17.1 prompt-toolkit==3.0.39 protobuf==4.21.12 psutil==5.9.5 psycopg2-binary==2.9.9 pycparser==2.21 pydantic==2.4.2 pydantic_core==2.10.1 Pygments==2.16.1 PyJWT==2.8.0 python-daemon==3.0.1 python-dateutil==2.8.2 python-nvd3==0.15.0 python-slugify==8.0.1 pytz==2023.3.post1 pytzdata==2020.1 PyYAML==6.0.1 referencing==0.30.2 requests==2.31.0 requests-toolbelt==1.0.0 rfc3339-validator==0.1.4 rfc3986==1.5.0 rich==13.6.0 rich-argparse==1.3.0 rpds-py==0.10.4 setproctitle==1.3.3 six==1.16.0 sniffio==1.3.0 SQLAlchemy==1.4.49 SQLAlchemy-JSONField==1.0.1.post0 SQLAlchemy-Utils==0.41.1 sqlparse==0.4.4 tabulate==0.9.0 tenacity==8.2.3 termcolor==2.3.0 text-unidecode==1.3 tornado==6.3.3 typing_extensions==4.8.0 tzdata==2023.3 uc-micro-py==1.0.2 unicodecsv==0.14.1 urllib3==1.26.17 vine==5.0.0 virtualenv==20.24.6 wcwidth==0.2.8 Werkzeug==2.2.3 wrapt==1.15.0 WTForms==3.0.1 yarl==1.9.2 zipp==3.17.0

Deployment

Other

Deployment details

pip install -U pip; \
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"; \
AIRFLOW_VERSION='2.7.2'; \
pip install wheel; \
pip install psycopg2-binary; \
pip install virtualenv; \
pip install "apache-airflow[celery,redis,postgres]==${AIRFLOW_VERSION}" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt";

Anything else

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Taragolis commented 1 year ago

I assume that you do not set correct executor type and in this case it won't load non-related configs in airflow cli

https://airflow.apache.org/docs/apache-airflow-providers-celery/3.3.4/cli-ref.html

Sub-commands celery Start celery components. Works only when using CeleryExecutor. For more information, see

❯ export AIRFLOW__CORE__EXECUTOR="CeleryExecutor"

❯ airflow celery --help
Usage: airflow celery [-h] COMMAND ...

Start celery components. Works only when using CeleryExecutor. For more information, see https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html

Positional Arguments:
  COMMAND
    flower    Start a Celery Flower
    stop      Stop the Celery worker gracefully
    worker    Start a Celery worker node

Optional Arguments:
  -h, --help  show this help message and exit

❯ airflow celery flower --help
Usage: airflow celery flower [-h] [-A BASIC_AUTH] [-a BROKER_API] [-D] [-c FLOWER_CONF] [-H HOSTNAME] [-l LOG_FILE] [--pid [PID]] [-p PORT] [--stderr STDERR] [--stdout STDOUT]
                             [-u URL_PREFIX] [-v]

Start a Celery Flower

Optional Arguments:
  -h, --help            show this help message and exit
  -A, --basic-auth BASIC_AUTH
                        Securing Flower with Basic Authentication. Accepts user:password pairs separated by a comma. Example: flower_basic_auth = user1:password1,user2:password2
  -a, --broker-api BROKER_API
                        Broker API
  -D, --daemon          Daemonize instead of running in the foreground
  -c, --flower-conf FLOWER_CONF
                        Configuration file for flower
  -H, --hostname HOSTNAME
                        Set the hostname on which to run the server
  -l, --log-file LOG_FILE
                        Location of the log file
  --pid [PID]           PID file location
  -p, --port PORT       The port on which to run the server
  --stderr STDERR       Redirect stderr to this file
  --stdout STDOUT       Redirect stdout to this file
  -u, --url-prefix URL_PREFIX
                        URL prefix for Flower
  -v, --verbose         Make logging output more verbose

kapuza commented 1 year ago

Thank you.

Who came up with the idea of giving help based on set parameters, and not all possible options?

# Create base config
airflow config list

# Edit config for CeleryExecutor
sed -i 's/^executor.*$/executor = CeleryExecutor/' ~/airflow/airflow.cfg

# Try airflow celery flower
airflow celery flower

potiuk commented 1 year ago

Who came up with the idea of giving help based on set parameters, and not all possible options?

Why do you want to know who? What are you going to do with that knowledge, I wonder?

BTW. In case, by any chance, you also happen to be interested WHY, (rather than who), you will find the information in the CLI https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html docs - look for executors

Basically Celery executor is part of "celery" provider and Celery provider "extends" CLI with new commands dynamically. Celery command is NOT airflow command, it's command contributed by Celery Executor.

If you would like to discover all providers and ask them for all CLI commands they could contribute it could delay your --help command sometimes by up to a second. Which is bad idea. That's why decison was made that rather than finding all possible executors and all possible CLI commands, only the current executor is quickly checked to see what commands it provides.

And if you have idea how to do it in the way to show possible executor commands (including upcoming executors - Amazon ones, possibly Google ones, possibly some others, and possibly commands from executors coming from providers developed outside by 3rd-parties and do it in the way that will not impact speed of CLI, PRs are most welcome.

But I am not sure if you are interested in answer "why" - your "who" question was a bit strange in this context, almost as if you knew that the person was wrong without knowing or even attempting to explore "why" it has happend. It's pretty odd choice of asking questions if you ask me.

apache / airflow