marc-pan / kg-cicada

Simple personal examples
Apache License 2.0
0 stars 0 forks source link

Investigation Airflow #34

Open marc-pan opened 3 weeks ago

marc-pan commented 3 weeks ago

The Airflow python environment is managed by Hatch tools.

The Airflow can be installed through following script with Python 3.12.

#!/bin/env bash

AIRFLOW_VERSION=2.10.2

# Extract the version of Python you have installed. If you're currently using a Python version that is not supported by Airflow, you may want to set this manually.
# See above for supported versions.
PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

# For example this would install 2.10.2 with python 3.8: https://raw.githubusercontent.com/apache/airflow/constraints-2.10.2/constraints-3.8.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

To start up an Airflow standalone instance by executing following command

# export AIRFLOW_HOME=~/airflow
# airflow standalone

By default, the local port 8080 is used for Airflow UI, http://localhost:8080 and the user admin and password can be obtained from the airflow start up output.

standalone | Airflow is ready
standalone | Login with username: admin  password: cEnTVWaKqzqqZ4cC
standalone | Airflow Standalone is for development purposes only. Do not use this in production!

If the port 8080 is occupied, it can be set in $AIRFLOW_HOME/airflow.cfg file.