carbonfact / lea

🏃‍♀️ Minimalist alternative to dbt
Apache License 2.0
202 stars 6 forks source link

Extending the CLI #20

Open MaxHalford opened 9 months ago

MaxHalford commented 9 months ago

lea's CLI can actually be extended. For instance, this is what we do at Carbonfact:

import datetime as dt
import os
import getpass

from google.oauth2 import service_account

import lea

def make_client(production):
    username = None if production else str(os.environ.get("LEA_USERNAME", getpass.getuser()))
    return BigQueryClient(
        credentials=service_account.Credentials.from_service_account_info(
            json.loads(os.environ["LEA_BQ_SERVICE_ACCOUNT"]),
            scopes=[
                "https://www.googleapis.com/auth/drive",
                "https://www.googleapis.com/auth/bigquery",
            ],
        ),
        location=os.environ["LEA_BQ_LOCATION"],
        project_id=os.environ["LEA_BQ_PROJECT_ID"],
        dataset_name=os.environ["LEA_SCHEMA"],
        username=username,
    )

app = make_app(make_client=make_client)

@app.command()
def archive(view_name: str):
    from lea import views

    client = make_client(production=True)

    today = dt.date.today()
    archive_view = lea.views.GenericSQLView(
        schema="",
        name=f"kaya__{view_name}__{today.strftime('%Y_%m_%d')}",
        query=f"SELECT * FROM kaya.{view_name}",
        sqlglot_dialect=client.sqlglot_dialect,
    )
    client.create(archive_view)

Although this works, it would be good to document it properly. Also, maybe passing a make_client factory function is a bit of an awkward interface.