[Bug] User unable to see the "dbt docs" menu item in Astro

tatiana commented 3 months ago

Astronomer Cosmos Version

Other Astronomer Cosmos version (please specify below)

If "Other Astronomer Cosmos version" selected, which one?

1.5.1

dbt-core version

1.7.2

Versions of dbt adapters

dbt-snowflake==1.7.3 apache-airflow-providers-snowflake==5.1.1 apache-airflow-providers-microsoft-azure==8.2.0 apache-airflow-providers-sendgrid==3.5.0 requests-futures==1.0.1 requests-toolbelt==1.0.0 polars[xlsx2csv]==0.18.15 azure-storage-blob==12.17.0 simple-salesforce==1.12.5 beautifulsoup4==4.12.2

LoadMode

AUTOMATIC

ExecutionMode

LOCAL

InvocationMode

None

airflow version

apache-airflow==2.9.1 (astro-runtime:11.3.00

Operating System

Any

If a you think it's an UI issue, what browsers are you seeing the problem on?

Chrome

Deployment

Astronomer

Deployment details

No response

What happened?

The user has successfully created the dbt docs and stored it in Azure Blob storage by using DbtDocsAzureStorageOperator.

They successfully configured the Cosmos plugin to render the docs, by using:

AIRFLOW__COSMOS__DBT_DOCS_DIR=wasb://user-bucket/dbt-docs
AIRFLOW__COSMOS__DBT_DOCS_CONN_ID=azure_conn

They are not able to see the menu item in the Airflow UI:

However, if they access the URL directly, they can see the dbt docs rendered in Airflow: http://airflow-host/cosmos/dbt_docs

Relevant log output

Apparently, there are no errors in the webserver logs. The only issue is that the menu item is not showing up.

How to reproduce

We have yet to reproduce this problem. We're currently suspecting this is a permission issue.

Anything else :)?

Internal slack thread where the problem was discussed: https://astronomer.slack.com/archives/C04EXKRQFD3/p1721637533315019

We'll likely have to add set additional level of permission in https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/plugin/__init__.py

The specific user seems to have workspace operator permissions in Astro, which should have "Use custom plugins from the Airflow UI menu": https://www.astronomer.io/docs/astro/user-permissions

The source of truth of perms is this file https://github.com/astronomer/astro/blob/fb0acfe9a0524aaceab61169fd7486862ddbe6e6/apps/authz/src/types/rolePermissions.ts#L134

We probably care about “deployment.airflow.customMenu.get” perm

Other references:

https://www.astronomer.io/docs/learn/using-airflow-plugins#appbuilder-menu-items
https://astronomer.slack.com/archives/C04JG8563HR/p1712242691378909 (a 216 thread about a similar issue in the past, with another provider)

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Contact Details

No response

tatiana commented 3 months ago

Something we could look into is how we define the custom menu item, to follow what is recommended in the docs: https://www.astronomer.io/docs/learn/using-airflow-plugins#appbuilder-menu-items

from airflow.plugins_manager import AirflowPlugin

# creating a new top-level menu item
appbuilder_mitem_toplevel = {
    "name": "Apache",
    "href": "https://www.apache.org/",
}

# creating a new sub-item in the Docs menu item
appbuilder_mitem_subitem = {
    "name": "Astro SDK Docs",
    "href": "https://astro-sdk-python.readthedocs.io/en/stable/index.html",
    "category": "Docs",
}

# defining the plugin class
class MyMenuItemsPlugin(AirflowPlugin):
    name = "Menu items plugin"

    # adding the menu items to the plugin
    appbuilder_menu_items = [appbuilder_mitem_toplevel, appbuilder_mitem_subitem]

tatiana commented 3 months ago

I just spoke to Vishnu U, and he was able to reproduce the problem in Astro 🎉

when he is an Organization Owner, he can see the dbt docs menu
when he is an Organization Member and Workspace Operator, he doesn't see the menu item

tatiana commented 2 months ago

We identified that the issue is indeed user permissions, and we'll try to fix this in 1.7.1.

tatiana commented 1 month ago

Suggestion from Alex Liotta, who experienced a similar problem in the past: https://astronomer.slack.com/archives/C04EXKRQFD3/p1722435658739929?thread_ts=1721637533.315019&cid=C04EXKRQFD3

Org owner gets the permission ‘deployment.airflow.admin’, via its inheritance of “workspaceOwnerActions”. This perm bypasses all auth checks on the airflow side hence why the user can see the menu. For Operator to work you will need to set the dag file up to respect the attached image from this docs https://www.astronomer.io/docs/learn/using-airflow-plugins#flask-blueprints-and-appbuilder-views

jessicaschueler commented 4 weeks ago

@tatiana Do we have an ETA for this fix?

tatiana commented 2 weeks ago

@jessicaschueler I've attempted to fix the issue (#1280) following Astro's docs, but it still needs validation in Astro Cloud.

If you want to help with this, please try out this alpha release: https://pypi.org/project/astronomer-cosmos/1.7.1a3/

I strongly suggest we validate this before sharing it with customers.

tatiana commented 2 weeks ago

I validated it in Astro Cloud, and the issue seems to be solved. However, I would appreciate a second pair of eyes to confirm before we release 1.71.

tatiana commented 1 week ago

I confirmed in a fresh deployment _tati_cosmos_test in a workspace ("OSS Team Integration Test Workspace") that I have Workspace Operator permissions (astronomer-stage environment):

This is with Cosmos 1.7:

This is with Cosmos 1.7.1a3:

astronomer / astronomer-cosmos