Global Aysnc queries doesn't work with Athena

Abhijith-Nagaraja commented 1 year ago

A clear and concise description of what the bug is.

Tried to enable Global async queries. Was successfully able to start it with mysql RDS but aync_event API didn't even get fired from dashboard with Athena datasource

Here are my configurations

.
.
.
FEATURE_FLAGS: Dict[str, bool] = {
  "ENABLE_TEMPLATE_PROCESSING": True,
  "ENABLE_TEMPLATE_REMOVE_FILTERS": True,
  "DASHBOARD_CACHE": True,
  "ALERT_REPORTS": True,
  "ALLOW_ADHOC_SUBQUERY": True,
  "GLOBAL_ASYNC_QUERIES": True
}

#Caching configurations
FILTER_STATE_CACHE_CONFIG = {
    "CACHE_TYPE": "RedisCache",
    "CACHE_DEFAULT_TIMEOUT": 86400,
    "CACHE_KEY_PREFIX": "superset_filter_cache",
    "CACHE_REDIS_URL": "redis://superset_cache:6379/0"
}
EXPLORE_FORM_DATA_CACHE_CONFIG = {
    "CACHE_TYPE": "RedisCache",
    "CACHE_DEFAULT_TIMEOUT": 86400,
    "CACHE_KEY_PREFIX": "superset_form_data_cache",
    "CACHE_REDIS_URL": "redis://superset_cache:6379/2"
}
DATA_CACHE_CONFIG = {
    "CACHE_TYPE": "SupersetMetastoreCache",
    "CACHE_KEY_PREFIX": "superset_data_cache",
    "CACHE_DEFAULT_TIMEOUT": 86400
}
CACHE_CONFIG = {
    "CACHE_TYPE": "RedisCache",
    "CACHE_DEFAULT_TIMEOUT": 86400,
    "CACHE_KEY_PREFIX": "superset_filter_cache",
    "CACHE_REDIS_URL": "redis://superset_cache:6379/1"
}

#GLOBAL ASYNC QUERIES CONFIG

GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = {
    "port": 6379,
    "host": "superset_cache",
    "db": 3,
    "ssl": False
}

GLOBAL_ASYNC_QUERIES_JWT_SECRET = "<FIX IT>"
GLOBAL_ASYNC_QUERIES_POLLING_DELAY = int(
    timedelta(milliseconds=2000).total_seconds() * 1000
)

class CeleryConfig(object):
    broker_url = "redis://superset_cache:6379/4"
    imports = (
        "superset.sql_lab",
        "superset.tasks.scheduler",
    )
    result_backend = "redis://superset_cache:6379/4"
    worker_prefetch_multiplier = 10
    task_acks_late = True
    task_annotations = {
        "sql_lab.get_sql_results": {
            "rate_limit": "100/s",
        },
    }

CELERY_CONFIG = CeleryConfig

docker ps

Async configuration for datasource in UI

Screenshot 2023-11-01 at 1 01 04 PM

How to reproduce the bug

Step 1: Create two dashboards. One dashboard containing SQL datasource such as mysql and second one with Athena Data source. Step 2: Enable Global Async Queries and corresponding setup Step 3: Configure Celery and Redis Step 4: Enable Asynchronous query execution from Database and Connections -> select datasource -> Edit -> Advanced -> check Asynchronous query execution

Expected results

Charts within the dashboard should be loaded Asynchronously for both Mysql Rds datasource and Athena datasource

Actual results

Charts within the dashboard loaded Asynchronously for Mysql Rds datasource but not for Athena datasource

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

(please complete the following information):

browser type and version: Chrome - 118.0.5993.117
superset version: 3.0
python library versions:
```
pyathena == 2.5.4
pyathenajdbc == 3.0.1
```
any feature flags active: "GLOBAL_ASYNC_QUERIES": True

Checklist

Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

Additional screenshots

Working with Mysql RDS datasource. As you can see in the below screenshot, async_event api got fired Screenshot 2023-11-01 at 1 02 59 PM

Not working with Athena datasource. As you can see in the below screenshot, no async_even api is fired Screenshot 2023-11-01 at 1 08 35 PM

rusackas commented 8 months ago

Hi there! Async queries are still a bit "beta" even though there have been some improvements since this issue was filed. I'm curious if you're still facing this in Superset 3.1 or newer. Also curious if anyone else running GAQ (maybe @villebro ) have the ability to reproduce the issue as well.

villebro commented 6 months ago

Is there a reason the DATA_CACHE_CONFIG is using SupersetMetastoreCache, and not RedisCache like the other caches in the config?

apache / superset