Unable to use Multi Stage Engine for Apache Pinot on Superset

[piby180]

I would like to enable multi stage engine for Apache Pinot on Superset. As per PinotDB docs, it is possible to enable multi stage engine by providing connect args in create_engine method of SQLAlchemy

from sqlalchemy.engine import create_engine

 engine = create_engine(
     "pinot://localhost:8000/query/sql?controller=http://localhost:9000/",     
      connect_args={"useMultistageEngine": "true"}
 )

I have tried adding connect_args to Engine Parameters in Superset UI but multistage engine is still not enabled.

Expected results

Multi stage queries which run fine by using pinotdb in python directly should run in Superset as well

Actual results

Multi stage queries does not run in Superset

Screenshots

Here is one example!

DISTINCT * syntax runs fine in Python with multistage enabled. It also runs fine in Pinot UI Query Console. It does not work in Superset.

Environment

superset version: 3.0.0 pinotdb==5.1.0 psycopg2-binary==2.9.9 pinot version: release-1.0.0

Additional context

I have deployed Superset on EKS using Helm chart. Same issue raised on Superset https://github.com/apache/superset/issues/25627

[davizucon]

Hello, can you try adding OPTION(useMultistageEngine=true) at end of your query? or {"connect_args":{"use_multistage_engine":"true"}} check it out : https://github.com/python-pinot-dbapi/pinot-dbapi/blob/master/pinotdb/sqlalchemy.py#L200

[piby180]

I have managed to make it work with {"connect_args":{"use_multistage_engine":"true"}} . Forgot to update it here.

I was trying to use useMultistageEngine earlier as mentioned here https://github.com/python-pinot-dbapi/pinot-dbapi/tree/master#using-sqlalchemy