NotImplementedError: Don't know how to literal-quote value datetime.date(2023, 5, 4)

rikirolly commented 1 year ago

Creating a dataset from a Trino table I get this error:

2023-05-09 07:13:20,067:ERROR:flask_appbuilder.api:Don't know how to literal-quote value datetime.date(2023, 5, 4)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/api/__init__.py", line 110, in wraps
    return f(self, *args, **kwargs)
  File "/app/superset/views/base_api.py", line 122, in wraps
    raise ex
  File "/app/superset/views/base_api.py", line 113, in wraps
    duration, response = time_function(f, self, *args, **kwargs)
  File "/app/superset/utils/core.py", line 1586, in time_function
    response = func(*args, **kwargs)
  File "/app/superset/utils/log.py", line 266, in wrapper
    value = f(*args, **kwargs)
  File "/app/superset/databases/api.py", line 685, in table_metadata
    table_info = get_table_metadata(database, table_name, schema_name)
  File "/app/superset/databases/utils.py", line 92, in get_table_metadata
    "selectStar": database.select_star(
  File "/app/superset/models/core.py", line 588, in select_star
    return self.db_engine_spec.select_star(
  File "/app/superset/db_engine_specs/base.py", line 1229, in select_star
    sql = database.compile_sqla_query(qry)
  File "/app/superset/models/core.py", line 568, in compile_sqla_query
    sql = str(qry.compile(engine, compile_kwargs={"literal_binds": True}))
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 494, in compile
    return self._compiler(dialect, **kw)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 558, in _compiler
    return dialect.statement_compiler(dialect, self, **kw)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 776, in __init__
    Compiled.__init__(self, dialect, statement, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 451, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 486, in process
    return obj._compiler_dispatch(self, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
    return meth(self, **kw)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 3405, in visit_select
    text = self._compose_select_body(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 3563, in _compose_select_body
    t = self._generate_delimited_and_list(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 1705, in _generate_delimited_and_list
    return separator.join(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 1705, in <genexpr>
    return separator.join(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 1707, in <genexpr>
    for s in (c._compiler_dispatch(self, **kw) for c in clauses)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
    return meth(self, **kw)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 2202, in visit_binary
    return self._generate_generic_binary(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 2258, in _generate_generic_binary
    + binary.right._compiler_dispatch(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
    return meth(self, **kw)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 2438, in visit_bindparam
    ret = self.render_literal_bindparam(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 2537, in render_literal_bindparam
    return self.render_literal_value(value, bindparam.type)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/compiler.py", line 2554, in render_literal_value
    raise NotImplementedError(
NotImplementedError: Don't know how to literal-quote value datetime.date(2023, 5, 4)

How to reproduce the bug

Probably just creating a dataset starting from a Trino table with a datetime.date column. In my case this column is also a partition.

Expected results

Correctly create the dataset.

Actual results

Superset report the error: Unable to load columns for the selected table. Please select a different table.

Screenshots

Environment

Chrome Version: 111.0.5563.146 (Official Build) (64-bit)
Superset Version: 2.1.0
Superset HelmChart repo: https://apache.github.io/superset
Superset HelmChart repo Version: 0.9.2
Trino Version: 416

Checklist

Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

I have already created different dataset in the same format and always worked correctly, I do not know if this could be a new bug of the latest release.

davinavarro commented 1 year ago

I'm getting the same error with the same stack. Superset + Trino + Datetime partitions

davinavarro commented 1 year ago

Just in case someone googles this bug, I was able to fix it. It is not pretty but it works. The bug is in the file superset/db_engine_specs/presto.py in the definition of the method latest_partition(). In my case, my datetime partition column is named dl_date so when I receive this column I change the type from datetime.date to string and recreate the dataframe.

    def latest_partition(
        cls,
        table_name: str,
        schema: Optional[str],
        database: Database,
        show_first: bool = False,
    ) -> Tuple[List[str], Optional[List[str]]]:
                                 .
                                 .
                                 .
                                 .
        column_names = indexes[0]["column_names"]
        part_fields = [(column_name, True) for column_name in column_names]
        sql = cls._partition_query(table_name, database, 1, part_fields)
        df = database.get_df(sql, schema)

       #========== MY UGLY FIX  ==========#
        if "dl_date" in column_names:
            lat_df = cls._latest_partition_from_df(df)
            dl_date: date = lat_df[2]
            return column_names, [lat_df[0], lat_df[1], dl_date.strftime("%Y-%m-%d")]
        else:
            return column_names, cls._latest_partition_from_df(df)
        #========== MY UGLY FIX  ==========#

rusackas commented 8 months ago

@davinavarro do you think there's a more generalized way to fix this that would work as a PR? Tempted to close this as stale and somewhat-solved, but wondering if there's a better solution here. CC @nytai @john-bodley since it's Trino-tastic.

rusackas commented 2 months ago

Curious if @dosu-bot can provide a suggestion for a permanent/generalized fix, or even a diff for a PR.

apache / superset