TIMESTAMP datatype causing failure in Supserset

[etadelta222]

When trying to add Pinot table to Superset, getting the following error:

Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: ERROR:root:'timestamp' Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: Traceback (most recent call last): Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/flask_appbuilder/api/__init__.py", line 84, in wraps Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return f(self, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/views/base_api.py", line 80, in wraps Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: duration, response = time_function(f, self, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/utils/core.py", line 1368, in time_function Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: response = func(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/utils/log.py", line 224, in wrapper Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: value = f(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/api.py", line 236, in post Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: new_model = CreateDatasetCommand(g.user, item).run() Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/commands/create.py", line 47, in run Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: self.validate() Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/commands/create.py", line 87, in validate Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: if database and not DatasetDAO.validate_table_exists( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/dao.py", line 81, in validate_table_exists Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: database.get_table(table_name, schema=schema) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/models/core.py", line 603, in get_table Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return Table( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "<string>", line 2, in __new__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 139, in warned Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return fn(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 560, in __new__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: metadata._remove_table(name, schema) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: compat.raise_( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: raise exception Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 555, in __new__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: table._init(name, metadata, *args, **kw) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 644, in _init Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: self._autoload( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 667, in _autoload Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: autoload_with.run_callable( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2212, in run_callable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return conn.run_callable(callable_, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1653, in run_callable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return callable_(self, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 469, in reflecttable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return insp.reflecttable( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 664, in reflecttable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: for col_d in self.get_columns( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 390, in get_columns Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: col_defs = self.dialect.get_columns( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 390, in get_columns Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: columns = [ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 393, in <listcomp> Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: "type": get_type(spec["dataType"], spec.get("fieldSize")), Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 458, in get_type Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return type_map[data_type.lower()] Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: KeyError: 'timestamp'

Looks like https://github.com/python-pinot-dbapi/pinot-dbapi/blob/8738d8ff135a4065b8f90ca2cc95b18521210f29/pinotdb/sqlalchemy.py#L450 is missing datetime data type.

Pinot version: 0.8.0 Table Type: Realtime Superset version: 1.1.0

[xiangfu0]

Thanks for reporting, we will support both TIMESTAMP and BOOLEAN from 0.8.0 in the next release.

[etadelta222]

Thank you @xiangfu0 . Is there an anticipated date for the next release?

[etadelta222]

Hi @xiangfu0 , I see that there was a release. Is timestamp supported now? I didn't see it in the sqlalchemy.py file.

[xiangfu0]

This is supported now along with json type.