Closed atul-r closed 2 weeks ago
Maybe we can create a PR to address this in SQLAlchemy. I found that the PostgreSQL dialect lowers the string before getting the schema. https://github.com/sqlalchemy/sqlalchemy/blob/6cf5e2a188fc5e337d22a098a5fe9a9fe10cc7e7/lib/sqlalchemy/dialects/postgresql/base.py#L3756
I'm also considering if we can provide a way to display the data type in lowercase.
Maybe we can create a PR to address this in SQLAlchemy. I found that the PostgreSQL dialect lowers the string before getting the schema. https://github.com/sqlalchemy/sqlalchemy/blob/6cf5e2a188fc5e337d22a098a5fe9a9fe10cc7e7/lib/sqlalchemy/dialects/postgresql/base.py#L3756
I'm also considering if we can provide a way to display the data type in lowercase.
@evenyag But there are non matching data types as well. For example STRING
, there is no data type of STRING in MySQL. possibly VARCHAR
. This needs to come from greptime
Maybe we can create a PR to address this in SQLAlchemy. I found that the PostgreSQL dialect lowers the string before getting the schema. https://github.com/sqlalchemy/sqlalchemy/blob/6cf5e2a188fc5e337d22a098a5fe9a9fe10cc7e7/lib/sqlalchemy/dialects/postgresql/base.py#L3756 I'm also considering if we can provide a way to display the data type in lowercase.
But there are non matching data types as well. For example
STRING
, there is no data type of STRING in MySQL. possiblyVARCHAR
. This needs to come from greptime
Since PostgreSQL does not support SHOW
statements, we can format the output to match MySQL
Can we add a new method display_mysql_dialect()
to the CreateTable
and Column
?
https://github.com/GreptimeTeam/greptimedb/blob/27d9aa0f3be492e4f6e420876193b29f937f6157/src/sql/src/statements/create.rs#L235-L246
We need to map some data types to MySQL's types in Column
.
https://github.com/GreptimeTeam/greptimedb/blob/27d9aa0f3be492e4f6e420876193b29f937f6157/src/sql/src/statements/create.rs#L135-L148
Another way is to map the data type while creating the Column
struct.
https://github.com/GreptimeTeam/greptimedb/blob/27d9aa0f3be492e4f6e420876193b29f937f6157/src/query/src/sql/show_create_table.rs#L116-L119
@atul-r I'm not quite familiar with SQLAlchemy, I wonder if this inspect function a must-have to use SQLAlchemy with GreptimeDB? Is this error blocking you from running SQL query in SQLAlchemy?
@atul-r I'm not quite familiar with SQLAlchemy, I wonder if this inspect function a must-have to use SQLAlchemy with GreptimeDB? Is this error blocking you from running SQL query in SQLAlchemy?
@sunng87 I am using apache superset to create charts from the data in greptime. Superset uses SQLAlchemy to connect to greptime. When connecting to any database, superset uses the inspect call to get the table details. This is where I am getting the issue. So it is a blocker
@atul-r Thank you for the information. That's a blocking issue for Superset.
@atul-r Thank you for the information. That's a blocking issue for Superset.
The MySQL endpoint of greptime should send responses that is inline with MySQL. This needs to be handled in greptime. If not any MySQL client will face issues with greptime
There is another issue that I have created as well (https://github.com/GreptimeTeam/greptimedb/issues/4542)
Yes. That's we need to address in GreptimeDB for support of Superset.
For #4542, most date/time functions should have an alternative in datafusion, which is compatible with PostgreSQL. https://datafusion.apache.org/user-guide/sql/scalar_functions.html#time-and-date-functions
Will it be easier to support PostgreSQL's driver than MySQL? We are working on https://github.com/GreptimeTeam/greptimedb/issues/3560
What do you think? @sunng87
@evenyag sounds good. I will give Superset's postgres datasource a try to see the gap.
I tried postgresql data source. It seems Superset doesn't offer configuration for psycopg's autocommit
option so the postgres client will send statements like BEGIN
which is not supported by us.
After a deep dive into postgres superset connector, even with #3560 and #4553 , there are still some functions like pg_catalog.pg_get_serial_sequence
, json_build_object
and etc. to be supported. We don't see this in near future.
Perhaps we need to build our own plugin or connector for superset, as well as metabase.
@atul-r I just created a greptimedb connector for sqlalchemy and superset. You can add greptimedb-sqlalchemy
to docker/requirements-local.txt
(if you are using docker-compose) or use pip install greptimedb-sqlalchemy
if you installed superset in your system.
There will be a GreptimeDB option when you are adding database connection.
This is an early release of the greptimedb native connector. Let me know if you find any issue with it.
@atul-r I just created a greptimedb connector for sqlalchemy and superset. You can add
greptimedb-sqlalchemy
todocker/requirements-local.txt
(if you are using docker-compose) or usepip install greptimedb-sqlalchemy
if you installed superset in your system.There will be a GreptimeDB option when you are adding database connection.
This is an early release of the greptimedb native connector. Let me know if you find any issue with it.
@sunng87 I am getting the below error
ERROR: (builtins.NoneType) None
[SQL: Failed to plan SQL: Error during planning: Invalid function 'pg_catalog.version'.
Did you mean 'APPROX_MEDIAN'?
]
(Background on this error at: https://sqlalche.me/e/14/dbapi)
My connection string is greptimedb://root:greptime_pwd@host.docker.internal:4003/supersetexample
Created a detailed issue here
This has been resolved by our superset driver, see https://github.com/sunng87/greptimedb-sqlalchemy/issues/1 and our docs https://docs.greptime.com/nightly/user-guide/integrations/superset
What type of bug is this?
Incorrect result
What subsystems are affected?
Standalone mode
Minimal reproduce step
I am encountering an issue when using SQLAlchemy's inspector to retrieve column details from a Greptime database using the MySQL dialect. Below is the code snippet I am using:
When I run this code, I receive the following error:
TypeError: NullType() takes no arguments
Analysis: During reflection, SQLAlchemy issues a SHOW CREATE TABLE query to retrieve the table schema. It then parses the returned string using regex to extract the table name and column details.
Greptime returns the following CREATE TABLE statement for the table:
The issue seems to stem from the data types being returned in uppercase (e.g., STRING, TIMESTAMP) also some data types are different for example STRING, there is no data type of STRING in mysql. The MySQL dialect in SQLAlchemy does not recognize these uppercase data types and expects them in lowercase. As a result, it raises a TypeError.
I manually changed the data types to lowercase during debugging, and the reflection worked without any issues. This suggests that the data type casing is the root cause of the problem.
Request: Is there a way to modify Greptime so that it returns the data types in lowercase, or provide an output consistent with what the MySQL dialect in SQLAlchemy expects? This would enhance compatibility and prevent errors during schema reflection.
What did you expect to see?
What did you see instead?
TypeError: NullType() takes no arguments
What operating system did you use?
greptime/greptimedb:v0.9.1,
What version of GreptimeDB did you use?
0.9.1
Relevant log output and stack trace