tobymao / sqlglot

Python SQL Parser and Transpiler
https://sqlglot.com/
MIT License
6.76k stars 713 forks source link

Incorect tranformation of create table using Starrocks #3997

Closed hellozepp closed 2 months ago

hellozepp commented 2 months ago

Before you file an issue sqlglot.transpile(sql, read="starrocks", write="hive") # or write="spark"

Fully reproducible code snippet

CREATE TABLE if not exists `sample_table` (
    `tenantid` varchar(1048576) NULL COMMENT "",
    `create_day` date NOT NULL COMMENT "",
    `shopsite` varchar(65533) NOT NULL COMMENT "shopsite",
    `id` varchar(65533) NOT NULL COMMENT "shopsite id",
    `price` decimal128(38, 10) NULL COMMENT "test the bigdecimal",
    `seq` int(11) NULL COMMENT "order",
    `use_status` smallint(6) NULL COMMENT "0,1",
    `created_user` bigint(20) NULL COMMENT "create user",
    `created_time` datetime NULL COMMENT "create time",
) ENGINE=OLAP
DUPLICATE KEY(tenantid)
PRIMARY KEY(`tenantid`, `shopsite`, `id`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`tenantid`, `shopsite`, `id`) BUCKETS 10
ORDER BY (`tenantid`, `shopsite`, `id`)
PROPERTIES (
    "replication_num" = "1",
    "in_memory" = "false",
    "enable_persistent_index" = "false",
    "replicated_storage" = "false",
    "storage_medium" = "HDD",
    "compression" = "LZ4"
)

The following create table statement of starrocks will report an error during parse_one:

Traceback (most recent call last):
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/parts/trans.py", line 74, in <module>
    raise ex
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/parts/trans.py", line 67, in <module>
    test_transpile_sql(os.path.join(dir, '../' + sys.argv[0]), read='starrocks', dialect='hive', pretty=True)
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/parts/trans.py", line 29, in test_transpile_sql
    raise ex
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/parts/trans.py", line 25, in test_transpile_sql
    transpiled_sql = sqlglot.transpile(sql, read=read, write=dialect)[0].strip()
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/__init__.py", line 174, in transpile
    for expression in parse(sql, read, error_level=error_level)
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/__init__.py", line 99, in parse
    return Dialect.get_or_raise(read or dialect).parse(sql, **opts)
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/dialects/dialect.py", line 894, in parse
    return self.parser(**opts).parse(self.tokenize(sql), sql)
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 1366, in parse
    return self._parse(
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 1435, in _parse
    expressions.append(parse_method(self))
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 1667, in _parse_statement
    return self.STATEMENT_PARSERS[self._prev.token_type](self)
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 758, in <lambda>
    TokenType.CREATE: lambda self: self._parse_create(),
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 1820, in _parse_create
    this = self._parse_schema(this=table_parts)
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 5225, in _parse_schema
    self._match_r_paren()
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 6956, in _match_r_paren
    self.raise_error("Expecting )")
  File "/Users/zhanglin/PycharmProjects/sqlglot_master/sqlglot/parser.py", line 1479, in raise_error
    raise error
sqlglot.errors.ParseError: Expecting ). Line 6, Col: 19.
  3) NOT NULL COMMENT "shopsite",
    `id` varchar(65533) NOT NULL COMMENT "shopsite id",
    `price` decimal128(38, 10) NULL COMMENT "test the bigdecimal",
    `seq` int(11) NULL COMMENT "order",
    `use_status
georgesittas commented 2 months ago

This is low priority for us right now, so I'll go ahead and close the ticket. Regardless, feel free to work on it. Make sure to check out the comments I left in #3998.