shinichi-takii / ddlparse

DDL parase and Convert to BigQuery JSON schema and DDL statements
https://pypi.org/project/ddlparse/
BSD 3-Clause "New" or "Revised" License
87 stars 29 forks source link

The parser stops when it encounters a NULL column definition #68

Closed sanimesa closed 3 years ago

sanimesa commented 3 years ago

The below SQL produces only one column as output.

Input:

CREATE TABLE sales (
    sale_id int4 NULL,
    posted_time timestamp,
    sale_hdr_title varchar,
    sale_inserted_at timestamp,
    sale_updated_at timestamp,
    sale_title varchar,
    sale_name varchar,
    sale_description text,
    sale_status varchar,
);

Output:

#standardSQL
CREATE TABLE `project.dataset.sales`
(
  sale_id INT64
)
waltervos commented 3 years ago

I can confirm this issue is bugging me as well. Modifying the "basic" testcase to include a NULL column throws an exception later down the line:

File "c:\git\ddlparse\test\test_ddlparse.py", line 1263, in test_parse assert table.columns.to_bigquery_fields() == "[{}]".format(",".join(data["bq_field"]))

I'm still trying to get my head around pyparsing, but it seems the issue is related to this line (648):

Optional(Regex(r"\b(?:NOT\s+)NULL?\b", re.IGNORECASE))("null")

This appears to only match NOT NULL, and not NULL :)