shinichi-takii / ddlparse

DDL parase and Convert to BigQuery JSON schema and DDL statements
https://pypi.org/project/ddlparse/
BSD 3-Clause "New" or "Revised" License
87 stars 29 forks source link

MySQL CREATE TABLE parsing fails. some columns are missing. #61

Closed sakamossan closed 3 years ago

sakamossan commented 4 years ago

I tried to output a BigQuery column definition from a MySQL table definition, but some columns are missing.

Please show me how to avoid it by crafting CREATE TABLE, or if I used ddlparse incorrectly.

reproduce

environment

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.15.7
BuildVersion:   19H2
$ python -V
Python 3.8.3
$ ll $(python -c "import site; print (site.getsitepackages()[0])")/ | grep ddlparse
drwxr-xr-x    5 sakamossan  staff   160B  9 25 12:49 ddlparse
drwxr-xr-x    8 sakamossan  staff   256B  9 25 12:49 ddlparse-1.8.0.dist-info

execution

I used following files

/tmp/_.sql

CREATE TABLE `t` (
  `id` INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
  `created` DATETIME NULL,
  `modified` DATETIME NULL
) ENGINE = InnoDB;

/tmp/_.py

#!/usr/bin/env python
import sys
import json
from ddlparse import DdlParse

sql = sys.stdin.read()
table = DdlParse().parse(ddl=sql, source_database=DdlParse.DATABASE.mysql)
print(table.to_bigquery_fields())
$ cat /tmp/_.json | /tmp/_.py | jq .
[
  {
    "name": "id",
    "type": "INTEGER",
    "mode": "REQUIRED"
  },
  {
    "name": "created",
    "type": "DATETIME",
    "mode": "NULLABLE"
  }
]

modified column is not output.

waltervos commented 3 years ago

This is related to issue #68, which I've attempted to fix in pull request #70.