moj-analytical-services / etl_manager

A python package to create a database on the platform using our moj data warehousing framework
21 stars 9 forks source link

Validation is incorrectly failing STRUCT and ARRAY columns #119

Closed isichei closed 4 years ago

isichei commented 4 years ago
from etl_manager.meta import TableMeta

tab = TableMeta(name="test", location="blah", data_format = "json")
tab.add_column(name="col1", type="struct<k:character>", description="blah")

Raises an error:

ValueError: The data_type provided must match the supported data_type names: character, int, long, float, double, date, datetime, boolean, struct, array

Check is probably just looking for struct or array specifically rather than doing a correct test.

Also I think the regex check is failing for columns that should be fine. Using this as an example. Should add to a unit test.

RobinL commented 4 years ago

I just ran into this too!