littleK0i / SnowDDL

Declarative database change management tool for Snowflake
Apache License 2.0
105 stars 28 forks source link

colon character unsupported in columns of table definition #129

Closed Xiaoming-lw closed 1 month ago

Xiaoming-lw commented 1 month ago

Describe the bug

Snowddl runs error while reading the config yaml files for a table column containing character ":".

snowddl -a *** -r *** -u *** --authenticator=externalbrowser --include-object-types=database,schema,table -c cdb_config_5 plan
2024-10-09 14:58:59.024 - WARNING - [/home/simon/src/snowddl/cdb_config_5/demo_db/run_roman_test/table/test.yaml]: Traceback (most recent call last):
  File "/home/simon/src/.venv/lib/python3.10/site-packages/snowddl/parser/abc_parser.py", line 48, in parse_schema_object_files
    callback(file)
  File "/home/simon/src/.venv/lib/python3.10/site-packages/snowddl/parser/table.py", line 204, in process_table
    name=Ident(col_name),
  File "/home/simon/src/.venv/lib/python3.10/site-packages/snowddl/blueprint/ident.py", line 75, in __init__
    self.name = self._validate_part(name)
  File "/home/simon/src/.venv/lib/python3.10/site-packages/snowddl/blueprint/ident.py", line 47, in _validate_part
    raise ValueError(
ValueError: Character [:] in not allowed in identifier [event_json:useridentity], only ASCII letters, digits and single underscores are accepted

Create table:

create schema DEMO_DB.RUN_ROMAN_TEST;

create or replace TABLE DEMO_DB.RUN_ROMAN_TEST.TEST (
    "EVENT_JSON:USERIDENTITY" VARIANT
);

Convert table into yaml:

snowddl-convert -a *** -r *** -u *** --authenticator=externalbrowser --include-object-types=database,schema,table --include-databases=DEMO_DB --ignore-ownership -c cdb_config_5
columns:
  event_json:useridentity:
    type: VARIANT
littleK0i commented 1 month ago

This is an intentional guardrail to prevent creation of poorly named entities.

In this specific case allowing "colon" character would potentially confuse users.

EVENT_JSON:USERIDENTITY    -- this is valid Snowflake syntax to access field in VARIANT column
"EVENT_JSON:USERIDENTITY"  -- this is a column name, looks similar, but nothing to do with Snowflake syntax

Also, it look weird in YAML config: event_json:useridentity:

Consider using one underscore, two underscores or dollar-sign instead of colon for better naming.

EVENT_JSON_USERIDENTITY
EVENT_JSON__USERIDENTITY
EVENT_JSON$USERIDENTITY

Column name of this kind will be accessible without double-quotes.

Xiaoming-lw commented 1 month ago

Thank you @littleK0i , it really bring some confusion. I thought it is related with some snowflake feature, but not. Maybe someone just copied it from another query statement and didn't change the name.