snowplow-archive / schema-guru

JSONs -> JSON Schema
http://snowplowanalytics.com
150 stars 20 forks source link

Fix handling of new major schema version #159

Closed yalisassoon closed 8 years ago

yalisassoon commented 8 years ago

I just tried using schema guru to generate a new SQL table definition for a new schema version 2-0-0.

The expected behaviour:

Instead it looks like the _1 version were updated with the fields in 2-0-0.

Steps to reproduce:

I generated a 1-0-0 schemas/com.example_company/example_event/1-0-0:

{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "description": "Schema for an example event",
    "self": {
        "vendor": "com.example_company",
        "name": "example_event",
        "format": "jsonschema",
        "version": "1-0-0"
    },

    "type": "object",
    "properties": {
        "exampleStringField": {
            "type": "string",
            "maxLength": 255
        },
        "exampleIntegerField": {
            "type": "integer"
        },
        "exampleNumericField": {
            "type": "number",
            "maxDecimal": 3
        },
        "exampleTimestampField": {
            "type": "string",
            "format": "date-time"
        }
    },
    "minProperties":1,
    "required": ["exampleStringField", "exampleIntegerField"],
    "additionalProperties": false
}

Then a new version 2-0-0 schema/com.example_company/example_event/2-0-0:

{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "description": "Schema for an example event",
    "self": {
        "vendor": "com.example_company",
        "name": "example_event",
        "format": "jsonschema",
        "version": "1-0-0"
    },

    "type": "object",
    "properties": {
        "exampleStringField": {
            "type": "string",
            "maxLength": 255
        },
        "exampleBigIntField": {
            "type": "integer",
            "maximum": 9223372036854775807,
            "minimum": -9223372036854775808
        },
        "exampleSmallIntField": {
            "type": "integer",
            "maximum": 32767,
            "minimum": -32768
        },
        "exampleNumericField": {
            "type": "number",
            "maxDecimal": 3
        },
        "exampleNumericField2": {
            "type": "number",
            "maxDecimal": 12
        },
        "exampleTimestampField": {
            "type": "string",
            "format": "date-time"
        }
    },
    "minProperties":1,
    "required": ["exampleStringField", "exampleIntegerField"],
    "additionalProperties": false
}

Ran schema guru at the command line:

$ ../../schema-guru-0.6.1 ddl --with-json-paths schemas/com.example_company/example_event/jsonschema --force
File [/Users/yalisassoon/Development/event-dictionaries/test/example-schema-registry/./sql/com.example_company/example_event_1.sql] was overriden successfully!
File [/Users/yalisassoon/Development/event-dictionaries/test/example-schema-registry/./jsonpaths/com.example_company/example_event_1.json] was overriden successfully!

Inspecting the table definitions:

$ ls sql/com.example_company
example_event_1.sql
$ cat sql/com.example_company/example_event_1.sql
-- AUTO-GENERATED BY schema-ddl DO NOT EDIT
-- Generator: schema-ddl 0.3.0
-- Generated: 2016-05-05 11:40

CREATE SCHEMA IF NOT EXISTS atomic;

CREATE TABLE IF NOT EXISTS atomic.com_example_company_example_event_1 (
    "schema_vendor"           VARCHAR(128)     ENCODE RUNLENGTH NOT NULL,
    "schema_name"             VARCHAR(128)     ENCODE RUNLENGTH NOT NULL,
    "schema_format"           VARCHAR(128)     ENCODE RUNLENGTH NOT NULL,
    "schema_version"          VARCHAR(128)     ENCODE RUNLENGTH NOT NULL,
    "root_id"                 CHAR(36)         ENCODE RAW       NOT NULL,
    "root_tstamp"             TIMESTAMP        ENCODE LZO       NOT NULL,
    "ref_root"                VARCHAR(255)     ENCODE RUNLENGTH NOT NULL,
    "ref_tree"                VARCHAR(1500)    ENCODE RUNLENGTH NOT NULL,
    "ref_parent"              VARCHAR(255)     ENCODE RUNLENGTH NOT NULL,
    "example_string_field"    VARCHAR(255)     ENCODE LZO       NOT NULL,
    "example_big_int_field"   BIGINT           ENCODE LZO,
    "example_numeric_field"   DOUBLE PRECISION ENCODE RAW,
    "example_numeric_field2"  DOUBLE PRECISION ENCODE RAW,
    "example_small_int_field" SMALLINT         ENCODE LZO,
    "example_timestamp_field" TIMESTAMP        ENCODE LZO,
    FOREIGN KEY (root_id) REFERENCES atomic.events(event_id)
)
DISTSTYLE KEY
DISTKEY (root_id)
SORTKEY (root_tstamp);

COMMENT ON TABLE com_example_company_example_event_1 IS 'iglu:com.example_company/example_event/jsonschema/1-0-0';

Notes:

  1. the table fields are as per schema version 2-0-0
  2. the table name and filename both suggest major version 1
  3. the comment on table also suggests version 1
yalisassoon commented 8 years ago

My bad - I had the schema version wrong in the file. Closing - sorry!