wintersrd / pipelinewise-tap-mssql

Pipelinewise tap for Microsoft SQL Server
GNU Affero General Public License v3.0
14 stars 48 forks source link

Schema invalid, maxlength: -1 #5

Closed visch closed 2 years ago

visch commented 3 years ago

Using this DDL and DML, the tap will generate the schema below that has a maxlength of -1.

CREATE TABLE abc.dbo.test (
    id varchar(450) ,
    name varchar(MAX)
)

INSERT INTO test
(id, name)
VALUES('1', 'abc');
INFO Server Parameters: version: Microsoft SQL Server 2019 (RTM-CU9) (KB5000642) - 15.0.4102.2 (X64)
        Jan 25 2021 20:16:12
        Copyright (C) 2019 Microsoft Corporation
        Developer Edition (64-bit) on Linux (Ubuntu 16.04.7 LTS) <X64>, lock_timeout: -1,
INFO Beginning sync
INFO Preparing Catalog
INFO Fetching tables
INFO Tables fetched, fetching columns
INFO Columns Fetched
INFO Catalog ready
INFO selected-by-default: False
INFO database-name: dbo
INFO is-view: False
INFO table-key-properties: []
INFO selected: True
INFO replication-method: FULL_TABLE
INFO Need to sync test
{"type": "STATE", "value": {"currently_syncing": "dbo-test"}}
INFO Table test proposes FULL_TABLE sync
INFO Table test will use FULL_TABLE sync
INFO syncing test full table
{"type": "SCHEMA", "stream": "dbo-test", "schema": {"properties": {"id": {"inclusion": "available", "maxLength": 450, "type": ["null", "string"]}, "name": {"inclusion": "available", "maxLength": -1, "type": ["null", "string"]}}, "type": "object"}, "key_properties": []}
visch commented 3 years ago

This is an issue because Data when getting validating by JSON schema fails validation

2021-08-04 09:02:50,000 INFO: stderr from second_pipe_cmd: b"jsonschema.exceptions.ValidationError: '*ABCDATA' is too long\r\n" 2021-08-04 09:02:50,000 INFO: stderr from second_pipe_cmd: b'\r\n' 2021-08-04 09:02:50,000 INFO: stderr from second_pipe_cmd: b"Failed validating 'maxLength' in schema['properties']['busnumber']:\r\n" 2021-08-04 09:02:50,000 INFO: stderr from second_pipe_cmd: b" {'inclusion': 'available', 'maxLength': -1, 'type': ['null', 'string']}\r\n" 2021-08-04 09:02:50,000 INFO: stderr from second_pipe_cmd: b'\r\n'