search

AutoIDM / tap-clickup

tap-clickup , singer compliant tap for pulling clickup data
MIT License
12 stars 19 forks source link

Flattening Breaks Schema #127

Closed balmasi closed 2 years ago

balmasi commented 2 years ago

I have some middleware that augments a tap's schema messages, and it's failing for this tap whenever flattening is enabled due to a malformed schema message.

When flattening is enabled, instead of this tap outputting

{"type": "SCHEMA", "stream": "task", "schema": {"type": "object", "properties": [SOME_JSON_SCHEMA]}}

It's outputting

{"type": "SCHEMA", "stream": "task", "schema": [SOME_JSON_SCHEMA]}

When flattening is disabled, it works as expected

This can also lead to issues with targets not being able to transform data types

visch commented 2 years ago

Can you offer some way to replicate this?

balmasi commented 2 years ago

@visch you can enable flattening (1 level is sufficient) and look at the schema message generated.

The task stream is a good candidate due to its nested properties.

Is there anything required beyond that?

visch commented 2 years ago

Took a quick look and verified your issue. Here's replication steps

  1. Set a config value of 
    {
"api_token": "xxx",
"flattening_enabled": true,
"flattening_max_depth": 1
}
  2. Run something like meltano elt tap-clickup target-jsonl You'll get a failure like
Traceback (most recent call last):
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/bin/target-jsonl", line 8, in <module>
    sys.exit(main())
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 92, in main
    state = persist_messages(
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 72, in persist_messages
    validators[stream] = Draft4Validator((o['schema']))
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 74, in __init__
    resolver = RefResolver.from_schema(schema)
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 318, in from_schema
    return cls(schema.get(u"id", u""), schema, *args, **kwargs)
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 296, in __init__
    self.store[base_uri] = referrer
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/_utils.py", line 26, in __setitem__
    self.store[self.normalize(uri)] = value
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/_utils.py", line 16, in normalize
    return urlsplit(uri).geturl()
  File "/home/visch/git/tap-clickup/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/compat.py", line 40, in urlsplit
    scheme, netloc, path, query, fragment = _urlsplit(url)
  File "/home/visch/.pyenv/versions/3.8.12/lib/python3.8/urllib/parse.py", line 431, in urlsplit
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/home/visch/.pyenv/versions/3.8.12/lib/python3.8/urllib/parse.py", line 127, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/home/visch/.pyenv/versions/3.8.12/lib/python3.8/urllib/parse.py", line 111, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/home/visch/.pyenv/versions/3.8.12/lib/python3.8/urllib/parse.py", line 111, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'dict' object has no attribute 'decode'
  1. Issue is with the SCHEMA as the values are not wrapped in properties

Schema without flattening for Tasks:

{"type": "SCHEMA", "stream": "task", "schema": {"properties": {"id": {"type": ["string"]}, "custom_id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "priority": {"properties": {"color": {"type": ["string"]}, "id": {"type": ["string"]}, "orderindex": {"type": ["string", "null", "integer"]}, "priority": {"type": ["string"]}}, "type": ["object", "null"]}, "time_spent": {"type": ["null", "number"]}, "text_content": {"type": ["null", "string"]}, "description": {"type": ["null", "string"]}, "status": {"properties": {"status": {"type": ["string", "null"]}, "color": {"type": ["string", "null"]}, "type": {"type": ["string", "null"]}, "orderindex": {"type": ["string", "integer", "null"]}}, "type": "object"}, "orderindex": {"type": ["string", "integer", "null"]}, "date_created": {"type": ["string", "null"]}, "date_updated": {"type": ["string", "null"]}, "date_closed": {"type": ["string", "null"]}, "archived": {"type": ["boolean", "null"]}, "creator": {"properties": {"id": {"type": ["integer", "null"]}, "username": {"type": ["string", "null"]}, "color": {"type": ["string", "null"]}, "email": {"type": ["string", "null"]}, "profilePicture": {"type": ["string", "null"]}}, "type": "object"}, "assignees": {"items": {"properties": {"id": {"type": ["integer", "null"]}, "username": {"type": ["string", "null"]}, "color": {"type": ["string", "null"]}, "initials": {"type": ["string", "null"]}, "email": {"type": ["string", "null"]}, "profilePicture": {"type": ["string", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "watchers": {"type": ["array", "null"]}, "checklists": {"type": ["array", "null"]}, "tags": {"items": {"properties": {"name": {"type": ["string", "null"]}, "tag_fg": {"type": ["string", "null"]}, "tag_bg": {"type": ["string", "null"]}, "creator": {"type": ["integer", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "parent": {"type": ["string", "null"]}, "due_date": {"type": ["null", "string"]}, "start_date": {"type": ["string", "null"]}, "points": {"type": ["string", "number", "null"]}, "time_estimate": {"type": ["integer", "null"]}, "custom_fields": {"items": {"properties": {"id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "type": {"type": ["string", "null"]}, "type_config": {"properties": {"default": {"type": ["integer", "null"]}, "placeholder": {"type": ["string", "null"]}, "options": {"items": {"properties": {"id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "color": {"type": ["string", "null"]}, "orderindex": {"type": ["integer", "string", "null"]}}, "type": "object"}, "type": ["array", "null"]}}, "type": "object"}, "date_created": {"type": ["string", "null"]}, "hide_from_guests": {"type": ["string", "boolean", "null"]}, "required": {"type": ["string", "boolean", "null"]}, "value": {"type": ["integer", "string", "boolean", "null", "array", "object", "number"]}}, "type": "object"}, "type": ["array", "null"]}, "dependencies": {"type": ["array", "null"]}, "linked_tasks": {"type": ["array", "null"]}, "team_id": {"type": ["string", "null"]}, "url": {"type": ["string", "null"]}, "permission_level": {"type": ["string", "null"]}, "list": {"properties": {"id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "access": {"type": ["boolean", "null"]}}, "type": "object"}, "project": {"properties": {"id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "hidden": {"type": ["boolean", "null"]}, "access": {"type": ["boolean", "null"]}}, "type": "object"}, "folder": {"properties": {"id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "hidden": {"type": ["boolean", "null"]}, "access": {"type": ["boolean", "null"]}}, "type": "object"}, "space": {"properties": {"id": {"type": ["string", "null"]}}, "type": "object"}}, "type": "object"}, "key_properties": ["id"], "bookmark_properties": ["date_updated"]}

Schema with flattening

{"type": "SCHEMA", "stream": "task", "schema": {"id": {"type": ["string"]}, "custom_id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "priority__color": {"type": ["string"]}, "priority__id": {"type": ["string"]}, "priority__orderindex": {"type": ["string", "null", "integer"]}, "priority__priority": {"type": ["string"]}, "time_spent": {"type": ["null", "number"]}, "text_content": {"type": ["null", "string"]}, "description": {"type": ["null", "string"]}, "status__status": {"type": ["string", "null"]}, "status__color": {"type": ["string", "null"]}, "status__type": {"type": ["string", "null"]}, "status__orderindex": {"type": ["string", "integer", "null"]}, "orderindex": {"type": ["string", "integer", "null"]}, "date_created": {"type": ["string", "null"]}, "date_updated": {"type": ["string", "null"]}, "date_closed": {"type": ["string", "null"]}, "archived": {"type": ["boolean", "null"]}, "creator__id": {"type": ["integer", "null"]}, "creator__username": {"type": ["string", "null"]}, "creator__color": {"type": ["string", "null"]}, "creator__email": {"type": ["string", "null"]}, "creator__profilePicture": {"type": ["string", "null"]}, "assignees": {"items": {"properties": {"id": {"type": ["integer", "null"]}, "username": {"type": ["string", "null"]}, "color": {"type": ["string", "null"]}, "initials": {"type": ["string", "null"]}, "email": {"type": ["string", "null"]}, "profilePicture": {"type": ["string", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "watchers": {"type": ["array", "null"]}, "checklists": {"type": ["array", "null"]}, "tags": {"items": {"properties": {"name": {"type": ["string", "null"]}, "tag_fg": {"type": ["string", "null"]}, "tag_bg": {"type": ["string", "null"]}, "creator": {"type": ["integer", "null"]}}, "type": "object"}, "type": ["array", "null"]}, "parent": {"type": ["string", "null"]}, "due_date": {"type": ["null", "string"]}, "start_date": {"type": ["string", "null"]}, "points": {"type": ["string", "number", "null"]}, "time_estimate": {"type": ["integer", "null"]}, "custom_fields": {"items": {"properties": {"id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "type": {"type": ["string", "null"]}, "type_config": {"properties": {"default": {"type": ["integer", "null"]}, "placeholder": {"type": ["string", "null"]}, "options": {"items": {"properties": {"id": {"type": ["string", "null"]}, "name": {"type": ["string", "null"]}, "color": {"type": ["string", "null"]}, "orderindex": {"type": ["integer", "string", "null"]}}, "type": "object"}, "type": ["array", "null"]}}, "type": "object"}, "date_created": {"type": ["string", "null"]}, "hide_from_guests": {"type": ["string", "boolean", "null"]}, "required": {"type": ["string", "boolean", "null"]}, "value": {"type": ["integer", "string", "boolean", "null", "array", "object", "number"]}}, "type": "object"}, "type": ["array", "null"]}, "dependencies": {"type": ["array", "null"]}, "linked_tasks": {"type": ["array", "null"]}, "team_id": {"type": ["string", "null"]}, "url": {"type": ["string", "null"]}, "permission_level": {"type": ["string", "null"]}, "list__id": {"type": ["string", "null"]}, "list__name": {"type": ["string", "null"]}, "list__access": {"type": ["boolean", "null"]}, "project__id": {"type": ["string", "null"]}, "project__name": {"type": ["string", "null"]}, "project__hidden": {"type": ["boolean", "null"]}, "project__access": {"type": ["boolean", "null"]}, "folder__id": {"type": ["string", "null"]}, "folder__name": {"type": ["string", "null"]}, "folder__hidden": {"type": ["boolean", "null"]}, "folder__access": {"type": ["boolean", "null"]}, "space__id": {"type": ["string", "null"]}}, "key_properties": ["id"], "bookmark_properties": ["date_updated"]}

Next steps:

  1. This looks like a Meltano SDK bug to me, we should try the latest version of the SDK to see if it was already fixed
  2. Submit a bug report to the SDK folks if that doesn't work
visch commented 2 years ago

@balmasi thanks for the issue submission!

balmasi commented 2 years ago

Thank you for jumping on this so fast! did not expect such a speedy resolution 👏🏽

visch commented 2 years ago

@balmasi

0.0.19 is released to pypi

I saw this and had a few minutes, no problem. Thanks again for the issue submission :D