The logic for flattening indexes in the flatten_schema function selects v.values()[0][0]here, which in the above case case is { "type": "null" }. This case is unhandled.
This bug means that any properties declaring an anyOf clause in its property definition are not guaranteed to be interpreted correctly, and in my case are excluded completely.
It's worth noting that I created the schema using the genson python tool, which is pretty common amongst tap developers. This likely means may taps (especially more recent ones) may use anyOf in this order.
To Reproduce
Steps to reproduce the behavior:
Send SCHEMA message that defines an anyOf property where the first element is { "type": "null" } to target-duckdb
Send a corresponding RECORD message with non-null data conforming to the above schema.
Observe that no corresponding column is created in duckdb.
Expected behavior
Ideally target-duckdb would iterate until the first non-null type anyOf, rather than assuming the first value will be the most likely intended type.
Screenshots
If applicable, add screenshots to help explain your problem.
Your environment
Version of target: [e.g. 2.0.0] 0.4.3
Version of python [e.g. 3.8] 3.10
Additional context
Add any other context about the problem here.
Describe the bug Given the following jsonschema property definition:
The logic for flattening indexes in the
flatten_schema
function selectsv.values()[0][0]
here, which in the above case case is{ "type": "null" }
. This case is unhandled.This bug means that any properties declaring an
anyOf
clause in its property definition are not guaranteed to be interpreted correctly, and in my case are excluded completely.It's worth noting that I created the schema using the
genson
python tool, which is pretty common amongst tap developers. This likely means may taps (especially more recent ones) may useanyOf
in this order.To Reproduce Steps to reproduce the behavior:
SCHEMA
message that defines ananyOf
property where the first element is{ "type": "null" }
totarget-duckdb
RECORD
message with non-null data conforming to the above schema.Expected behavior Ideally
target-duckdb
would iterate until the first non-null typeanyOf
, rather than assuming the first value will be the most likely intended type.Screenshots If applicable, add screenshots to help explain your problem.
Your environment
Additional context Add any other context about the problem here.