Open tturkenitz opened 2 months ago
I also hit this issue while working on changes for #32814.
I've wanted to add, that this happens when parsing the response from the server (ie. already have to have something set up, and talking to some end point), and when it happens, there's no request nor response log shown, which makes it more difficult to debug (since it's response parsing, but it's unclear what's the response that triggers it. For example in my case most of the tested API endpoints work just fine, but some reliably triggers this issue.
@tturkenitz could you try the proposed changes in the above PR, if you have a chance? Edit: It's slightly different from your workaround, but the spirit is the same (ie. setting defaults when "type" not available) it should be getting to the root of the problem, see the comment below.
I got my debug system working and actually it's likely a more complex edge case than the linked MR. The node
values that triggers for me has this value:
{
"anyOf": [
{"type": ["boolean", "null", "string"]},
{
"type": "object",
"properties": {
"type": {"type": "string"},
"interval": {"type": "string"},
"maxValue": {"type": "number"},
"minValue": {"type": "number"},
"currencyCode": {"type": "string"},
},
},
]
}
Thus what happens is not that "there's no type info", it's that "the type info is not correctly extracted from an anyOf
?
I've updated the MR to fix the issues correctly for my case, I do wonder if these changes would work for you too, @tturkenitz (or your issue is triggered by some other cleanup edge case)? Is there a chance that you can test-drive this PR?
Hey @imrehg, thanks a lot for picking this up and providing a solution. I'm going to implement the MR on my end and test if it fixes my issue too. I'll report back with my results shortly.
I've tested the fix on my end, but it doesn't resolve my specific issue. The type
field is missing from the document entirely, and re-adding it as STRING
resolves the problem. This occurs when I interact with the Coupa ExpenseReports endpoint and only happens when extracting more than one record. I suspect a schema difference between the records could be the issue but I'm not familiar enough with Airbyte code and my assumption is that Airbyte is able to reconcile the schemas into one master schema document when such things happen, but maybe not in this case?
I can share the schema, but it's quite large at 15k lines, and I’m not sure if it reflects the missing type
accurately. It seems to be the schema Airbyte retrieved from the first document.
Adding
if node.get("type", "") == "":
node["type"] = "string"
does solve the issue for me, but I doubt it's production ready code 😅
@tturkenitz for testing could you try a debug step?
In schema_inferrer.py
:
below that line add:
if "type" not in node:
print(node)
which should print the node data in the logs when the cleaning step fails. This would show what's your offending schema content.
Let me know if any of this is unclear 😬
I would be surprised if the schema inferrer wouldn't have any info to go on it (the type
doesn't come from your source, but the tool that looks at the source's response).
I suspect a schema difference between the records could be the issue
That's totally the case for my breakage as well, that's when the inferrer would end up with an anyOf
entry (a collection of variations of the data encoded in them), and the handling of that anyOf
is the problematic bit in the code.
@imrehg,
I captured the malformed schema. It seems I was actually wrong, type
does exist in the schema, but it is set to Null
. It confuses me, because my solution assumed that type
is completly missing and it adds it back to the schema. But maybe, type
is removed by Airbyte as it processes the schema and my solution re-adds it? Lots of assumptions, sorry!
Here is the node and you can see that there are multiple attributes like parent-id
, salesforce-id
and avatar-thumb-url
where type is null.
{
"anyOf": [
{
"type": "string"
},
{
"type": "object",
"properties": {
"parent-id": {
"type": "null"
},
"lookup": {
"type": "object",
"properties": {
"content-groups": {
"type": "array",
"items": {
"type": "object",
"properties": {
"updated-by": {
"type": "object",
"properties": {
"salesforce-id": {
"type": "null"
},
"avatar-thumb-url": {
"type": "null"
}
}
}
}
}
}
}
}
}
}
]
}
This is the log row header:
airbyte-connector-builder-server | 2024-06-06 14:14:47 INFO i.a.w.i.VersionedAirbyteStreamFactory(logMalformedLogMessage):390
Hey @tturkenitz thanks for passing on the node information!
I'm skeptical of the null
causing any issues. I think schema inferrer getting null
would happen if all the examples the inferrer has seen had null
as the value (and APIs can send that back: send a field name but setting it explicitly to null
, rather than only sending the field if it has a non-null value).
Instead, your troublesome node seems to have the same characteristics as mine (anyOf
that isn't just a null
and something else as two entries)
I've tested your node
info, and I've found that.
master
branch indeed breaks with that KeyError
Sorry for sounding basic, could you check if you were actually using the patched version of the CDK from that merge request?
Hi @imrehg , your patch works fine and behaves as expected.
Looks good and hopefully it can be merged and released in future versions soon. Really appreciate you taking the lead on this 🙇
Connector Name
connector-builder
Connector Version
0.61.0
What step the error happened?
Configuring a new connector
Relevant information
Environment
Airbyte version: OSS Step where error happened: Connector Builder: Test
Current Behavior
When attempting to test Connector Builder configuration the process will fail during schema inference if
type
is Null or missing.Stacktrace:
Issue seems to be when evaluating
schema_inferrer.py
at line 112:I was able to get around it by modifying the code locally and adding this check before line 112:
Expected Behavior
I think it would be great if the connector builder can implicitly convert null types to STRING. It would also be nice if a more detailed error regarding which "Attribute" (aka Node) is causing the issue.
I can't upload the full logs. I noticed that
Secrets
are included in the logs for the Connector Builder I would rather not risk it.Relevant log output
No response
Contribute