Open cjwooo opened 5 months ago
I found on an old thread on the Airbyte community Slack suggesting that nested cursor fields are not supported due to restrictions for supporting normalization. If this is true, the Airbyte Protocol documentation and CDK should be updated, because they both show nested fields as a valid option.
@natikgadzhi can you confirm this is an issue or responsibility of extensibility team?
Platform Version
0.51.0, 0.53.1
What step the error happened?
Other
Revelant information
I have a custom source that contains a stream with
source_defined_cursor = true
anddefault_cursor_field = ['computedProperties', 'updatedAt']
. After creating the source and destination via the Airbyte UI, theNew connection
page throws an error:There is no error actually surfaced in the logs shown by the UI. But in the docker-compose logs, I see that Airbyte errored on validation of the source's
discover
output:Source defined cursor validation failed for stream: pipelines. Error: key: 'updatedAt' of path: '[computedProperties, updatedAt]' not found in schema: {...}
.The full message is in the "Relevant log output" section, and towards the end, you can see that there is an
updatedAt
field in the stream schema that is a nested field in thecomputedProperties
object param.This issue does not occur on our currently deployed Airbyte version 0.50.31. I was able to produce this issue with a Docker deployment on version 0.51.0 and 0.53.1.
Relevant log output