airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.74k stars 3.79k forks source link

Connector UI builder generates invalid YAML #25004

Closed svlada closed 1 year ago

svlada commented 1 year ago

Environment

Current Behavior

Connector is generated using connector builder UI tool. This is the segment of the generated YAML file - when incremental sync option is selected on the UI:

incremental_sync:
  type: DatetimeBasedCursor
  cursor_field: lastModifiedDate
  datetime_format: '%Y-%m-%d'
  cursor_granularity: P1D
  end_datetime: '2024-04-01'
  start_datetime: '2023-04-01'
  step: P1D
  start_time_option:
    inject_into: body_json
    type: RequestOption
    field_name: ddd

The problem is that DatetimeBasedCursor is not available in cdk 0.28.0.

Here is the exception thrown when python main.py spec is executed:

airbyte-integrations/connectors/source-custom-connector/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 198, in _create_subcomponent
    class_name = CLASS_TYPES_REGISTRY[object_type]
KeyError: 'DatetimeBasedCursor

Expected Behavior

Connector UI builder should generate file compatible with the version it generates.

Logs

airbyte-integrations/connectors/source-custom-connector/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py", line 198, in _create_subcomponent
    class_name = CLASS_TYPES_REGISTRY[object_type]
KeyError: 'DatetimeBasedCursor
timroes commented 1 year ago

cc @lmossman @flash1293

lmossman commented 1 year ago

@svlada This issue was caused by some breaking changes being made to the CDK and the connector builder.

To resolve this, you will need to either downgrade your Airbyte deployment to version 0.40.32, or upgrade it to the most recent version. Do note however, that if you upgrade to the most recent version, you will need to update your connector manifest to fix the breaking changes as described in the changelog here and on slack here

Upgrading airbyte will also clear out your existing data in the connector builder as we have changed where manifests are stored, so you will need to fix your manifest as described above and then import it into the connector builder through the UI (importing YAML files is now an option when going to localhost:8000/connector-builder on the most recent version of Airbyte)

flash1293 commented 1 year ago

Thanks for reporting! Closing as this is fixed in the most recent version