airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.1k stars 4.12k forks source link

Source Pipedrive - Do not recognize the right data type of a stream field #27462

Open quangdvn-sun opened 1 year ago

quangdvn-sun commented 1 year ago

Connector Name

Pipedrive

Connector Version

0.1.17

What step the error happened?

During the sync

Revelant information

When syncing product_fields, Pipedrive connector auto detect some fields data type as number even though the data contains "true" and "false" value, which causes the sync flow failed

product_fields consists of numbers of boolean fields

image

Relevant log output

100038 (22018): Numeric value 'false' is not recognized
  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/XXXXX/PRODUCT_FIELDS_OPTIONS.sql,retryable=<null>,timestamp=1687152540064], io.airbyte.config.FailureReason@443adae1[failureOrigin=normalization,failureType=system_error,internalMessage=100038 (22018): Numeric value 'false' is not recognized,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@1dc57d82[additionalProperties={attemptNumber=2, jobId=7518, from_trace_message=true}],stacktrace=AirbyteDbtError:

Contribute

quangdvn-sun commented 1 year ago

This is likely the root cause,

"options": {
      "type": ["null", "array"],
      "items": {
        "type": "object",
        "properties": {
          "id": {
            "type": ["null", "boolean", "integer"]
          },
          "label": {
            "type": ["null", "string"]
          }
        }
      }
    }

options.id allows boolean value but somehow Airbyte only check if it's an integer value