search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.2k stars 4.14k forks source link

[connector-builder] Custom Connector NULL values escaped #36061

Open radaniba opened 8 months ago

radaniba commented 8 months ago

Connector Name

custom

Connector Version

3

What step the error happened?

Configuring a new connector

Relevant information

Hello I built a custom connector using your tutorial and everything works perfectly well except one issue I noticed that all NULL values are omitted in the records generated, the response body shows these attributes with NULL data but the records generated escaped these fields for some reason, and the schema does detect them in the Builder. Any idea to tall Airbyte NOT to bypass these attributes ?

Relevant log output

No response

Contribute

marcosmarxm commented 8 months ago

Are you able to share the custom code?

radaniba commented 8 months ago

Would love to, I just used the builder GUI didn't really write a code for it

marcosmarxm commented 8 months ago

Do you mind sharing the YAML file?

radaniba commented 8 months ago

just checked the yaml and for privacy reasons I am not able to share everything But will try to do my best to clarify the problem

My Schema detector for some reason is skipping the null attributes even though they are reported by the response API endpoint

example

Response :

{
"enGaUsWks": "21",
      "enGaUsDys": "0",
      "delOutcome": null,
      "Outcom": null,
      "neoOut": null
}

What is returned by the connector

{
"enGaUsWks": "21",
    "enGaUsDys": "0"
}

The Schema only detects those 

"enGaUsDys": {
      "type": [
        "null",
        "string"
      ]
    },
    "enGaUsWks": {
      "type": [
        "null",
        "string"
      ]
    },