search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.45k stars 3.98k forks source link

Pipedrive connector is not bringing all the deal fields #37747

Open laumc98 opened 4 months ago

laumc98 commented 4 months ago

Connector Name

source-pipedrive

Connector Version

2.2.2

What step the error happened?

During the sync

Relevant information

I've set a connection between Pipedrive and Redshift, but there are several deal custom fields (the ones with long hashes as keys) that are not being synchronized to my warehouse. I've already looked at the raw data, and the fields are not there either.

Relevant log output

No response

Contribute

natikgadzhi commented 4 months ago

Note: Pipedrive connector needs a bit of cleanup, this PR should get most of the boilerplate out of the way: https://github.com/airbytehq/airbyte/pull/37224

Once that's merged, we should take a look at how the additional schema fields are populated.

@laumc98, is it that absolutely none of the dynamic fields are getting synced, or only the ones with long names? Can you share examples of fields that worked and field names that don't show up?

neurotroph commented 2 months ago

Posted in another issue (#9433), but as that is closed, I'd better post it here again: https://github.com/airbytehq/airbyte/issues/9433#issuecomment-2196962520

By Pipedrive's documentation:

Each deal consists of standard data fields but can also contain a number of custom fields. The custom fields can be recognized by long hashes as keys. These hashes can be mapped against DealField.key. The corresponding label for each such custom field can be obtained from DealField.name.

These custom fields are missing from both the normalized tables and the raw JSON data in airbyte_internal. It would be great if this gets addressed.

My recommended solution would be to include a column custom_fields in the normalized output which includes the custom field data as JSON. To my understanding this would be the most flexible solution.

natikgadzhi commented 2 months ago

@neurotroph thank you for pinging us with this again. We should get to it, but Pipedrive is one of the marketplace connectors, and we don't quite have the capacity to work on it right away.

I hope we can get to this in a couple months after we get all connectors on more standardized tooling.

In the meantime, if anyone has an implementation for custom fields in their connector builder and wants to copy it here, I'm happy to try and push it in