search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.08k stars 4.12k forks source link

[source-pipedrive] Pipedrive custom fields still do not sync #33622

Open matebaranyi1995 opened 10 months ago

matebaranyi1995 commented 10 months ago

Connector Name

source-pipedrive

Connector Version

2.2.1

What step the error happened?

During the sync

Relevant information

This issue was fixed according to this: https://github.com/airbytehq/airbyte/issues/25766, but it seems to be not the case with any versions since the fix (2.0.1, 2.1, 2.2, 2.2.1).

The data from the Pipedrive custom fields (https://pipedrive.readme.io/docs/core-api-concepts-custom-fields) still do not sync to our Snowflake destination (v3.1.2 and v3.4.2). I tried to start from scratch, deleting destination tables and connections first but that didn't help. The custom fields only appear in the xxx_fields tables but the custom fields and their values won't appear in the xxx tables or the raw internal tables. I could not find any reference to the custom fields on the Airbyte replication page or in the sync logs.

I also could not find any reference to the custom fields in the Airbyte Pipedrive connector's source code. I thought that there should be something about this in your code because according to the Pipedrive API docs (https://developers.pipedrive.com/docs/api/v1/Deals#getDeals), custom fields appear with their UUID in the API request results (we see these in the _fields tables), and I would imagine that some transformation is needed on the connector side to map those UUIDs to a human-readable name in the destination tables.

Sorry for opening a new issue. I am not sure if it is a different issue or if the previous one wasn't solved.

Relevant log output

No response

Contribute

matebaranyi1995 commented 8 months ago

I tried to check if something is misconfigured in our Pipedrive: directly calling the Pipedrive API endpoints (https://developers.pipedrive.com/docs/api/v1/Deals#getDeals) returns the custom fields and their values as expected with their UUID as key. I think something is missing from the Airbyte connector.

laumc98 commented 6 months ago

Are there any updates on this? I'm facing the same issue

rbeilvert commented 5 months ago

@marcosmarxm any idea why this still does not work ? This would be a key feature, not having access to these fields defeats the purpose of using airbyte for this connector for us