search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.94k stars 3.84k forks source link

[source-greenhouse] Application Stream Incremental on wrong cursor and missing custom fields #38236

Open jimmc2 opened 2 months ago

jimmc2 commented 2 months ago

Connector Name

source-greenhouse

Connector Version

0.5.1

What step the error happened?

Other

Relevant information

The "Application" stream for Greenhouse is set to be incremental based on the cursor field "created_after" and table field "applied_at" based on the manifest here. This is an issue because any updates to an application are not caught by this connector set to "incremental" mode because the connector only fetches new applications that are since the last polled "applied_at" date. This requires anyone wanting updated data that would include all new applications and all updated applications to use a full refresh sync which greatly increases the number of requests sent to Greenhouse.

Additionally, the Application stream does not include the custom_fields and keyed_custom_fields included in the api response despite them being available in the raw data stored by Airbyte. I will submit a fix for both of these.

Relevant log output

No response

Contribute

jimmc2 commented 2 months ago

Will base my PR on the changes in this PR where custom_fields were added to the Candidates stream

marcosmarxm commented 2 months ago

Thanks @jimmc2 I'll review during the week your contribution!