New Source: Lever Hiring

sherifnada commented 3 years ago

Tell us about the new integration you’d like to have

https://hire.lever.co/developer

Describe the context around this new integration

Describe the alternative you are considering or using

┆Issue is synchronized with this Asana task by Unito

yevhenii-ldv commented 3 years ago

Integration Vetting

Webhook-based?

no

Available authentication modes (API key/Oauth/other)

Access and Refresh Tokens can be used

Creating an account

@sherifnada @arhip11 I could not create an account for Lever in any way, I could only send a request for a demo. I need help from Airbyte to create account for Lever.

How to populate the account with data?

Using API and UI.

Available streams for sync

Applications
Archive Reasons
Audit Events
Feedback
Feedback Templates
Files
Interviews
Notes
Offers
Opportunities
Panels
Postings
Profile Forms
Profile Form Templates
Referrals
Requisitions
Requisition fields
Resumes
Sources
Stages
Tags
Users

Integration supports incremental sync?

Some

Other information/blockers

sherifnada commented 3 years ago

Email sent to integration-test@airbyte.io titled "Product Integration with Lever - Next Steps" with next steps on how to create account

johnlafleur commented 3 years ago

@sherifnada we now have our own account on Lever. Don't hesitate to ping Anthony to get an access

sherifnada commented 3 years ago

@johnlafleur thanks for the heads up! We also have a sandbox we can use for testing ;) (separate from the recruiting one)

sherifnada commented 3 years ago

@karinakuz can you move this into the sprint once it's ready for implementation?

karinakuz commented 3 years ago

Lever

Integration Vetting

Webhook-based? (no/partially/yes) if partially, mention which endpoints are webhook-based yes https://hire.sandbox.lever.co/developer/documentation#webhooks

Available authentication modes (API key/Oauth/other) Oauth (API Keys are used by customers to build internal workflows. Integrations are built using OAuth authentication.)

Connecting to the data source what kind of API is it? (SOAP/GraphQL/REST/other) REST API

is there an SDK available? Can we connect with HTTP requests directly? no

If this is a database/datalake/pubsub source, does it have a JDBC driver or Java SDK? no

Creating an account yes, Sandbox access will last three months

How to populate the account with data?

Can it be done entirely through the API? no, we sent request to get token with write access, but no answer for more than a week

Will it require some work through the UI? If so, what is required? yes

Will it require performing some paid action like running an Ad campaign? If so, please note the action needed, estimated cost, and estimated effort in hours. If a task will take more than 4 hours, please break it down into smaller tasks each of which take no more than 4 hours no

Do you need any actions from Airbyte to populate the account with data? If so, please tag @ sherifnada for support no

Available streams for sync If this is an API connector:

Provide a link to the reference docs _https://hire.sandbox.lever.co/developer/documentation#lever-api-reference Main streams:

Please mention if any of the endpoints are asynchronous (meaning we have to make a request, wait for the server to compute a response, then make another request to obtain the result)

if this is a source where the schema is dynamic/not fixed e.g: database, mention the data model of the connector i.e: relational, document-based, graph-based, etc. no

Do any streams support incremental sync? not necessary to mention all of them, just saying “none”, “some”, or “all” is sufficient. If only some streams, please mention which ones support incremental sync some

Preparation scope & blockers What information, materials, or anything else you will need from Airbyte in order to perform preparation for this connector? no

Other information/blockers place any other useful information you learned here no

karinakuz commented 3 years ago

Populating the account with data [ ] Do activities on the account: open/close 10 times, change profile data, fill account with the data manually (Lever didn't give write access to token). Create API for main streams and verify I get in response correct data from account. Add notes to LastPass Description (8h).

We don't have Lever access to streams:

audit_events
contact
feedback
requisition_fields
requisitions
tags
webhooks

karinakuz commented 3 years ago

Describing the output schema [ ] Converting JSON to JSON Schema (30m) https://drive.google.com/drive/folders/1H1XQkC5-avs97zZ9ELvpgTEmtbRaDKRN?usp=sharing

yevhenii-ldv commented 3 years ago

Scoping summary

1. What is the “ideal end state” for the features offered by this connector?

Lever REST entities (subentities) map to Airbyte streams
We will use client_id, client_secretand refresh_token to obtain access_token, and use it to send HTTP requests.

2. What is the release roadmap for this connector?

Which features are absolutely necessary for the MVP release?

I believe that all streams can be realized in MVP, since there are no particular problems to get each of them.
Which features can be released as follow ups?

So far there are none
- Task breakdown
- [x] Create source scaffolding and populate schemas folder
- [x] Implement spec, check, and discover
- [x] Implement Read
  - [x] Create API base abstract class
- [x] Implement Full Refresh for all streams
- [x] Implement Incremental for streams, that support this feature
- [x] Handle rate limiting
- [ ] Implement SAT

3. Check if and how Fivetran or Stitch Data support this connector

Do they do something better than our proposed implementation?

I believe that no, we can receive absolutely all streams that REST API allows us. Fivetran does the same.
Is there something we can/should do better?

We will remove duplicate elements in streams from other streams (for example, the parent stream's entity, minimizing it to the <parent_stream_name>_id)

airbytehq / airbyte