airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.39k stars 3.97k forks source link

New Source: Paypal Reports #2801

Open sherifnada opened 3 years ago

sherifnada commented 3 years ago

Tell us about the new integration you’d like to have

A user (jtalmi ) has indicated interest in pulling data from the Paypal Reports API

Describe the context around this new integration

## Describe the alternative you are considering or using Currently the only alternative is writing a custom script since other ELT providers don't offer this connector ┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368158845034) by [Unito](https://www.unito.io)
sherifnada commented 3 years ago

@jtalmi could you comment about the following:

  1. Can you speak to the schema that would be most consumable for this report data? Rough outline is fine, doesn't have to be exact schema e.g: should each report type land in a different table? Which reports would you be most interested in pulling?
  2. What is the interface/UX you would expect for the reporting API? What would be the inputs/configurations you expect to supply?
  3. Would you want to pull these reports on a recurring, incremental basis (e.g: once a day) or would you only need to do this in very specified batches?

Any other context you can provide is super helpful.

karinakuz commented 3 years ago

Integration Vetting

Webhook-based? (no/partially/yes) if partially, mention which endpoints are webhook-based Partially (no events for Payflow Reports) https://developer.paypal.com/docs/api-basics/notifications/webhooks/event-names/

Available authentication modes (API key/Oauth/other) OAuth 2.0

Connecting to the data source what kind of API is it? (SOAP/GraphQL/REST/other) Payflow Gateway has an XML API which enables you to develop web and desktop applications that create, schedule, and run standard (pre-defined) and custom Payflow services reports. https://developer.paypal.com/docs/reports/payflow-reports/

is there an SDK available? Can we connect with HTTP requests directly? To submit a request to the Reporting API, you post your XML request through HTTPS to the host URLs in the body of the request. No SDK is needed since you just perform a standard HTTPS post using the language of your choice. The Content-Type for the request should be text/plain.

If this is a database/datalake/pubsub source, does it have a JDBC driver or Java SDK? not for Payflow Reports

Creating an account Select one of the below options self-service free account available?_ If yes then create it using integration-test@airbyte.io (you should be getting emails sent to this email address) yes, Sandbox

How to populate the account with data?

Can it be done entirely through the API? yes

Will it require some work through the UI? If so, what is required? yes, create transactions

Will it require performing some paid action like running an Ad campaign? If so, please note the action needed, estimated cost, and estimated effort in hours. If a task will take more than 4 hours, please break it down into smaller tasks each of which take no more than 4 hours no

Do you need any actions from Airbyte to populate the account with data? If so, please tag @ sherifnada for support no

karinakuz commented 3 years ago

Available streams for sync

If this is an API connector:

  1. Provide a link  to the reference docs Use the Reporting API https://developer.paypal.com/docs/reports/payflow-reports/use-the-api/ Samples https://developer.paypal.com/docs/reports/payflow-reports/samples/ Report Parameters https://developer.paypal.com/docs/reports/payflow-reports/report-parameters/

Main streams - Standard Reports:

1. Activity report 2. Settlement report 3. Transactions Summary report 4. Shipping and Billing report 5. Batch ID report 6. Audit Activity report

  1. Please mention if any of the endpoints are asynchronous (meaning we have to make a request, wait for the server to compute a response, then make another request to obtain the result)

if this is a source where the schema is dynamic/not fixed e.g: database, mention the data model of the connector i.e: relational, document-based, graph-based, etc. yes

Do any streams support incremental sync? not necessary to mention all of them, just saying “none”, “some”, or “all” is sufficient. If only some streams, please mention which ones support incremental sync no

Preparation scope & blockers What information, materials, or anything else you will need from Airbyte in order to perform preparation for this connector? no

Other information/blockers place any other useful information you learned here no

karinakuz commented 3 years ago

Populating the account with data [ ] Do activities on the account: open/close 10 times, add transactions (UI), create API queries in Postman to create the reports and get data of the reports. Add notes to LastPass Description with name Payflow Reports (12h).

Get data in responses for the next requests https://docs.google.com/document/d/1TY4H1uQwV-pJWJh_t01DhJekTMxelrNQOwX6w9zzzPQ/edit?usp=sharing

Describing the output schema XML Schema https://developer.paypal.com/docs/reports/payflow-reports/xml-schema/