airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.53k stars 4k forks source link

influxDB connector #4767

Open ManuelRomer opened 3 years ago

ManuelRomer commented 3 years ago

Tell us about the new connector you’d like to have

Describe the context around this new connector

InfluxDB is one of the best DB for IoT data streaming. We are studying related use-cases, and we don't really know how oftenly syncs will be made.

Describe the alternative you are considering or using

we are also applying postgreSQL to manage it

Are you willing to submit a PR?

In fact, we are now trying to develop another custom connector that will be available when it's ready and confirmed by community experts, about another kind of thigs, based on API. When we finish it, we will consider if it proceeds.

We understand there's a lot of job ahead, and even in my case, I'm still getting started. Thank you so much for doing these great things possible

karinakuz commented 3 years ago

Integration Vetting

Webhook-based? (no/partially/yes)
if partially, mention which endpoints are webhook-based
 no

Available authentication modes (API key/Oauth/other)
 API token with organisation id and user id https://docs.influxdata.com/influxdb/cloud/security/tokens/

Connecting to the data source
what kind of API is it? (SOAP/GraphQL/REST/other)
 HTTP API https://docs.influxdata.com/influxdb/cloud/api/

is there an SDK available? Can we connect with HTTP requests directly? 
no

If this is a database/datalake/pubsub source, does it have a JDBC driver or Java SDK?
 no

Creating an account
Select one of the below options
self-service free account available?_ If yes then create it using integration-test@airbyte.io (you should be getting emails sent to this email address)
 30-day free trial

Will it require some work through the UI? If so, what is required?
 no

Available streams for sync
If this is an API connector:
 Provide a link  to the reference docs
https://docs.influxdata.com/influxdb/cloud/api/#section/Authentication

Main streams: 1. Dashboard https://docs.influxdata.com/influxdb/cloud/api/#operation/GetDashboards 2. Dashboard https://docs.influxdata.com/influxdb/cloud/api/#operation/GetDashboardsID (use Dashboard ID 0824f509d4c99000) 3. Labels https://docs.influxdata.com/influxdb/cloud/api/#operation/GetLabels 4. Dashboard Labels https://docs.influxdata.com/influxdb/cloud/api/#operation/GetDashboardsIDLabels (use Dashboard ID 0824f509d4c99000) _5. Dashboard Cel_l https://docs.influxdata.com/influxdb/cloud/api/#operation/GetDashboardsIDCellsIDView (use Dashboard ID 0824f509d4c99000 and Cell ID 0824f509e44bf000) 6. Dashboard Members https://docs.influxdata.com/influxdb/cloud/api/#operation/GetDashboardsIDMembers
7. Dashboard Owners https://docs.influxdata.com/influxdb/cloud/api/#operation/GetDashboardsIDOwners
8. Query Suggestions https://docs.influxdata.com/influxdb/cloud/api/#operation/GetQuerySuggestions 9. Buckets https://docs.influxdata.com/influxdb/cloud/api/#operation/GetBuckets 10. Bucket https://docs.influxdata.com/influxdb/cloud/api/#operation/GetBucketsID (use Bucket ID 08d1416d213812de) 11. Organisations https://docs.influxdata.com/influxdb/cloud/api/#tag/Organizations 12. Organisation https://docs.influxdata.com/influxdb/cloud/api/#operation/GetOrgsID (use Organisation ID e918412d6d1caff5) 13. Organisation Members https://docs.influxdata.com/influxdb/cloud/api/#operation/GetOrgsIDMembers 14. Organisation Owners https://docs.influxdata.com/influxdb/cloud/api/#operation/GetOrgsIDOwners 15. Tasks https://docs.influxdata.com/influxdb/cloud/api/#operation/GetTasks 16. Task https://docs.influxdata.com/influxdb/cloud/api/#operation/GetTasksID (use Task ID 082519e4b49fe000) 17. Task Logs https://docs.influxdata.com/influxdb/cloud/api/#operation/GetTasksIDLogs (use Task ID 082519e4b49fe000) 18. Users https://docs.influxdata.com/influxdb/cloud/api/#operation/GetUsers 19. User https://docs.influxdata.com/influxdb/cloud/api/#operation/GetUsersID 20. Notification Rules https://docs.influxdata.com/influxdb/cloud/api/#operation/GetNotificationRules 21. Notification Rule https://docs.influxdata.com/influxdb/cloud/api/#operation/GetNotificationRulesID (Use Rule ID 08251d80e751d000) 22. Notification Endpoints https://docs.influxdata.com/influxdb/cloud/api/#operation/GetNotificationEndpoints 23. Notification Endpoint https://docs.influxdata.com/influxdb/cloud/api/#operation/GetNotificationEndpointsID (Use Endpoint ID 08251d34ca60e000)


Please mention if any of the endpoints are asynchronous (meaning we have to make a request, wait for the server to compute a response, then make another request to obtain the result)
 no

if this is a source where the schema is dynamic/not fixed e.g: database, mention the data model of the connector i.e: relational, document-based, graph-based, etc
queries no

Do any streams support incremental sync?
not necessary to mention all of them, just saying “none”, “some”, or “all” is sufficient. If only some streams, please mention which ones support incremental sync no

karinakuz commented 3 years ago

Creating the sandbox account and generating access credentials to the account [ ] create an account, get Token, add credentials to LastPass in 'influxDB' (2h)


Populating the account with data and receiving them from API [ ] Create in Postman API requests for the main streams (8h)

[] Describing the output schema Schemas https://drive.google.com/drive/folders/1vZPAKtBvlhoJ4iYl3il0otpO1mkDY7bU?usp=sharing (1h)