Scrape EL Connector Libraries

[pnadolny13]

Related to https://github.com/meltano/internal-marketing/issues/579#issuecomment-1215620793 (internal).

We should scrape other EL connector ecosystem to understand what niche taps are available in the Singer ecosystem but not in others.

• Fivetran https://www.fivetran.com/connectors
• Stitch https://www.stitchdata.com/integrations/sources/
• Airbyte https://docs.airbyte.com/integrations/ - It might be easier to scrape Airbyte's github repo https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors

Scope:

• write a python script to scrape the 3 sites
• load results to snowflake
• stage data
• output a diff, maybe a view

[pnadolny13]

Airbyte sources that I dont see on our hub, theres a chance I missed mapping a few because the names are slightly different:

• [x] adjust
• [x] alloydb
• [x] amazon-sqs
• [x] amplitude
• [x] apify-dataset
• [x] appfollow
• [x] aws-cloudtrail
• [x] cart
• [x] chartmogul
• [x] clickhouse
• [x] cockroachdb
• [x] commercetools
• [x] courier
• [x] db2
• [x] delighted
• [x] dockerhub
• [x] drift
• [x] dv-360
• [x] elasticsearch
• [x] fauna
• [ ] file-secure
• [x] firebolt
• [x] flexport
• [x] freshcaller
• [x] freshsales
• [x] freshservice
• [x] glassfrog
• [ ] google-analytics-data-api
• [ ] google-analytics-v4
• [ ] google-directory
• [x] google-workspace-admin-reports
• [x] gutendex
• [x] hubplanner
• [x] jdbc
• [x] kyriba
• [x] lemlist
• [x] linkedin-pages
• [x] linnworks
• [x] metabase
• [x] mssql ???
• [x] my-hours
• [x] notion
• [x] onesignal
• [ ] orb
• [x] paystack
• [x] plaid
• [x] pokeapi
• [x] posthog
• [x] prestashop
• [x] primetric
• [x] qualaroo
• [ ] relational-db
• [x] retently
• [ ] rki-covid
• [x] salesloft
• [x] search-metrics
• [x] shortio
• [ ] stock-ticker-api-tutorial
• [x] talkdesk-explore
• [x] tempo
• [x] tidb
• [x] timely
• [x] us-census
• [x] webflow
• [x] whisky-hunter
• [x] wrike
• [x] xkcd
• [x] yahoo-finance-price
• [x] yandex-metrica
• [x] zendesk-sunshine
• [x] zendesk-support
• [x] zendesk-talk
• [x] zenloop
• [x] zoho-crm

[pnadolny13]

Airbyte targets not listed on the hub:

1. • kvdb
2. • firestore
3. • mariadb-columnstore
4. • r2
5. • scylla
6. • firebolt
7. • mssql
8. • oracle
9. • redis
10. • elasticsearch
11. • rabbitmq
12. • tidb
13. • clickhouse
14. • rockset
15. • amazon-sqs
16. • databricks
17. • jdbc
18. • keen
19. • pubsub
20. • kafka
21. • mongodb

[aaronsteers]

Posting here just the reminder that targets should not be in scope. There are no adapters as of now for Airbyte targets, only tap-airbyte exists.

[tayloramurphy]

These targets would be in scope for any sort of comparison page though!

[pnadolny13]

@tayloramurphy This is kind of meta but as part of testing I used the airbyte dockerhub connector to pull the airbyte dockerhub download stats and here are the top sources in descending order. We might want to target those! Although I think all of these would be alternative variants minus a few (source-amplitude) that we already added.

• [ ] source-postgres 551335
• [ ] source-pokeapi 213368
• [ ] source-google-sheets 187293
• [ ] source-facebook-marketing 157901
• [ ] source-mysql 150080
• [ ] source-stripe 129416
• [ ] source-salesforce 116326
• [ ] source-google-ads 113864
• [x] ~source-e2e-test 111173~
• [ ] source-hubspot 110908
• [ ] source-shopify 98628
• [ ] source-bing-ads 94777
• [ ] source-s3 82659
• [x] ~source-googleanalytics-singer 73310~
• [ ] source-bigquery 67111
• [ ] source-file 63454
• [ ] source-mssql 56329
• [x] ~source-acceptance-test 56245~
• [ ] source-cockroachdb 53423
• [ ] source-zendesk-support 52631
• [ ] source-intercom 50184
• [ ] source-github 47156
• [ ] source-mongodb-v2 43456
• [ ] source-google-analytics-v4 42416
• [ ] source-snowflake 40736
• [ ] source-mixpanel 37895
• [ ] source-amplitude 37304
• [ ] source-jira 33997
• [ ] source-gitlab 31445
• [ ] source-redshift 29425
• [ ] source-freshdesk 26205
• [ ] source-amazon-seller-partner 26102
• [ ] source-twilio 25993
• [ ] source-typeform 25241
• [ ] source-chargebee 24887
• [ ] source-google-search-console 24305
• [ ] source-oracle 24231
• [ ] source-amazon-ads 24206
• [ ] source-kafka 23899
• [ ] source-linkedin-ads 23465
• [ ] source-sendgrid 23103
• [ ] source-db2 22267
• [ ] source-exchange-rates 21896
• [ ] source-tiktok-marketing 21816
• [ ] source-klaviyo 21664
• [ ] source-greenhouse 21404
• [ ] source-mailchimp 21390
• [ ] source-paypal-transaction 21020
• [ ] source-recharge 20289
• [ ] source-slack 20164

[pnadolny13]

@tayloramurphy I improved my script for diffing the hub and airbyte. I have it pointed at our branch now so this is the list of extractors they have that arent listed on ours, then theres a much larger list of extractors we have where airbyte can be an alternative variant:

• [x] aha
• [x] alpha-vantage
• [x] apify-dataset
• [x] ashby
• [x] azure-table
• [x] babelforce
• [x] braze
• [x] breezometer
• [x] callrail
• [x] coda
• [x] coin-api
• [x] coinmarketcap
• [x] configcat
• [x] convertkit
• [x] convex
• [x] copper
• [x] datascope
• [x] emailoctopus
• [x] faker
• [x] fastbill
• [ ] file
• [ ] file-secure
• [x] genesys
• [x] getlago
• [x] gnews
• [x] gocardless
• [x] gong
• [ ] google-analytics-data-api
• [ ] google-analytics-v4
• [ ] google-directory
• [ ] google-pagespeed-insights
• [ ] google-webfonts
• [x] gridly
• [x] intruder
• [x] ip2whois
• [x] k6-cloud
• [x] klarna
• [x] launchdarkly
• [x] lokalise
• [x] mailersend
• [x] mailjet-sms
• [x] microsoft-dataverse
• [x] n8n
• [x] nasa
• [x] news-api
• [x] newsdata
• [x] nytimes
• [x] omnisend
• [ ] orb
• [x] oura
• [x] partnerstack
• [x] pexels-api
• [x] plausible
• [x] polygon-stock-api
• [ ] public-apis
• [ ] punk-api
• [x] pypi
• [x] qonto
• [x] rd-station-marketing
• [x] recreation
• [ ] relational-db
• [x] reply-io
• [ ] rki-covid
• [x] rocket-chat
• [x] sap-fieldglass
• [ ] scaffold-http
• [x] secoda
• [x] senseforce
• [x] sftp-bulk
• [x] smaily
• [x] smartengage
• [x] snapchat-marketing
• [x] sonar-cloud
• [x] spacex-api
• [x] statuspage
• [x] survey-sparrow
• [x] surveycto
• [x] the-guardian-api
• [x] tmdb
• [x] tvmaze-schedule
• [x] tyntec-sms
• [x] vantage
• [x] visma-economic
• [x] vitally
• [x] waiteraid
• [x] weatherstack
• [x] wikipedia-pageviews
• [x] workable
• [x] younium
• [x] zapier-supported-storage

We have a plugin listed already but the airbyte version isnt listed:

• [ ] 3plcentral
• [ ] activecampaign
• [ ] airtable
• [ ] appsflyer
• [ ] appstore
• [ ] asana
• [ ] auth0
• [ ] bamboohr
• [ ] bigcommerce
• [ ] bigquery
• [ ] bing-ads
• [ ] braintree
• [ ] chargebee
• [ ] chargify
• [ ] clickup
• [ ] clockify
• [ ] closeio
• [ ] coingecko
• [ ] confluence
• [ ] datadog
• [ ] db2
• [ ] dixa
• [ ] dynamodb
• [ ] exchangeratesapi
• [ ] facebook
• [ ] facebook-pages
• [ ] feed
• [ ] getpocket
• [ ] github
• [ ] gitlab
• [ ] google-search-console
• [ ] google-sheets
• [ ] googleads
• [ ] greenhouse
• [ ] harvest
• [ ] hellobaton
• [ ] hubspot
• [ ] insightly
• [ ] instagram
• [ ] intercom
• [ ] iterable
• [ ] kafka
• [ ] klaviyo
• [ ] kustomer
• [ ] lever
• [ ] looker
• [ ] mailchimp
• [ ] mailerlite
• [ ] mailgun
• [ ] mailjet
• [ ] marketo
• [ ] mixpanel
• [ ] monday
• [ ] mongodb
• [ ] ms-teams
• [ ] mssql
• [ ] mysql
• [ ] netsuite
• [ ] okta
• [ ] openweathermap
• [ ] oracle
• [ ] orbit
• [ ] outreach
• [ ] pardot
• [ ] paypal
• [ ] persistiq
• [ ] pinterest-ads
• [ ] pipedrive
• [ ] pivotal-tracker
• [ ] postgres
• [ ] postmark
• [ ] quickbooks
• [ ] recharge
• [ ] recruitee
• [ ] recurly
• [ ] redshift
• [ ] salesforce
• [ ] sendgrid
• [ ] sendinblue
• [ ] sentry
• [ ] sftp
• [ ] shopify
• [ ] slack
• [ ] smartsheet
• [ ] square
• [ ] strava
• [ ] stripe
• [ ] surveymonkey
• [ ] tiktok
• [ ] toggl
• [ ] trello
• [ ] twilio
• [ ] twitter
• [ ] typeform
• [ ] woocommerce
• [ ] workramp
• [ ] xero
• [ ] youtube-analytics
• [ ] zenefits
• [ ] zoom
• [ ] zuora

[tayloramurphy]

@pnadolny13 Logos I have not yet gotten or were tough to pick one:

• [ ] file
• [ ] file-secure
• [ ] google-analytics-data-api
• [ ] google-analytics-v4
• [ ] google-directory
• [ ] google-pagespeed-insights
• [ ] google-webfonts
• [ ] orb
• [ ] public-apis
• [ ] punk-api
• [ ] relational-db
• [ ] rki-covid
• [ ] scaffold-http
• [ ] stock-ticker-api-tutorial

[pnadolny13]

@tayloramurphy just for development purposes I put a placeholder image in there and referenced it from those that you listed so that our build will succeed and I can test installing some of these. We can swap out those images once we find better ones.

[pnadolny13]

I put snapchat-marketing under the existing tap-snapcat-ads.

Still working on:

• file (DONE)

Needs more consideration:

• mailjet-sms - we already have mailjet on the hub, theres airbytes mailjet mail which i listed as an alternative variant. I'm not sure where SMS fits in. Probably a new plugin.
• google-analytics-v4 - We have a GA tap but its not v4. HotGlue also recently said that they ported our meltanolabs version to v4 but its in their gitlab hard fork. We should probably create a whole new v4 plugin.
• sftp-bulk - we have an SFTP tap already and airbyte's SFTP connector, not sure where this fits in.
• mongodb - I added their mongo db as a variant under our existing tap. They also have mongodb_v2, need to figure out what that is especially since it's high on the dockerhub download list.
• all strict encrypt variants

Not Adding:

• google-analytics-data-api - skipping since it links to v4 now so I think its legacy
• newsdata (failed)
• news-api (failed)
• relational-db - The readme says We are not planning to expose this source in the UI yet. It serves as a base upon which we can build all of our other Relational Database-compliant sources.
• file-secure