search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.36k stars 4.16k forks source link

Mark datefields in Beta/GA connectors properly as dates #20204

Closed timroes closed 1 year ago

timroes commented 1 year ago

Since we merged https://github.com/airbytehq/airbyte/pull/19678 which enables the date picker for all date fields, we should make sure that all connector specs properly mark date fields with format: date or format: date-time in their spec. We noticed a lot of connectors only attach a pattern to it, but not the format key. We should to a spike to update all specs to correct those, and make sure all date fields can therefore properly use the datepicker.

evantahler commented 1 year ago

Here's a quick check of the list of connectors that need to be fixed:

const yaml = require('js-yaml'); // npm install -g js-yaml
const fs   = require('fs');
const FILE = '/Users/evan/workspace/airbyte/airbyte/airbyte-config/init/src/main/resources/seed/source_specs.yaml'

const doc = yaml.load(fs.readFileSync(FILE, 'utf8'));
let connector_count = 0

for (const connector of doc) {
  let problems = [];
  for (const [key, value] of Object.entries(connector.spec.connectionSpecification.properties)) {
    if (!value.format && value.pattern && value.pattern.startsWith('^[0-9]{4}-')) {
      problems.push(key)
    }
  }
  if (problems.length > 0) {
    connector_count++;
    console.log("")
    console.log(`### Connector: ${connector.dockerImage}`)
    for (p of problems) {
      console.log(` * ${p}`)
    }
  }
}

console.log("---")
console.log(`${connector_count} total connectors with problems`)

Connector: airbyte/source-aws-cloudtrail:0.1.4

Connector: airbyte/source-amazon-seller-partner:0.2.28

Connector: airbyte/source-amplitude:0.1.18

Connector: airbyte/source-appsflyer:0.1.0

Connector: airbyte/source-appstore-singer:0.2.6

Connector: airbyte/source-ashby:0.1.0

Connector: airbyte/source-bigcommerce:0.1.7

Connector: airbyte/source-callrail:0.1.0

Connector: airbyte/source-cart:0.2.0

Connector: airbyte/source-chargebee:0.1.16

Connector: airbyte/source-chartmogul:0.1.1

Connector: airbyte/source-close-com:0.2.0

Connector: airbyte/source-coin-api:0.1.0

Connector: airbyte/source-commercetools:0.1.0

Connector: airbyte/source-datadog:0.1.0

Connector: airbyte/source-delighted:0.1.4

Connector: airbyte/source-dixa:0.1.3

Connector: airbyte/source-dv-360:0.1.0

Connector: airbyte/source-exchange-rates:1.2.7

Connector: airbyte/source-facebook-marketing:0.2.78

Connector: airbyte/source-freshservice:0.1.1

Connector: airbyte/source-genesys:0.1.0

Connector: airbyte/source-github:0.3.8

Connector: airbyte/source-gitlab:0.1.8

Connector: airbyte/source-gnews:0.1.0

Connector: airbyte/source-gocardless:0.1.0

Connector: airbyte/source-gong:0.1.0

Connector: airbyte/source-google-ads:0.2.5

Connector: airbyte/source-google-search-console:0.1.18

Connector: airbyte/source-harvest:0.1.11

Connector: airbyte/source-hubspot:0.2.3

Connector: airbyte/source-insightly:0.1.0

Connector: airbyte/source-intercom:0.1.29

Connector: airbyte/source-iterable:0.1.22

Connector: airbyte/source-jira:0.3.0

Connector: airbyte/source-klaviyo:0.1.10

Connector: airbyte/source-lever-hiring:0.1.3

Connector: airbyte/source-linkedin-ads:0.1.12

Connector: airbyte/source-mailgun:0.1.0

Connector: airbyte/source-marketo:0.1.11

Connector: airbyte/source-mixpanel:0.1.29

Connector: airbyte/source-my-hours:0.1.1

Connector: airbyte/source-nasa:0.1.0

Connector: airbyte/source-netsuite:0.1.1

Connector: airbyte/source-news-api:0.1.0

Connector: airbyte/source-notion:0.1.10

Connector: airbyte/source-nytimes:0.1.0

Connector: airbyte/source-okta:0.1.13

Connector: airbyte/source-onesignal:0.1.2

Connector: airbyte/source-orb:0.1.4

Connector: airbyte/source-orbit:0.1.1

Connector: airbyte/source-outreach:0.1.2

Connector: airbyte/source-pardot:0.1.0

Connector: airbyte/source-partnerstack:0.1.0

Connector: airbyte/source-paypal-transaction:0.1.10

Connector: airbyte/source-paystack:0.1.1

Connector: airbyte/source-pipedrive:0.1.13

Connector: airbyte/source-plaid:0.3.2

Connector: airbyte/source-plausible:0.1.0

Connector: airbyte/source-polygon-stock-api:0.1.0

Connector: airbyte/source-posthog:0.1.7

Connector: airbyte/source-prestashop:0.3.0

Connector: airbyte/source-qonto:0.1.0

Connector: airbyte/source-qualaroo:0.1.2

Connector: airbyte/source-quickbooks-singer:0.1.5

Connector: airbyte/source-recharge:0.2.4

Connector: airbyte/source-recurly:0.4.1

Connector: airbyte/source-rd-station-marketing:0.1.1

Connector: airbyte/source-salesloft:0.1.3

Connector: airbyte/source-salesforce:1.0.27

Connector: airbyte/source-senseforce:0.1.0

Connector: airbyte/source-shopify:0.3.0

Connector: airbyte/source-slack:0.1.19

Connector: airbyte/source-snapchat-marketing:0.1.8

Connector: airbyte/source-sonar-cloud:0.1.0

Connector: airbyte/source-square:0.1.4

Connector: airbyte/source-stripe:0.1.40

Connector: airbyte/source-surveymonkey:0.1.13

Connector: airbyte/source-talkdesk-explore:0.1.0

Connector: airbyte/source-tiktok-marketing:0.1.17

Connector: airbyte/source-timely:0.1.0

Connector: airbyte/source-toggl:0.1.0

Connector: airbyte/source-trello:0.1.6

Connector: airbyte/source-tvmaze-schedule:0.1.0

Connector: airbyte/source-twilio:0.1.14

Connector: airbyte/source-typeform:0.1.9

Connector: airbyte/source-xero:0.1.0

Connector: airbyte/source-woocommerce:0.1.2

Connector: airbyte/source-wrike:0.1.0

Connector: airbyte/source-zendesk-chat:0.1.11

Connector: airbyte/source-zendesk-sunshine:0.1.1

Connector: airbyte/source-zendesk-support:0.2.18

Connector: airbyte/source-zendesk-talk:0.1.5

Connector: airbyte/source-zuora:0.1.3

Connector: airbyte/source-waiteraid:0.1.0

Connector: airbyte/source-yandex-metrica:0.1.0

YowanR commented 1 year ago

@lazebnyi @YuliiaNahoha Can you take a look at this work and estimate how much work this would be? If we have the cycles, we could try to get this done this quarter but happy to discuss further.

bleonard commented 1 year ago

@YowanR coordinating here. Scanning the list, I don't immediately notice any that are databases/java. If there are any and you can't handle them, can you make a new issue and tag it to our team.

timroes commented 1 year ago

Updated the list above in the comment, to include 3 connectors that were missing from the list, that also had date fields who were not marked correctly (since some connectors use a pattern like ^$|^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$).

evantahler commented 1 year ago

@YowanR is this on your radar? If so, I'll remove this from the Connector Ops backlog

YowanR commented 1 year ago

@evantahler yes it is - GL is already working on it. Feel free to remove it from your backlog.