airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.18k stars 4.14k forks source link

Source Mixpanel: miss 'end_date' field in specs. #12605

Closed vladimir-remar closed 2 years ago

vladimir-remar commented 2 years ago

In source.py the end_date is getting from config map, but is not included in the spec.json, I dont know if is intended or not. https://github.com/airbytehq/airbyte/blob/fced083be5f9a15eb1f5643eb4e725ce0b109dc3/airbyte-integrations/connectors/source-mixpanel/source_mixpanel/source.py#L893

spec.json content

{
  "documentationUrl": "https://docs.airbyte.io/integrations/sources/mixpanel",
  "connectionSpecification": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Source Mixpanel Spec",
    "type": "object",
    "required": ["api_secret"],
    "additionalProperties": true,
    "properties": {
      "api_secret": {
        "title": "Project Token",
        "type": "string",
        "description": "Mixpanel project token. See the <a href=\"https://help.mixpanel.com/hc/en-us/articles/115004502806-Find-Project-Token-\">docs</a> for more information on how to obtain this.",
        "airbyte_secret": true
      },
      "attribution_window": {
        "title": "Attribution Window",
        "type": "integer",
        "description": " A period of time for attributing results to ads and the lookback period after those actions occur during which ad results are counted. Default attribution window is 5 days.",
        "default": 5
      },
      "project_timezone": {
        "title": "Project Timezone",
        "type": "string",
        "description": "Time zone in which integer date times are stored. The project timezone may be found in the project settings in the <a href=\"https://help.mixpanel.com/hc/en-us/articles/115004547203-Manage-Timezones-for-Projects-in-Mixpanel\">Mixpanel console</a>.",
        "default": "US/Pacific",
        "examples": ["US/Pacific", "UTC"]
      },
      "select_properties_by_default": {
        "title": "Select Properties By Default",
        "type": "boolean",
        "description": "Setting this config parameter to TRUE ensures that new properties on events and engage records are captured. Otherwise new properties will be ignored.",
        "default": true
      },
      "start_date": {
        "title": "Start Date",
        "type": "string",
        "description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. If this option is not set, the connector will replicate data from up to one year ago by default.",
        "examples": ["2021-11-16"],
        "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}(T[0-9]{2}:[0-9]{2}:[0-9]{2}Z)?$"
      },
      "region": {
        "title": "Region",
        "description": "The region of mixpanel domain instance either US or EU.",
        "type": "string",
        "enum": ["US", "EU"],
        "default": "US"
      }
    }
  }
}

By adding this it will let to users sync data between time intervals instead a big one sync.

marcosmarxm commented 2 years ago

Closed by https://github.com/airbytehq/airbyte/pull/12801