Mob Session: Hub Definition Updates

[aaronsteers]

Tacking issue for Oct 13 mob session re: plugin definition updates.

PR: https://github.com/meltano/hub/pull/895 Spreadsheet: https://docs.google.com/spreadsheets/d/1PdM1Mp-PMF6V1r1gbI7d1qcJzGPtbqgmCOMjbdj4drI/edit#gid=1111558337

[aaronsteers]

/test-plugin name=tap-github

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-github --help usage: tap-github [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[aaronsteers]

/test-plugin name=tap-github pip-url=git+https://github.com/meltanolabs/tap-github.git@main

Note: github collapses the text https://github.com/ when rendering the above. The actual full command is:

/test-plugin name=tap-github pip-url=git+https://github.com/meltanolabs/tap-github.git@main

Starting test job... ... Job completed.

Auto-generated README.md

# `tap-github` GitHub tap class. Built with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets. ## Capabilities * `catalog` * `state` * `discover` * `about` * `stream-maps` * `schema-flattening` ## Settings | Setting | Required | Default | Description | |:----------------------|:--------:|:-------:|:------------| | user_agent | False | None | | | metrics_log_level | False | None | | | auth_token | False | None | GitHub token to authenticate with. | | additional_auth_tokens| False | None | List of GitHub tokens to authenticate with. Streams will loop through them when hitting rate limits. | | rate_limit_buffer | False | None | Add a buffer to avoid consuming all query points for the token at hand. Defaults to 1000. | | searches | False | None | | | organizations | False | None | | | repositories | False | None | | | user_usernames | False | None | | | user_ids | False | None | | | start_date | False | None | | | stream_maps | False | None | | | stream_map_config | False | None | | | skip_parent_streams | False | None | Set to true to skip API calls for the parent streams (such as repositories) if it is not selected but children are | | flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. | | flattening_max_depth | False | None | The max depth to flatten schemas. | A full list of supported settings and capabilities is available by running: `tap-github --about`

Version info

``` tap-github v1.1.1, Meltano SDK v0.11.1 ```

Usage info

``` melty-bot % tap-github --help Usage: tap-github [OPTIONS] Execute the Singer tap. Options: --state PATH Use a bookmarks file for incremental replication. --catalog PATH Use a Singer catalog file with the tap. --test TEXT Use --test to sync a single record for each stream. Use --test=schema to test schema output without syncing records. --discover Run the tap in discovery mode. --config TEXT Configuration file location or 'ENV' to use environment variables. --format [json|markdown] Specify output style for --about --about Display package metadata and settings. --version Display the package version. --help Show this message and exit. ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ✅ 'about'

JSON Metadata

```json { "name": "tap-github", "description": "GitHub tap class.", "version": "1.1.1", "sdk_version": "0.11.1", "capabilities": [ "catalog", "state", "discover", "about", "stream-maps", "schema-flattening" ], "settings": { "type": "object", "properties": { "user_agent": { "type": [ "string", "null" ] }, "metrics_log_level": { "type": [ "string", "null" ] }, "auth_token": { "type": [ "string", "null" ], "description": "GitHub token to authenticate with." }, "additional_auth_tokens": { "type": [ "array", "null" ], "items": { "type": [ "string" ] }, "description": "List of GitHub tokens to authenticate with. Streams will loop through them when hitting rate limits." }, "rate_limit_buffer": { "type": [ "integer", "null" ], "description": "Add a buffer to avoid consuming all query points for the token at hand. Defaults to 1000." }, "searches": { "type": [ "array", "null" ], "items": { "type": "object", "properties": { "name": { "type": [ "string" ] }, "query": { "type": [ "string" ] } }, "required": [ "name", "query" ] } }, "organizations": { "type": [ "array", "null" ], "items": { "type": [ "string" ] } }, "repositories": { "type": [ "array", "null" ], "items": { "type": [ "string" ] } }, "user_usernames": { "type": [ "array", "null" ], "items": { "type": [ "string" ] } }, "user_ids": { "type": [ "array", "null" ], "items": { "type": [ "string" ] } }, "start_date": { "type": [ "string", "null" ], "format": "date-time" }, "stream_maps": { "type": [ "object", "null" ], "properties": {} }, "stream_map_config": { "type": [ "object", "null" ], "properties": {} }, "skip_parent_streams": { "type": [ "boolean", "null" ], "description": "Set to true to skip API calls for the parent streams (such as repositories) if it is not selected but children are" }, "flattening_enabled": { "type": [ "boolean", "null" ], "description": "'True' to enable schema flattening and automatically expand nested properties." }, "flattening_max_depth": { "type": [ "integer", "null" ], "description": "The max depth to flatten schemas." } } } } ```

Discovered streams

- `anonymous_contributors` - `assignees` - `collaborators` - `commit_comments` - `commits` - `community_profile` - `contributors` - `dependencies` - `dependents` - `events` - `issue_comments` - `issue_events` - `issues` - `languages` - `milestones` - `organizations` - `project_cards` - `project_columns` - `projects` - `pull_request_commits` - `pull_requests` - `readme` - `readme_html` - `releases` - `repositories` - `review_comments` - `reviews` - `stargazers` - `stargazers_rest` - `starred` - `stats_contributors` - `team_members` - `team_roles` - `teams` - `user_contributed_to` - `users` - `workflow_run_jobs` - `workflow_runs` - `workflows`

[aaronsteers]

/test-plugin name=notathingithink pip-url=thisisnotright

Starting test job... ... Job completed.

melty-bot % pipx install thisisnotright creating virtual environment... installing thisisnotright... Fatal error from pip prevented installation. Full pip output in file: /opt/pipx/logs/cmd_2022-10-13_15.44.47_pip_errors.log

Some possibly relevant errors from pip install: ERROR: Could not find a version that satisfies the requirement thisisnotright (from versions: none) ERROR: No matching distribution found for thisisnotright Error installing thisisnotright.

[aaronsteers]

/test-plugin name=target-jsonl pip-url=target-jsonl

Starting test job... ... Job completed.

Usage info

``` melty-bot % target-jsonl --help usage: target-jsonl [-h] [-c CONFIG] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file ```

Detected capabilities

> - ❌ 'catalog' > - ❌ 'properties' > - ❌ 'state' > - ❌ 'about'

[tayloramurphy]

/test-plugin name=target-yaml pip-url=git+https://github.com/MeltanoLabs/target-yaml.git

Starting test job... ... Job completed.

Auto-generated README.md

# `target-yaml` A Singer target that generates YAML files. Built with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets. ## Capabilities * `about` * `stream-maps` ## Settings | Setting | Required | Default | Description | |:-------------------------|:--------:|:-------:|:------------| | file_naming_scheme | False | None | The scheme with which output files will be named. Naming scheme may leverage any of the following substitutions:

- `{stream_name}`- `{datestamp}`- `{timestamp}` | | datestamp_format | False | %Y-%m-%d | A python format string to use when outputting the `{datestamp}` string. For reference, see: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes | | timestamp_format | False | %Y-%m-%d.T%H%M%S | A python format string to use when outputting the `{timestamp}` string. For reference, see: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes | | timestamp_timezone | False | UTC | The timezone code or name to use when generating `{timestamp}` and `{datestamp}`. Defaults to 'UTC'. For a list of possible values, please see: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones | | stream_maps | False | None | Allows inline stream transformations and aliasing. For more information see: https://sdk.meltano.com/en/latest/stream_maps.html | | record_insert_jsonpath | False | $.metrics | A jsonpath string determining the insertion point for new records. Currently, this must be the path to a map key which will be populated by a list of records.

For example '$.metrics' will populate the file with `metrics: [{},{},...]`

For JSONPath syntax help, see: https://jsonpath.com | | record_key_property_name | False | None | A property in the record which will be used as the dictionary key.

If this property is provided, records will be written as key-value objects; if omitted, records are written as a list. | | record_sort_property_name| False | None | A property in the record which will be used as a sort key.

If this property is omitted, records will not be sorted. | | overwrite_behavior | False | replace_records | Determines the overwrite behavior if destination file already exists. Must be one of the following string values:

- `append_records` (default) - append records at the insertion point
- `replace_records` - replace all records at the insertion point
- `replace_file` - replace entire file using `default_yaml_template`
| | default_yaml_template | False | None | Text string to use for a yaml template file. This text will be used to create a new file if the destination file does not exist. | A full list of supported settings and capabilities is available by running: `target-yaml --about`

Version info

``` target-yaml v0.0.1, Meltano SDK v0.3.18) ```

Usage info

``` melty-bot % target-yaml --help Usage: target-yaml [OPTIONS] Execute the Singer target. Options: --input FILENAME A path to read messages from instead of from standard in. --config TEXT Configuration file location or 'ENV' to use environment variables. --format [json|markdown] Specify output style for --about --about Display package metadata and settings. --version Display the package version. --help Show this message and exit. ```

Detected capabilities

> - ❌ 'catalog' > - ❌ 'properties' > - ❌ 'state' > - ✅ 'about'

JSON Metadata

```json { "name": "target-yaml", "description": "A Singer target that generates YAML files.", "version": "0.0.1", "sdk_version": "0.3.18", "capabilities": [ "about", "stream-maps" ], "settings": { "type": "object", "properties": { "file_naming_scheme": { "type": [ "string", "null" ], "description": "The scheme with which output files will be named. Naming scheme may leverage any of the following substitutions: \n\n- `{stream_name}`- `{datestamp}`- `{timestamp}`" }, "datestamp_format": { "type": [ "string", "null" ], "default": "%Y-%m-%d", "description": "A python format string to use when outputting the `{datestamp}` string. For reference, see: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes" }, "timestamp_format": { "type": [ "string", "null" ], "default": "%Y-%m-%d.T%H%M%S", "description": "A python format string to use when outputting the `{timestamp}` string. For reference, see: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes" }, "timestamp_timezone": { "type": [ "string", "null" ], "default": "UTC", "description": "The timezone code or name to use when generating `{timestamp}` and `{datestamp}`. Defaults to 'UTC'. For a list of possible values, please see: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones" }, "stream_maps": { "type": [ "object", "null" ], "properties": {}, "description": "Allows inline stream transformations and aliasing. For more information see: https://sdk.meltano.com/en/latest/stream_maps.html" }, "record_insert_jsonpath": { "type": [ "string", "null" ], "default": "$.metrics", "description": "A jsonpath string determining the insertion point for new records. Currently, this must be the path to a map key which will be populated by a list of records. \n\nFor example '$.metrics' will populate the file with `metrics: [{},{},...]` \n\nFor JSONPath syntax help, see: https://jsonpath.com" }, "record_key_property_name": { "type": [ "string", "null" ], "description": "A property in the record which will be used as the dictionary key.\n\nIf this property is provided, records will be written as key-value objects; if omitted, records are written as a list." }, "record_sort_property_name": { "type": [ "string", "null" ], "description": "A property in the record which will be used as a sort key.\n\nIf this property is omitted, records will not be sorted." }, "overwrite_behavior": { "type": [ "string", "null" ], "default": "replace_records", "description": "Determines the overwrite behavior if destination file already exists. Must be one of the following string values: \n\n- `append_records` (default) - append records at the insertion point\n- `replace_records` - replace all records at the insertion point\n- `replace_file` - replace entire file using `default_yaml_template`\n" }, "default_yaml_template": { "type": [ "string", "null" ], "description": "Text string to use for a yaml template file. This text will be used to create a new file if the destination file does not exist." } } } } ```

[WillDaSilva]

/test-plugin name=target-postgres pip-url=pipelinewise-target-postgres

Starting test job... ... Job completed.

Usage info

``` melty-bot % target-postgres --help usage: target-postgres [-h] [-c CONFIG] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file ```

Detected capabilities

> - ❌ 'catalog' > - ❌ 'properties' > - ❌ 'state' > - ❌ 'about'

[pandemicsyn]

/test-plugin name=target-sqlite pip-url=git+https://github.com/MeltanoLabs/target-sqlite.git

Starting test job... ... Job completed.

Usage info

``` melty-bot % target-sqlite --help usage: target-sqlite [-h] -c CONFIG optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file ```

Detected capabilities

> - ❌ 'catalog' > - ❌ 'properties' > - ❌ 'state' > - ❌ 'about'

[aaronsteers]

/test-plugin name=tap-carbon-intensity pip-url=git+https://gitlab.com/meltano/tap-carbon-intensity.git

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-carbon-intensity --help usage: tap-carbon-intensity [-h] [-c CONFIG] [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

Discovered streams

- `region` - `entry` - `generationmix`

[tayloramurphy]

/test-plugin name=tap-zendesk pip-url=twilio-tap-zendesk

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-zendesk --help usage: tap-zendesk [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[WillDaSilva]

/test-plugin name=target-snowflake pip-url=pipelinewise-target-snowflake

Starting test job... ... Job completed.

Usage info

``` melty-bot % target-snowflake --help usage: target-snowflake [-h] [-c CONFIG] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file ```

Detected capabilities

> - ❌ 'catalog' > - ❌ 'properties' > - ❌ 'state' > - ❌ 'about'

[pandemicsyn]

/test-plugin name=target-csv pip-url=git+https://github.com/hotgluexyz/target-csv.git

Starting test job... ... Job completed.

Usage info

``` melty-bot % target-csv --help usage: target-csv [-h] [-c CONFIG] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file ```

Detected capabilities

> - ❌ 'catalog' > - ❌ 'properties' > - ❌ 'state' > - ❌ 'about'

[WillDaSilva]

/test-plugin name=tap-mysql pip-url=pipelinewise-tap-mysql

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-mysql --help usage: tap-mysql [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[edgarrmondragon]

/test-plugin name=tap-spreadsheets-anywhere pip-url=git+https://github.com/ets/tap-spreadsheets-anywhere.git

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-spreadsheets-anywhere --help usage: tap-spreadsheets-anywhere [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[pandemicsyn]

/test-plugin name=tap-csv pip-url=git+https://github.com/MeltanoLabs/tap-csv.git

Starting test job... ... Job completed.

Auto-generated README.md

# `tap-csv` CSV tap class. Built with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets. ## Capabilities * `sync` * `catalog` * `discover` ## Settings | Setting | Required | Default | Description | |:--------------------|:--------:|:-------:|:------------| | files | False | None | An array of csv file stream settings. | | csv_files_definition| False | None | A path to the JSON file holding an array of file settings. | A full list of supported settings and capabilities is available by running: `tap-csv --about`

Version info

``` tap-csv v0.0.8, Meltano SDK v0.11.1 ```

Usage info

``` melty-bot % tap-csv --help Usage: tap-csv [OPTIONS] Execute the Singer tap. Options: --state PATH Use a bookmarks file for incremental replication. --catalog PATH Use a Singer catalog file with the tap. --test TEXT Use --test to sync a single record for each stream. Use --test=schema to test schema output without syncing records. --discover Run the tap in discovery mode. --config TEXT Configuration file location or 'ENV' to use environment variables. --format [json|markdown] Specify output style for --about --about Display package metadata and settings. --version Display the package version. --help Show this message and exit. ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ✅ 'about'

JSON Metadata

```json { "name": "tap-csv", "description": "CSV tap class.", "version": "0.0.8", "sdk_version": "0.11.1", "capabilities": [ "sync", "catalog", "discover" ], "settings": { "type": "object", "properties": { "files": { "type": [ "array", "null" ], "items": { "type": "object", "properties": { "entity": { "type": [ "string" ] }, "path": { "type": [ "string" ] }, "keys": { "type": "array", "items": { "type": [ "string" ] } }, "encoding": { "type": [ "string", "null" ] }, "delimiter": { "type": [ "string", "null" ] }, "doublequote": { "type": [ "boolean", "null" ] }, "escapechar": { "type": [ "string", "null" ] }, "quotechar": { "type": [ "string", "null" ] }, "skipinitialspace": { "type": [ "boolean", "null" ] }, "strict": { "type": [ "boolean", "null" ] } }, "required": [ "entity", "path", "keys" ] }, "description": "An array of csv file stream settings." }, "csv_files_definition": { "type": [ "string", "null" ], "description": "A path to the JSON file holding an array of file settings." } } } } ```

[pandemicsyn]

/test-plugin name=tap-github pip-url=tap-github

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-github --help usage: tap-github [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[tayloramurphy]

/test-plugin name=tap-mongodb pip-url=tap-mongodb

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-mongodb --help usage: tap-mongodb [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[edgarrmondragon]

/test-plugin name=tap-google-analytics pip-url=git+https://github.com/MeltanoLabs/tap-google-analytics.git

Starting test job... ... Job completed.

Auto-generated README.md

# `tap-google-analytics` GoogleAnalytics tap class. Built with the [Meltano SDK](https://sdk.meltano.com) for Singer Taps and Targets. ## Capabilities * `catalog` * `state` * `discover` * `about` * `stream-maps` * `schema-flattening` ## Settings | Setting | Required | Default | Description | |:--------------------|:--------:|:-------:|:------------| | start_date | True | None | The earliest record date to sync | | view_id | True | None | Google Analytics View ID | | client_secrets | False | None | Google Analytics Client Secrets Dictionary | | key_file_location | False | None | File Path to Google Analytics Client Secrets | | oauth_credentials | False | None | Google Analytics OAuth Credentials | | reports | False | None | Google Analytics Reports Definition | | end_date | False | None | The last record date to sync | | stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). | | stream_map_config | False | None | User-defined config values to be used within map expressions. | | flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. | | flattening_max_depth| False | None | The max depth to flatten schemas. | A full list of supported settings and capabilities is available by running: `tap-google-analytics --about`

Version info

``` tap-google-analytics v0.0.5, Meltano SDK v0.11.1 ```

Usage info

``` melty-bot % tap-google-analytics --help Usage: tap-google-analytics [OPTIONS] Execute the Singer tap. Options: --state PATH Use a bookmarks file for incremental replication. --catalog PATH Use a Singer catalog file with the tap. --test TEXT Use --test to sync a single record for each stream. Use --test=schema to test schema output without syncing records. --discover Run the tap in discovery mode. --config TEXT Configuration file location or 'ENV' to use environment variables. --format [json|markdown] Specify output style for --about --about Display package metadata and settings. --version Display the package version. --help Show this message and exit. ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ✅ 'about'

JSON Metadata

```json { "name": "tap-google-analytics", "description": "GoogleAnalytics tap class.", "version": "0.0.5", "sdk_version": "0.11.1", "capabilities": [ "catalog", "state", "discover", "about", "stream-maps", "schema-flattening" ], "settings": { "type": "object", "properties": { "start_date": { "type": [ "string" ], "format": "date-time", "description": "The earliest record date to sync" }, "view_id": { "type": [ "string" ], "description": "Google Analytics View ID" }, "client_secrets": { "type": [ "object", "null" ], "properties": {}, "description": "Google Analytics Client Secrets Dictionary" }, "key_file_location": { "type": [ "string", "null" ], "description": "File Path to Google Analytics Client Secrets" }, "oauth_credentials": { "type": [ "object", "null" ], "properties": { "access_token": { "type": [ "string", "null" ], "description": "Google Analytics Access Token" }, "refresh_token": { "type": [ "string", "null" ], "description": "Google Analytics Refresh Token" }, "client_id": { "type": [ "string", "null" ], "description": "Google Analytics Client ID" }, "client_secret": { "type": [ "string", "null" ], "description": "Google Analytics Client Secret" } }, "description": "Google Analytics OAuth Credentials" }, "reports": { "type": [ "string", "null" ], "description": "Google Analytics Reports Definition" }, "end_date": { "type": [ "string", "null" ], "description": "The last record date to sync" }, "stream_maps": { "type": [ "object", "null" ], "properties": {}, "description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)." }, "stream_map_config": { "type": [ "object", "null" ], "properties": {}, "description": "User-defined config values to be used within map expressions." }, "flattening_enabled": { "type": [ "boolean", "null" ], "description": "'True' to enable schema flattening and automatically expand nested properties." }, "flattening_max_depth": { "type": [ "integer", "null" ], "description": "The max depth to flatten schemas." } }, "required": [ "start_date", "view_id" ] } } ```

[edgarrmondragon]

/test-plugin name=tap-hubspot pip-url=git+https://github.com/singer-io/tap-hubspot.git

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-hubspot --help usage: tap-hubspot [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[edgarrmondragon]

/test-plugin name=tap-airtable pip-url=git+https://github.com/goes-funky/tap-airtable.git

Starting test job... ... Job completed.

Usage info

``` melty-bot % tap-airtable --help usage: tap-airtable [-h] -c CONFIG [-s STATE] [-p PROPERTIES] [--catalog CATALOG] [-d] optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG Config file -s STATE, --state STATE State file -p PROPERTIES, --properties PROPERTIES Property selections: DEPRECATED, Please use --catalog instead --catalog CATALOG Catalog file -d, --discover Do schema discovery ```

Detected capabilities

> - ✅ 'catalog' > - ✅ 'state' > - ❌ 'about'

[edgarrmondragon]

/test-plugin name=tap-airtable pip-url=git+goes-funky/tap-airtable.git

Starting test job... ... Job completed.

Usage info Detected capabilities

• ✅ 'catalog'
• ✅ 'state'
• ❌ 'about'

@aaronsteers this one's a false positive for the state capability: https://github.com/goes-funky/tap-airtable/blob/46b9b3e010abdc4bde84e38a1fdf4dd0411ee121/tap_airtable/__init__.py#L11-L17

[aaronsteers]

@edgarrmondragon - thanks for raising. It looks like our heuristic is fine (screenshot below) but perhaps a case where the developer perhaps started as a copy of another repo and never removed the --state flag from the CLI.

[aaronsteers]

Correction:

It looks like this plugin is leveraging a generic function from singer.util, which itself does not take any flags or inputs about specific expected capabilities:

The relevant singer library code:

https://github.com/transferwise/pipelinewise-singer-python/blob/master/singer/utils.py#L149-L151

The tap-airbyte code which references it:

https://github.com/goes-funky/tap-airtable/blob/46b9b3e010abdc4bde84e38a1fdf4dd0411ee121/tap_airtable/__init__.py#L11-L12

From this observation, perhaps we cannot infer as much from the --help text as previously thought.

[edgarrmondragon]

@aaronsteers I guess that's an issue even for SDK-based taps. There may be some sources that don't really support state, but it will still declare the state capability and show it in the --help output.

[aaronsteers]

@edgarrmondragon - Thanks for calling this out. You prompted me to do an analysis of capabilities here.

I didn't see any big risks or issues in that analysis. What I realized when writing that was that, even if none of a tap's streams support incremental, there may still be value in accepting a state file via --state. State files also can contain elements such as 'last sync time' - and while not adding a ton of value, I don't see a strong reason to reject the file or fail the process if --state is passed to the tap.

What do you think?