airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.78k stars 4.04k forks source link

[source-quickbooks] discovered schema is empty #40596

Closed mkrawc closed 2 months ago

mkrawc commented 3 months ago

Connector Name

source-quickbooks

Connector Version

3.0.4-3.0.7

What step the error happened?

Updating the connector

Relevant information

After upgrading quickbooks connector from 3.0.3 to 3.0.4 (and newer) all table schemas are gone and Airbyte tries to remove all fields. I reviewed the source code changes and there doesn't seem to be any changes in the connector code, but just library updates (airbyte_cdk from 0.63 to 1.0).

When I run discover command on the airbyte/source-quickbooks:3.0.4 docker image (and newer) or using poetry locally it returns "json_schema": {} for all streams, but when I run it on airbyte/source-quickbooks:3.0.3 it returns correct schema.

I noticed that upgrading airbyte_cdk to 1.7.1 fixes the schema discovery but it' incompatible with custom connector components. Can somebody look into it?

Relevant log output

✗ poetry run source-quickbooks discover --config secrets/config.json
The currently activated Python version 3.12.3 is not supported by the project (^3.9,<3.12).
Trying to find and use a compatible version. 
Using python3 (3.10.14)
{"type": "CATALOG", "catalog": {"streams": [{"name": "accounts", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "bill_payments", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "bills", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "budgets", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "classes", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "credit_memos", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "customers", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "departments", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "deposits", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "employees", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "estimates", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "invoices", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "items", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "journal_entries", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "payment_methods", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "payments", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "purchase_orders", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "purchases", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "refund_receipts", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "sales_receipts", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "tax_agencies", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "tax_codes", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "tax_rates", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "terms", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "time_activities", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "transfers", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "vendor_credits", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "vendors", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["airbyte_cursor"], "source_defined_primary_key": [["Id"]]}, {"name": "profit_and_loss", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["end_date"], "source_defined_primary_key": [["start_date"], ["end_date"]]}, {"name": "profit_and_loss_rows", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["end_date"], "source_defined_primary_key": [["start_date"], ["end_date"], ["section"], ["account"], ["account_id"]]}, {"name": "balance_sheet", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["end_date"], "source_defined_primary_key": [["start_date"], ["end_date"]]}, {"name": "balance_sheet_rows", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["end_date"], "source_defined_primary_key": [["start_date"], ["end_date"], ["section"], ["account"], ["account_id"]]}, {"name": "trial_balance", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["end_date"], "source_defined_primary_key": [["start_date"], ["end_date"]]}, {"name": "trial_balance_rows", "json_schema": {}, "supported_sync_modes": ["full_refresh", "incremental"], "source_defined_cursor": true, "default_cursor_field": ["end_date"], "source_defined_primary_key": [["start_date"], ["end_date"], ["section"], ["account"], ["account_id"]]}]}}

Contribute

Loky1s commented 3 months ago

Quit taking my money if I ever c u it's over

btkcodedev commented 3 months ago

Quit taking my money if I ever c u it's over

For conversations: Please follow the standards mentioned in Our Standards section in code of conduct Thanks

Loky1s commented 3 months ago

Like your r doing stealing