airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.08k stars 4.12k forks source link

Postgres source shows tables in logs but not in UI #29431

Open joseph-maddox opened 1 year ago

joseph-maddox commented 1 year ago

Connector Name

source-postgres

Connector Version

3.1.3

What step the error happened?

Configuring a new connector

Revelant information

All source checks pass (in the source config page). The logs show that tables are discovered (see below) but the UI does not show any tables (preventing connection setup from proceeding).

Relevant log output

{
    "catalog": {
        "streams": []
    },
    "jobInfo": {
        "id": "ddd2b6d3-2058-49f2-8c5c-9b2f47d3e224",
        "configType": "discover_schema",
        "configId": "Optional[decd338e-5647-4c0b-adf4-da0e75f5a750]",
        "createdAt": 1692108818850,
        "endedAt": 1692108823150,
        "succeeded": true,
        "connectorConfigurationUpdated": false,
        "logs": {
            "logLines": [
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/ddd2b6d3-2058-49f2-8c5c-9b2f47d3e224/0/logs.log",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.50.16",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.a.c.AirbyteApiClient(retryWithJitterThrows):251 - Attempt 0 to save workflow id for cancellation",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''",
                "2023-08-15 14:13:38 \u001B[32mINFO\u001B[m i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-postgres:3.1.3 exists...",
                "2023-08-15 14:13:39 \u001B[32mINFO\u001B[m i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-postgres:3.1.3 was found locally.",
                "2023-08-15 14:13:39 \u001B[32mINFO\u001B[m i.a.w.p.DockerProcessFactory(create):143 - Creating docker container = source-postgres-discover-ddd2b6d3-2058-49f2-8c5c-9b2f47d3e224-0-xrhyc with resources io.airbyte.config.ResourceRequirements@7dbd9ae4[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null",
                "2023-08-15 14:13:39 \u001B[32mINFO\u001B[m i.a.w.p.DockerProcessFactory(create):196 - Preparing command: docker run --rm --init -i -w /data/ddd2b6d3-2058-49f2-8c5c-9b2f47d3e224/0 --log-driver none --name source-postgres-discover-ddd2b6d3-2058-49f2-8c5c-9b2f47d3e224-0-xrhyc --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-postgres:3.1.3 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.16 -e WORKER_JOB_ID=ddd2b6d3-2058-49f2-8c5c-9b2f47d3e224 airbyte/source-postgres:3.1.3 discover --config source_config.json",
                "2023-08-15 14:13:39 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(create):177 - Reading messages from protocol version 0.2.0",
                "2023-08-15 14:13:40 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.p.PostgresSource(main):636 starting source: class io.airbyte.integrations.source.postgres.PostgresSource",
                "2023-08-15 14:13:40 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {discover=null, config=source_config.json}",
                "2023-08-15 14:13:40 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):130 Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource",
                "2023-08-15 14:13:40 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):131 Command: DISCOVER",
                "2023-08-15 14:13:40 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):132 Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword always_show - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword min - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword max - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2023-08-15 14:13:40 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.s.SshTunnel(getInstance):204 Starting connection with method: SSH_KEY_AUTH",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO o.a.s.c.u.s.AbstractSecurityProviderRegistrar(getOrCreateProvider):112 getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO o.a.s.c.i.DefaultIoServiceFactoryFactory(getIoServiceProvider):71 Using MinaServiceFactoryFactory",
                "2023-08-15 14:13:41 \u001B[33mWARN\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - WARN o.a.s.c.k.StaticServerKeyVerifier(handleAcceptance):59 Server at /34.136.85.141:22 presented unverified EC key: SHA256:lSUYtVqfptYepl8h78thFCJjCBjkq7NEShl0pplTs9E",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.s.SshTunnel(openTunnel):367 Established tunneling session to 127.0.0.1:5434. Port forwarding started on /127.0.0.1:41813 ",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.p.PostgresSource(toSslJdbcParamInternal):673 DISABLED toSslJdbcParam disable",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO c.z.h.HikariDataSource(<init>):80 HikariPool-1 - Starting...",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO c.z.h.HikariDataSource(<init>):82 HikariPool-1 - Start completed.",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.p.PostgresUtils(isCdc):61 using CDC: true",
                "2023-08-15 14:13:41 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.d.j.s.AdaptiveStreamingQueryConfig(initialize):31 Set initial fetch size: 10 rows",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.d.j.s.AdaptiveStreamingQueryConfig(accept):40 Set new fetch size: 1197005 rows",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.d.j.s.TwoStageSizeEstimator(getTargetBufferByteSize):71 Max memory limit: 12591300608, JDBC buffer size: 7554780365",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.d.j.s.AdaptiveStreamingQueryConfig(accept):40 Set new fetch size: 32704677 rows",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.p.PostgresCatalogHelper(getPublicizedTables):122 For CDC, only tables in publication alltables will be included in the sync: [public.ir_act_window_view, public.crm_iap_lead_mining_request, public.project_task_type_project_task_type_delete_wizard_rel, public.ir_act_server_group_rel, public.account_move_line_l10n_us_1099_wizard_rel, public.stock_inventory_adjustment_name, public.account_fiscal_year, public.account_invoice_send, public.crm_team_member, public.ir_model_constraint, public.sms_resend, public.crm_lead_convert2ticket, public.website_snippet_filter, public.choose_delivery_package, public.shopify_product_data_queue_line_ept, public.documents_account_folder_setting_documents_tag_rel, public.queue_job_queue_jobs_to_cancelled_rel, public.res_country_state, public.project_create_sale_order_line, public.dms_acc_rep_export_wizard_format_rel, public.res_bank, public.mail_shortcode, public.mrp_bom_byproduct_product_template_attribute_value_rel, public.ramp_transactions, public.rma_delivery_wizard, public.muk_webhooks_logging, public.portal_wizard_user, public.res_country_group, public.expiry_picking_confirmation, public.account_automatic_entry_wizard_account_move_line_rel, public.purchase_order_stock_picking_rel, public.account_reconcile_model, public.stock_orderpoint_snooze, public.account_account_tag_account_tax_repartition_line_rel, public.calendar_event_res_partner_rel, public.repair_tags, public.timer_timer, public.stock_immediate_transfer_line, public.sale_order_template, public.account_analytic_applicability, public.account_tour_upload_bill_ir_attachments_rel, public.auth_oauth_provider, public.account_journal, public.mail_compose_message_ir_attachments_rel, public.sale_advance_payment_inv_sale_order_rel, public.crossovered_budget, public.calendar_event, public.account_report_horizontal_group_rule, public.account_budget_rel, public.res_groups_spreadsheet_dashboard_rel, public.account_tax, public.hr_employee, public.slide_channel_partner, public.ir_model_relation, public.quality_point_test_type, public.account_reconcile_model_template_res_partner_category_rel, public.documents_request_wizard, public.account_tax_repartition_line, public.account_analytic_account, public.helpdesk_sla_helpdesk_tag_rel, public.slide_answer, public.sms_sms, public.shopify_instance_order_status_rel, public.stock_return_picking_line, public.account_report_external_value, public.stock_track_line, public.rma, public.slide_channel_tag, public.hr_plan, public.purchase_tag, public.cash_flow_period_setting, public.hr_employee_category, public.base_module_install_review, public.stock_orderpoint_snooze_stock_warehouse_orderpoint_rel, public.account_move_line_account_tax_rel, public.res_partner_category, public.account_reconcile_model_line_account_tax_rel, public.stock_quant_package, public.slide_question, public.account_move_account_invoice_send_rel, public.sale_payment_provider_onboarding_wizard, public.shopify_onboarding_confirmation_ept, public.stock_valuation_adjustment_lines, public.gamification_karma_tracking, public.repair_line, public.account_tax_sale_advance_payment_inv_rel, public.snailmail_letter, public.gamification_challenge_users_rel, public.helpdesk_ticket_to_lead, public.account_report_horizontal_group, public.stock_quantity_history, public.project_task_stop_timers_wizard, public.stock_valuation_layer, public.account_link_journal, public.stock_package_type_stock_putaway_rule_rel, public.print_prenumbered_checks, public.stock_warn_insufficient_qty_scrap, public.slp_wiz, public.mrp_immediate_production_line, public.crm_iap_lead_mining_request_crm_tag_rel, public.stock_location, public.crm_lead_pls_update, public.payment_check_void, public.base_import_tests_models_o2m, public.account_model_rel, public.lot_label_layout, public.choose_delivery_carrier_sale_order_line_rel, public.project_task_stop_timers_wizard_line, public.muk_rest_access_rules, public.mail_activity, public.hr_expense_approve_duplicate_hr_expense_sheet_rel, public.phone_blacklist_remove, public.delivery_carrier, public.mail_message, public.mrp_eco_routing_change, public.payment_country_rel, public.muk_rest_oauth, public.employee_category_rel, public.ramp_reimbursements, public.shopify_res_config_settings_res_users_rel, public.crm_team, public.base_import_tests_models_o2m_child, public.mrp_production_backorder_line, public.res_currency_rate, public.account_incoterms, public.account_payment_term, public.account_tax_repartition_line_template, public.gamification_badge_user, public.ir_config_parameter, public.hr_contract, public.account_report_manager, public.base_geo_provider, public.account_fiscal_position_template, public.shopify_instance_ept, public.base_import_tests_models_char_noreadonly, public.product_tag, public.avalara_salestax_country_rel, public.sale_order_cancel, public.mail_mail_res_partner_rel, public.crm_convert_lead_mass_lead_rel, public.account_group, public.calendar_alarm, public.ir_model_fields, public.mrp_production_mrp_production_backorder_rel, public.hr_employee_mrp_workcenter_rel, public.account_tax_rep_template_plus, public.purchase_order_landed_costs_rel, public.project_favorite_user_rel, public.privacy_lookup_wizard, public.hr_department, public.account_analytic_line_mrp_workorder_rel, public.documents_request_wizard_documents_tag_rel, public.documents_share, public.sale_order_line, public.muk_webhooks_preview, public.reset_view_arch_wizard, public.quality_check, public.ir_filters, public.product_removal, public.shopify_order_risk, public.account_account_template, public.res_config_installer, public.project_tags, public.hr_departure_reason, public.quality_alert_stage, public.base_language_install_website_rel, public.mrp_production_production_rel, public.base_import_tests_models_m2o, public.cash_flow_lead_time_setting, public.ir_model_access, public.procurement_group, public.mrp_workorder_dependencies_rel, public.sale_order_template_line, public.mrp_eco_approval, public.ramp_configuration, public.account_bank_statement_ir_attachment_rel, public.documents_account_folder_setting, public.project_milestone, public.documents_folder, public.mail_mail, public.res_groups_report_rel, public.ir_act_window, public.team_stage_auto_close_from_rel, public.delivery_zip_prefix, public.account_account_tag_account_move_line_rel, public.product_template_attribute_value, public.fsm_stock_tracking_line, public.stock_package_destination, public.rel_followup_manual_reminder_res_partner, public.hr_employee_mrp_workorder_rel, public.stock_move_line_consume_rel, public.ir_ui_view_custom, public.wizard_fixed_asset_report, public.slide_channel_tag_group, public.account_fiscal_position_res_country_state_rel, public.base_import_tests_models_char_readonly, public.account_payment_term_line, public.muk_rest_endpoint, public.mail_message_reaction, public.stock_inventory_warning_stock_quant_rel, public.lot_label_layout_stock_picking_rel, public.account_batch_payment_rejection_account_payment_rel, public.l10n_us_1099_wizard, public.slide_slide_partner, public.gamification_goal_definition, public.shopify_payout_account_config_ept, public.shopify_product_product_ept, public.account_reports_export_wizard, public.product_tax_code, public.stock_barcode_cancel_operation, public.res_company, public.muk_webhooks_csrf_setting, public.ir_attachment, public.iap_account, public.res_company_worksheet_template_rel, public.sale_order, public.shopify_sale_auto_workflow_conf_rel, public.mail_message_res_partner_starred_rel, public.studio_approval_entry, public.cash_flow_user_configuration, public.quality_reason, public.account_move_account_resequence_wizard_rel, public.gamification_goal_wizard, public.stock_rules_report_stock_warehouse_rel, public.payment_provider, public.base_import_tests_models_float, public.portal_wizard, public.mrp_workcenter, public.business_line_route_per_warehouse, public.product_attribute, public.mrp_routing_workcenter, public.tier_review, public.muk_webhooks_authentication_parameter, public.res_country_res_country_group_rel, public.res_partner_industry, public.maintenance_team_users_rel, public.stock_lot, public.stock_wh_resupply_table, public.account_invoice_extract_words, public.mail_channel_member, public.mrp_production, public.account_analytic_line_hr_timesheet_merge_wizard_rel, public.stock_adjustment_reason, public.change_password_own, public.document_tag_rel, public.website_robots, public.ir_sequence_date_range, public.sale_payment_term, public.deposit_order, public.product_category, public.res_users_log, public.slide_slide_resource, public.stock_warehouse, public.ir_profile, public.activity_cost_rule, public.muk_rest_callback, public.shopify_product_template_ept, public.ir_act_client, public.crm_lead_crm_lead2opportunity_partner_mass_rel, public.sale_order_tag_rel, public.stock_move, public.shopify_export_stock_queue_ept, public.mrp_routing_workcenter_dependencies_rel, public.avalara_salestax_ping, public.stock_request_count, public.shopify_payout_report_line_ept, public.account_tax_filiation_rel, public.rel_badge_auth_users, public.res_config_shopify_instance, public.account_payment_method, public.repair_order_make_invoice, public.shopify_prepare_product_for_export_ept, public.rma_operation, public.expense_tax, public.mrp_workcenter_productivity_loss, public.account_batch_payment_rejection, public.account_chart_template, public.hr_expense_hr_expense_refuse_wizard_rel, public.gamification_challenge, public.account_transfer_model_line_res_partner_rel, public.account_fiscal_position_account, public.account_reconcile_model_partner_mapping, public.avalara_salestax, public.sale_order_line_invoice_rel, public.product_template_attribute_line, public.crm_lead_scoring_frequency_field, public.tier_definition, public.account_analytic_line, public.mail_channel, public.product_attr_exclusion_value_ids_rel, public.sms_resend_recipient, public.hr_plan_activity_type, public.hr_payroll_structure_type, public.account_accrued_orders_wizard, public.mrp_production_backorder, public.stock_picking_backorder_rel, public.common_log_lines_ept, public.sms_composer, public.documents_folder_deletion_wizard, public.repair_order_repair_tags_rel, public.mrp_mps_forecast_details, public.base_partner_merge_automatic_wizard_res_partner_rel, public.ir_act_window_group_rel, public.mail_activity_rel, public.account_reconcile_model_line_template, public.ir_ui_menu_group_rel, public.base_import_import, public.documents_tag, public.project_documents_tag_rel, public.calendar_attendee, public.account_reconcile_model_res_partner_rel, public.sms_template_preview, public.website_visitor, public.purchase_requisition_line, public.stock_conflict_quant_rel, public.mail_message_schedule, public.stock_return_picking, public.utm_medium, public.snailmail_letter_format_error, public.product_attribute_custom_value, public.account_fiscal_position_template_res_country_state_rel, public.base_module_update, public.documents_folder_read_groups, public.crm_iap_lead_industry_crm_iap_lead_mining_request_rel, public.shopify_export_stock_queue_line_ept, public.stock_inventory_conflict_stock_quant_rel, public.comment_wizard_tier_review_rel, public.account_resequence_wizard, public.stock_scheduler_compute, public.mrp_bom_line, public.bus_presence, public.picking_label_type_stock_picking_rel, public.excluded_tag_ids_rule_table, public.product_pricelist_item, public.base_import_tests_models_m2o_required, public.account_bank_statement, public.avalara_salestax_getcompany, public.sale_order_option, public.base_import_tests_models_char, public.digest_tip_res_users_rel, public.res_users_identitycheck, public.shopify_customer_data_queue_ept, public.product_attribute_value, public.stock_inventory_stock_lot_rel, public.crm_merge_opportunity, public.gamification_badge, public.project_project_project_tags_rel, public.product_category_scrap_reason_code_rel, public.theme_ir_attachment, public.project_task_type_rel, public.bus_bus, public.mail_wizard_invite, public.res_partner_bank, public.account_tax_purchase_order_line_rel, public.muk_rest_oauth1, public.documents_tag_mail_activity_type_rel, public.mrp_workorder_additional_product, public.project_task_recurrence, public.shopify_customer_data_queue_line_ept, public.maintenance_equipment, public.crm_iap_lead_mining_request_res_country_state_rel, public.mail_compose_message_res_partner_rel, public.account_journal_group, public.quality_tag, public.report_layout, public.res_partner_stage, public.repair_propose_change, public.rule_group_rel, public.mail_followers_mail_message_subtype_rel, public.res_config_settings, public.account_fiscal_position_account_template, public.sale_workflow_process_ept, public.project_tags_project_task_rel, public.marabunta_version, public.muk_rest_oauth2, public.product_label_layout_stock_move_line_rel, public.project_create_invoice, public.delivery_zip_prefix_rel, public.purchase_requisition, public.crm_lead2opportunity_partner, public.helpdesk_team, public.change_password_wizard, public.maintenance_stage, public.product_packaging, public.ir_act_report_xml, public.muk_rest_client_generator, public.account_payment_register_move_line_rel, public.sparse_fields_test, public.mrp_eco_approval_template_res_users_rel, public.mrp_workorder_res_users_rel, public.stock_warehouse_orderpoint, public.mail_guest, public.website, public.stock_inventory_stock_quant_rel, public.account_move_reversal, public.account_analytic_tracking_item_mrp_production_rel, public.crm_tag_rel, public.crm_quotation_partner, public.quality_alert_quality_tag_rel, public.res_groups_spreadsheet_document_to_dashboard_rel, public.account_account_account_journal_rel, public.account_multicurrency_revaluation_wizard, public.purchase_requisition_create_alternative, public.portal_share_res_partner_rel, public.shopify_instance_import_export_rel, public.stock_scrap, public.decimal_precision, public.queue_jobs_to_done, public.stock_package_level, public.phone_blacklist, public.res_users_apikeys, public.propose_change, public.account_tax_repartition_financial_tags, public.stock_backorder_confirmation_line, public.mrp_repair_workorder_res_users_rel, public.quality_alert_team, public.hr_plan_wizard, public.account_unreconcile, public.crm_recurring_plan, public.account_tax_group, public.res_users_settings_volumes, public.calendar_alarm_calendar_event_rel, public.gamification_badge_rule_badge_rel, public.account_analytic_tracking_item, public.cash_flow_transaction_type, public.muk_rest_access_token, public.res_partner_slide_channel_invite_rel, public.stock_warn_insufficient_qty_repair, public.res_country, public.uom_category, public.mrp_routing_workcenter_product_template_attribute_value_rel, public.merge_opportunity_rel, public.product_replenish_stock_route_rel, public.res_lang_install_rel, public.base_import_tests_models_complex, public.mrp_consumption_warning_mrp_production_rel, public.shopify_instance_ept_res_users_rel, public.account_payment_account_bank_statement_line_rel, public.mrp_eco_tag, public.account_reconcile_model_template_res_partner_rel, public.res_groups, public.slide_slide, public.cash_flow_projection, public.stock_picking, public.helpdesk_sla, public.quality_alert, public.website_track, public.purchase_order_group, public.queue_jobs_to_cancelled, public.website_page, public.ir_exports_line, public.website_menu, public.web_tour_tour, public.res_users_tier_review_rel, public.account_tax_template_filiation_rel, public.stock_assign_serial, public.documents_document_documents_link_to_record_wizard_rel, public.ir_model_fields_group_rel, public.payment_link_wizard, public.account_budget_post, public.account_transfer_model_line, public.account_move_sale_order_rel, public.account_tax_unit, public.account_fiscal_position, public.journal_account_control_rel, public.purchase_order_line, public.barcode_rule, public.gamification_challenge_line, public.shopify_order_payment_ept, public.stock_route_move, public.muk_webhooks_authentication, public.email_template_attachment_rel, public.stock_inventory_adjustment_name_stock_quant_rel, public.stock_landed_cost, public.res_company_users_rel, public.project_project, public.mail_followers, public.calendar_event_type, public.crm_lead_website_visitor_rel, public.crm_stage, public.base_import_mapping, public.shopify_payout_report_ept, public.stock_picking_type, public.ir_ui_menu, public.res_groups_slide_channel_rel, p…",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.j.AbstractJdbcSource(discoverInternal):167 Internal schemas to exclude: [catalog_history, information_schema, pg_catalog, pg_internal]",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.d.j.s.AdaptiveStreamingQueryConfig(initialize):31 Set initial fetch size: 10 rows",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.j.AbstractJdbcSource(discoverPrimaryKeys):267 Discover primary keys for tables: []",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO c.z.h.HikariDataSource(close):350 HikariPool-1 - Shutdown initiated...",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO c.z.h.HikariDataSource(close):352 HikariPool-1 - Shutdown completed.",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.p.PostgresUtils(isCdc):61 using CDC: true",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.b.IntegrationRunner(runInternal):226 Completed integration: io.airbyte.integrations.base.ssh.SshWrappedSource",
                "2023-08-15 14:13:42 \u001B[32mINFO\u001B[m i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - INFO i.a.i.s.p.PostgresSource(main):638 completed source: class io.airbyte.integrations.source.postgres.PostgresSource",
                "2023-08-15 14:13:43 \u001B[32mINFO\u001B[m i.a.a.c.AirbyteApiClient(retryWithJitterThrows):251 - Attempt 0 to call to write discover schema result",
                "2023-08-15 14:13:43 \u001B[32mINFO\u001B[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling..."
            ]
        }
    },
    "catalogId": "33bbbb07-a138-4923-bd48-d4b317f8fb29"
}

Contribute

sh4sh commented 1 year ago

This occurs with or without CDC. Connector check passes but when setting up a new connection, tables are not populated in catalog / don't show in the UI (it's greyed out and says "No streams")

We re-ran setup for the airbyte db user, and checked replication identity etc but it didn't help. Either way, the connector seems to find the tables, they just don't get added to the catalog for some reason.

joseph-maddox commented 1 year ago

We think this may have been due to the large number of tables (more than 10k) visible to the db user that we were using. Assuming that is the case, it would have been helpful to have had that show up in the logs as an error.