Open arashlayeghi opened 2 years ago
@arashlayeghi could you check with the latest MySQL Source Connector version and see if you are still running into this error? Thank you
@prateekmukhedkar I have tried just now. The problem persists. I think problem is with the destination clickhouse connector, because data is being fetched correctly from mysql source. It fails when it runs dbt normalization models on destination clickhouse database. For example, tables do not have the column _airbyte_unique_key
created in them, which was supposed to be created automatically at the time of the sync.
Any update on this? I am also facing this issue.
2023-10-17 06:55:13 normalization > 06:55:13 Completed with 1 error and 0 warnings: 2023-10-17 06:55:13 normalization > 06:55:13
2023-10-17 06:55:13 normalization > 06:55:13 Database Error in model pdd_request_scd (models/generated/airbyte_incremental/scd/airbyte_stage_db/pdd_request_scd.sql) 2023-10-17 06:55:13 normalization > 06:55:13 :HTTPDriver for http://172.16.X.YYY:8123 returned response code 404) 2023-10-17 06:55:13 normalization > 06:55:13 Code: 47. DB::Exception: Missing columns: 'inc_data._ab_cdc_log_file' 'inc_data._ab_cdc_log_pos' 'inc_data._ab_cdc_cursor' while processing query: 'SELECT _airbyte_pdd_request_hashid, status_code, method, pdd_id, inc_data._ab_cdc_log_pos AS2023-10-17 06:55:16 normalization > 06:55:12.117260 [debug] [Thread-1 (]: Timing info for model.airbyte_utils.pdd_request_scd (execute): 2023-10-17 06:55:12.062759 => 2023-10-17 06:55:12.117175 2023-10-17 06:55:16 normalization > 06:55:12.119386 [debug] [Thread-1 (]: Database Error in model pdd_request_scd (models/generated/airbyte_incremental/scd/airbyte_stage_db/pdd_request_scd.sql) 2023-10-17 06:55:16 normalization > :HTTPDriver for http://172.16.X.YYY:8123 returned response code 404) 2023-10-17 06:55:16 normalization > Code: 47. DB::Exception: Missing columns: 'inc_data._ab_cdc_log_file' 'inc_data._ab_cdc_log_pos' 'inc_data._ab_cdc_cursor' while processing query: 'SELECT _airbyte_pdd_request_hashid, status_code, method, pdd_id, inc_data._ab_cdc_log_pos AS 2023-10-17 06:55:16 normalization > 06:55:12.119979 [error] [Thread-1 (]: 6 of 9 ERROR creating sql incremental model airbyte_stage_db.pdd_request_scd ........................................... [ERROR in 0.13s]
2023-10-17 06:55:16 normalization > 06:55:13.471437 [info ] [MainThread]: Completed with 1 error and 0 warnings: 2023-10-17 06:55:16 normalization > 06:55:13.471743 [info ] [MainThread]: 2023-10-17 06:55:16 normalization > 06:55:13.472043 [error] [MainThread]: Database Error in model pdd_request_scd (models/generated/airbyte_incremental/scd/airbyte_stage_db/pdd_request_scd.sql) 2023-10-17 06:55:16 normalization > 06:55:13.472410 [error] [MainThread]: :HTTPDriver for http://172.16.X.YYY:8123 returned response code 404) 2023-10-17 06:55:16 normalization > 06:55:13.472752 [error] [MainThread]: Code: 47. DB::Exception: Missing columns: 'inc_data._ab_cdc_log_file' 'inc_data._ab_cdc_log_pos' 'inc_data._ab_cdc_cursor' while processing query: 'SELECT _airbyte_pdd_request_hashid, status_code, method, pdd_id, inc_data._ab_cdc_log_pos AS
06:55:12.119979 [error] [Thread-1 (]: 6 of 9 ERROR creating sql incremental model airbyte_stage_db.pdd_request_scd ........................................... [ERROR in 0.13s] 06:55:13.472043 [error] [MainThread]: Database Error in model pdd_request_scd (models/generated/airbyte_incremental/scd/airbyte_stage_db/pdd_request_scd.sql) 06:55:13.472410 [error] [MainThread]: :HTTPDriver for http://172.16.X.YYY:8123 returned response code 404) 06:55:13.472752 [error] [MainThread]: Code: 47. DB::Exception: Missing columns: 'inc_data._ab_cdc_log_file' 'inc_data._ab_cdc_log_pos' 'inc_data._ab_cdc_cursor' while processing query: 'SELECT _airbyte_pdd_request_hashid, status_code, method, pdd_id, inc_data._ab_cdc_log_pos AS,retryable=
,timestamp=1697525716373,additionalProperties={}], io.airbyte.config.FailureReason@4fb84519[failureOrigin=normalization,failureType=system_error,internalMessage=[0m06:55:13.472410 [error] [MainThread]: :HTTPDriver for http://172.16.X.YYY:8123 returned response code 404),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@c98d1ce[additionalProperties={attemptNumber=5, jobId=100, from_trace_message=true}],stacktrace=AirbyteDbtError: 06:55:12.119979 [error] [Thread-1 (]: 6 of 9 ERROR creating sql incremental model airbyte_stage_db.pdd_request_scd ........................................... [ERROR in 0.13s] 06:55:13.472043 [error] [MainThread]: Database Error in model pdd_request_scd (models/generated/airbyte_incremental/scd/airbyte_stage_db/pdd_request_scd.sql) 06:55:13.472410 [error] [MainThread]: :HTTPDriver for http://172.16.X.YYY:8123 returned response code 404) 06:55:13.472752 [error] [MainThread]: Code: 47. DB::Exception: Missing columns: 'inc_data._ab_cdc_log_file' 'inc_data._ab_cdc_log_pos' 'inc_data._ab_cdc_cursor' while processing query: 'SELECT _airbyte_pdd_request_hashid, status_code, method, pdd_id, inc_data._ab_cdc_log_pos AS,retryable=,timestamp=1697525716373,additionalProperties={}]],additionalProperties={}] 2023-10-17 06:55:16 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-10-17 06:55:16 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END DEFAULT NORMALIZATION ----- 2023-10-17 06:55:16 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-10-17 06:55:16 INFO i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):56 - Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=5, totalCompleteFailures=5, successivePartialFailures=0, totalPartialFailures=1) Backoff before next attempt: 13 minutes 30 seconds
LOG
``` replace this with your long log output here ```