search

apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.65k stars 1.91k forks source link

com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0} #432

Closed hwzp99 closed 1 year ago

hwzp99 commented 3 years ago

Describe the bug A clear and concise description of what the bug is. 2021-09-16 14:41:58,859 ERROR io.debezium.connector.mysql.BinlogReader [] - Failed due to error: Error processing binlog event org.apache.kafka.connect.errors.ConnectException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0} at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:218) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:607) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1104) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:955) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:595) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:839) [darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] Caused by: java.lang.RuntimeException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0} at io.debezium.connector.mysql.BinlogReader.handleServerIncident(BinlogReader.java:675) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:590) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] ... 5 more Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1631767918000, eventType=UPDATE_ROWS, serverId=2121124478, headerLength=19, dataLength=7701, nextPosition=217369687, flags=0} at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:304) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:227) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:252) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:934) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] ... 3 more Caused by: java.io.EOFException at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:192) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at java.io.InputStream.read(InputStream.java:170) ~[?:1.8.0_202] at java.io.InputStream.skip(InputStream.java:224) ~[?:1.8.0_202] at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.skipToTheEndOfTheBlock(ByteArrayInputStream.java:219) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:300) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:227) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:252) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:934) ~[darwinKafkaDataToHive-1.0-SNAPSHOT.jar:?] ... 3 more 2021-09-16 14:41:58,862 INFO io.debezium.connector.mysql.BinlogReader [] - Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored.

Environment :

To Reproduce Steps to reproduce the behavior:

  1. Thes test data :
  2. The test code :
  3. The error :

Additional Description sql: "CREATE TABLE ocean_creative (\n" + "id BIGINT ,\n"+ "created_at TIMESTAMP(3),\n"+ "updated_at TIMESTAMP(3),\n"+ "deleted_at TIMESTAMP(3),\n"+ "creative_id BIGINT ,\n"+ "ad_id BIGINT ,\n"+ "advertiser_id BIGINT ,\n"+ "title STRING,\n"+ "creative_word_ids STRING,\n"+ "status STRING,\n"+ "opt_status STRING,\n"+ "image_mode STRING,\n"+ "image_ids STRING,\n"+ "image_id STRING,\n"+ "video_id STRING,\n"+ "third_party_id STRING,\n"+ "materials_title STRING,\n"+ "materials_image_id STRING,\n"+ "materials_video_id STRING,\n"+ "creative_create_time STRING,\n"+ "creative_modify_time STRING\n"+ ")WITH (\n" + " 'connector' = 'mysql-cdc',\n" + " 'hostname' = '%s',\n" + " 'port' = '3306',\n" + " 'username' = '',\n" + " 'password' = '',\n" + " 'database-name' = 'darwin',\n" + " 'debezium.snapshot.locking.mode' = 'none',\n" + " 'server-id' = '5427',\n" + " 'table-name' = 'ocean_creative'\n" + ")";

"CREATE TABLE ocean_creative_report (\n" + "id BIGINT,\n"+ "created_at TIMESTAMP(3),\n"+ "updated_at TIMESTAMP(3),\n"+ "deleted_at TIMESTAMP(3),\n"+ "marked BIGINT,\n"+ "advertiser_id BIGINT,\n"+ "advertiser_name STRING,\n"+ "ads_type STRING,\n"+ "campaign_id BIGINT,\n"+ "campaign_name STRING,\n"+ "campaign_type STRING,\n"+ "ad_id BIGINT,\n"+ "ad_name STRING,\n"+ "creative_id BIGINT,\n"+ "report_group_by STRING,\n"+ "stat_datetime STRING,\n"+ "stat_date DATE ,\n"+ "stat_month DATE ,\n"+ "stat_year INT,\n"+ "stat_week INT,\n"+ "stat_hour INT,\n"+ "ac STRING,\n"+ "active BIGINT,\n"+ "active_cost DECIMAL(20,4),\n"+ "active_pay_amount DECIMAL(20,4),\n"+ "active_pay_cost DECIMAL(20,4),\n"+ "active_pay_rate DECIMAL(20,4),\n"+ "active_rate DECIMAL(20,4),\n"+ "active_register_cost DECIMAL(20,4),\n"+ "active_register_rate DECIMAL(20,4),\n"+ "ad_tag STRING,\n"+ "advanced_creative_counsel_click BIGINT,\n"+ "advanced_creative_coupon_addition BIGINT,\n"+ "advanced_creative_form_click BIGINT,\n"+ "advanced_creative_form_submit BIGINT,\n"+ "advanced_creative_phone_click BIGINT,\n"+ "age STRING,\n"+ "approval_count BIGINT,\n"+ "attribution_active_pay_7d_per_count BIGINT,\n"+ "attribution_convert BIGINT,\n"+ "attribution_convert_cost DECIMAL(20,4),\n"+ "attribution_deep_convert BIGINT,\n"+ "attribution_deep_convert_cost DECIMAL(20,4),\n"+ "attribution_game_pay_7d_cost DECIMAL(20,4),\n"+ "attribution_game_pay_7d_count BIGINT,\n"+ "attribution_next_day_open_cnt BIGINT,\n"+ "attribution_next_day_open_cost DECIMAL(20,4),\n"+ "attribution_next_day_open_rate DECIMAL(20,4),\n"+ "attribution_wechat_first_pay_30d_cost DECIMAL(20,4),\n"+ "attribution_wechat_first_pay_30d_count BIGINT,\n"+ "attribution_wechat_first_pay_30d_rate DECIMAL(20,4),\n"+ "attribution_wechat_login_30d_cost DECIMAL(20,4),\n"+ "attribution_wechat_login_30d_count BIGINT,\n"+ "attribution_wechat_pay_30d_amount DECIMAL(20,4),\n"+ "attribution_wechat_pay_30d_roi DECIMAL(20,4),\n"+ "average_play_progress DECIMAL(20,4),\n"+ "average_play_time_per_play DECIMAL(20,4),\n"+ "average_video_play DECIMAL(20,4),\n"+ "avg_click_cost DECIMAL(20,4),\n"+ "avg_rank DECIMAL(20,4),\n"+ "avg_show_cost DECIMAL(20,4),\n"+ "bidword STRING,\n"+ "bidword_id BIGINT,\n"+ "button BIGINT,\n"+ "card_show BIGINT,\n"+ "city_name STRING,\n"+ "click BIGINT,\n"+ "click_call_dy BIGINT,\n"+ "click_download BIGINT,\n"+ "click_install BIGINT,\n"+ "click_landing_page BIGINT,\n"+ "click_shopwindow BIGINT,\n"+ "click_website BIGINT,\n"+ "comment BIGINT,\n"+ "commute_first_pay_count BIGINT,\n"+ "consult BIGINT,\n"+ "consult_effective BIGINT,\n"+ "convert BIGINT,\n"+ "convert_cost DECIMAL(20,4),\n"+ "convert_rate DECIMAL(20,4),\n"+ "convert_show_rate DECIMAL(20,4),\n"+ "cost DECIMAL(20,4),\n"+ "coupon BIGINT,\n"+ "coupon_single_page BIGINT,\n"+ "cpa DECIMAL(20,4),\n"+ "cpc DECIMAL(20,4),\n"+ "cpm DECIMAL(20,4),\n"+ "creative_material_mode STRING,\n"+ "ctr DECIMAL(20,4),\n"+ "deep_convert BIGINT,\n"+ "deep_convert_cost DECIMAL(20,4),\n"+ "deep_convert_rate DECIMAL(20,4),\n"+ "dislike BIGINT,\n"+ "download BIGINT,\n"+ "download_finish BIGINT,\n"+ "download_finish_cost DECIMAL(20,4),\n"+ "download_finish_rate DECIMAL(20,4),\n"+ "download_start BIGINT,\n"+ "download_start_cost DECIMAL(20,4),\n"+ "download_start_rate DECIMAL(20,4),\n"+ "external_action STRING,\n"+ "first_order_count BIGINT,\n"+ "first_rental_order_count BIGINT,\n"+ "follow BIGINT,\n"+ "form BIGINT,\n"+ "game_addiction BIGINT,\n"+ "game_addiction_cost DECIMAL(20,4),\n"+ "game_addiction_rate DECIMAL(20,4),\n"+ "game_pay_cost BIGINT,\n"+ "game_pay_count BIGINT,\n"+ "gender STRING,\n"+ "home_visited BIGINT,\n"+ "ies_challenge_click BIGINT,\n"+ "ies_music_click BIGINT,\n"+ "image_mode STRING,\n"+ "in_app_cart BIGINT,\n"+ "in_app_detail_uv BIGINT,\n"+ "in_app_order BIGINT,\n"+ "in_app_pay BIGINT,\n"+ "in_app_uv BIGINT,\n"+ "install_finish BIGINT,\n"+ "install_finish_cost DECIMAL(20,4),\n"+ "install_finish_rate DECIMAL(20,4),\n"+ "interact_per_cost DECIMAL(20,4),\n"+ "interest_tag STRING,\n"+ "inventory STRING,\n"+ "landing_type STRING,\n"+ "like BIGINT,\n"+ "live_fans_club_join_cnt BIGINT,\n"+ "live_watch_one_minute_count BIGINT,\n"+ "loan_completion BIGINT,\n"+ "loan_completion_cost DECIMAL(20,4),\n"+ "loan_completion_rate DECIMAL(20,4),\n"+ "loan_credit BIGINT,\n"+ "loan_credit_cost DECIMAL(20,4),\n"+ "loan_credit_rate DECIMAL(20,4),\n"+ "location_click BIGINT,\n"+ "lottery BIGINT,\n"+ "luban_live_click_product_cnt BIGINT,\n"+ "luban_live_comment_cnt BIGINT,\n"+ "luban_live_enter_cnt BIGINT,\n"+ "luban_live_follow_cnt BIGINT,\n"+ "luban_live_gift_amount DECIMAL(20,4),\n"+ "luban_live_gift_cnt BIGINT,\n"+ "luban_live_pay_order_count BIGINT,\n"+ "luban_live_pay_order_stat_cost DECIMAL(20,4),\n"+ "luban_live_share_cnt BIGINT,\n"+ "luban_live_slidecart_click_cnt BIGINT,\n"+ "luban_order_cnt BIGINT,\n"+ "luban_order_roi DECIMAL(20,4),\n"+ "luban_order_stat_amount DECIMAL(20,4),\n"+ "map_search BIGINT,\n"+ "material_id BIGINT,\n"+ "message BIGINT,\n"+ "message_action BIGINT,\n"+ "next_day_open BIGINT,\n"+ "next_day_open_cost DECIMAL(20,4),\n"+ "next_day_open_rate DECIMAL(20,4),\n"+ "pay_count BIGINT,\n"+ "phone BIGINT,\n"+ "phone_confirm BIGINT,\n"+ "phone_connect BIGINT,\n"+ "phone_effective BIGINT,\n"+ "platform STRING,\n"+ "play_100_feed_break BIGINT,\n"+ "play_100_feed_break_rate DECIMAL(20,4),\n"+ "play_25_feed_break BIGINT,\n"+ "play_25_feed_break_rate DECIMAL(20,4),\n"+ "play_50_feed_break BIGINT,\n"+ "play_50_feed_break_rate DECIMAL(20,4),\n"+ "play_75_feed_break BIGINT,\n"+ "play_75_feed_break_rate DECIMAL(20,4),\n"+ "play_duration DECIMAL(20,4),\n"+ "play_duration_10s DECIMAL(20,4),\n"+ "play_duration_10s_rate DECIMAL(20,4),\n"+ "play_duration_2s DECIMAL(20,4),\n"+ "play_duration_2s_rate DECIMAL(20,4),\n"+ "play_duration_3s DECIMAL(20,4),\n"+ "play_duration_3s_rate DECIMAL(20,4),\n"+ "play_duration_5s_rate DECIMAL(20,4),\n"+ "play_duration_sum BIGINT,\n"+ "play_over_rate DECIMAL(20,4),\n"+ "playable_id BIGINT,\n"+ "playable_name STRING,\n"+ "playable_orientation STRING,\n"+ "playable_preview_url STRING,\n"+ "playable_url STRING,\n"+ "poi_address_click BIGINT,\n"+ "poi_collect BIGINT,\n"+ "pre_loan_credit BIGINT,\n"+ "pre_loan_credit_cost DECIMAL(20,4),\n"+ "pricing STRING,\n"+ "pricing_category STRING,\n"+ "province_name STRING,\n"+ "qq BIGINT,\n"+ "query STRING,\n"+ "redirect BIGINT,\n"+ "redirect_to_shop BIGINT,\n"+ "register BIGINT,\n"+ "report BIGINT,\n"+ "share BIGINT,\n"+ "shopping BIGINT,\n"+ "show BIGINT,\n"+ "submit_certification_count BIGINT,\n"+ "total_play BIGINT,\n"+ "valid_play BIGINT,\n"+ "valid_play_cost DECIMAL(20,4),\n"+ "valid_play_rate DECIMAL(20,4),\n"+ "view BIGINT,\n"+ "vote BIGINT,\n"+ "wechat BIGINT,\n"+ "wechat_first_pay_cost DECIMAL(20,4),\n"+ "wechat_first_pay_count BIGINT,\n"+ "wechat_first_pay_rate DECIMAL(20,4),\n"+ "wechat_login_cost DECIMAL(20,4),\n"+ "wechat_login_count BIGINT,\n"+ "wechat_pay_amount DECIMAL(20,4),\n"+ "wifi_play BIGINT,\n"+ "wifi_play_rate DECIMAL(20,4),\n"+ "luban_live_pay_order_count_by_author_15days BIGINT,\n"+ "luban_live_pay_order_count_by_author_30days BIGINT,\n"+ "luban_live_pay_order_count_by_author_3days BIGINT,\n"+ "luban_live_pay_order_count_by_author_7days BIGINT,\n"+ "luban_live_pay_order_stat_cost_by_author_15days DECIMAL(20,4),\n"+ "luban_live_pay_order_stat_cost_by_author_30days DECIMAL(20,4),\n"+ "luban_live_pay_order_stat_cost_by_author_3days DECIMAL(20,4),\n"+ "luban_live_pay_order_stat_cost_by_author_7days DECIMAL(20,4)\n"+ ")WITH (\n" + " 'connector' = 'mysql-cdc',\n" + " 'hostname' = '%s',\n" + " 'port' = '3306',\n" + " 'username' = '',\n" + " 'password' = '',\n" + " 'database-name' = 'darwin',\n" + " 'debezium.snapshot.locking.mode' = 'none',\n" + " 'server-id' = '5426',\n" + " 'table-name' = 'ocean_creative_report'\n" + ")";

stmtSet.addInsertSql("INSERT INTO ocean_creative_report_index select a.id,a.created_at,a.updated_at,a.deleted_at,a.marked,a.advertiser_id,a.advertiser_name,a.ads_type,a.campaign_id,a.campaign_name,a.campaign_type,a.ad_id,a.ad_name,a.creative_id,a.report_group_by,a.stat_datetime,a.stat_date,a.stat_year,a.stat_month,a.stat_week,a.stat_hour,a.ac,a.active,a.active_cost,a.active_pay_amount,a.active_pay_cost,a.active_pay_rate,a.active_rate,a.active_register_cost,a.active_register_rate,a.ad_tag,a.advanced_creative_counsel_click,a.advanced_creative_coupon_addition,a.advanced_creative_form_click,a.advanced_creative_form_submit,a.advanced_creative_phone_click,a.age,a.approval_count,a.attribution_active_pay_7d_per_count,a.attribution_convert,a.attribution_convert_cost,a.attribution_deep_convert,a.attribution_deep_convert_cost,a.attribution_game_pay_7d_cost,a.attribution_game_pay_7d_count,a.attribution_next_day_open_cnt,a.attribution_next_day_open_cost,a.attribution_next_day_open_rate,a.attribution_wechat_first_pay_30d_cost,a.attribution_wechat_first_pay_30d_count,a.attribution_wechat_first_pay_30d_rate,a.attribution_wechat_login_30d_cost,a.attribution_wechat_login_30d_count,a.attribution_wechat_pay_30d_amount,a.attribution_wechat_pay_30d_roi,a.average_play_progress,a.average_play_time_per_play,a.average_video_play,a.avg_click_cost,a.avg_rank,a.avg_show_cost,a.bidword,a.bidword_id,a.button,a.card_show,a.city_name,a.click,a.click_call_dy,a.click_download,a.click_install,a.click_landing_page,a.click_shopwindow,a.click_website,a.comment,a.commute_first_pay_count,a.consult,a.consult_effective,a.convert,a.convert_cost,a.convert_rate,a.convert_show_rate,a.cost,a.coupon,a.coupon_single_page,a.cpa,a.cpc,a.cpm,a.creative_material_mode,a.ctr,a.deep_convert,a.deep_convert_cost,a.deep_convert_rate,a.dislike,a.download,a.download_finish,a.download_finish_cost,a.download_finish_rate,a.download_start,a.download_start_cost,a.download_start_rate,a.external_action,a.first_order_count,a.first_rental_order_count,a.follow,a.form,a.game_addiction,a.game_addiction_cost,a.game_addiction_rate,a.game_pay_cost,a.game_pay_count,a.gender,a.home_visited,a.ies_challenge_click,a.ies_music_click,a.image_mode,a.in_app_cart,a.in_app_detail_uv,a.in_app_order,a.in_app_pay,a.in_app_uv,a.install_finish,a.install_finish_cost,a.install_finish_rate,a.interact_per_cost,a.interest_tag,a.inventory,a.like,a.live_fans_club_join_cnt,a.live_watch_one_minute_count,a.loan_completion,a.loan_completion_cost,a.loan_completion_rate,a.loan_credit,a.loan_credit_cost,a.loan_credit_rate,a.location_click,a.lottery,a.luban_live_click_product_cnt,a.luban_live_comment_cnt,a.luban_live_enter_cnt,a.luban_live_follow_cnt,a.luban_live_gift_amount,a.luban_live_gift_cnt,a.luban_live_pay_order_count,a.luban_live_pay_order_stat_cost,a.luban_live_share_cnt,a.luban_live_slidecart_click_cnt,a.luban_order_cnt,a.luban_order_roi,a.luban_order_stat_amount,a.map_search,a.material_id,a.message,a.message_action,a.next_day_open,a.next_day_open_cost,a.next_day_open_rate,a.pay_count,a.phone,a.phone_confirm,a.phone_connect,a.phone_effective,a.platform,a.play_100_feed_break,a.play_100_feed_break_rate,a.play_25_feed_break,a.play_25_feed_break_rate,a.play_50_feed_break,a.play_50_feed_break_rate,a.play_75_feed_break,a.play_75_feed_break_rate,a.play_duration,a.play_duration_10s,a.play_duration_10s_rate,a.play_duration_2s,a.play_duration_2s_rate,a.play_duration_3s,a.play_duration_3s_rate,a.play_duration_5s_rate,a.play_duration_sum,a.play_over_rate,a.playable_id,a.playable_name,a.playable_orientation,a.playable_preview_url,a.playable_url,a.poi_address_click,a.poi_collect,a.pre_loan_credit,a.pre_loan_credit_cost,a.pricing,a.pricing_category,a.province_name,a.qq,a.query,a.redirect,a.redirect_to_shop,a.register,a.report,a.share,a.shopping,a.show,a.submit_certification_count,a.total_play,a.valid_play,a.valid_play_cost,a.valid_play_rate,a.view,a.vote,a.wechat,a.wechat_first_pay_cost,a.wechat_first_pay_count,a.wechat_first_pay_rate,a.wechat_login_cost,a.wechat_login_count,a.wechat_pay_amount,a.wifi_play,a.wifi_play_rate,a.luban_live_pay_order_count_by_author_15days,a.luban_live_pay_order_count_by_author_30days,a.luban_live_pay_order_count_by_author_3days,a.luban_live_pay_order_count_by_author_7days,a.luban_live_pay_order_stat_cost_by_author_15days,a.luban_live_pay_order_stat_cost_by_author_30days,a.luban_live_pay_order_stat_cost_by_author_3days,a.luban_live_pay_order_stat_cost_by_author_7days,b.title,b.creative_word_ids,b.status,b.opt_status,b.image_ids,b.image_id,b.video_id,b.third_party_id,b.materials_title,b.materials_image_id,b.materials_video_id,b.creative_create_time,b.creative_modify_time,c.marketing_purpose,c.landing_type from ocean_creative_report a left join ocean_creative b on a.creative_id = b.creative_id left join ocean_campaign c on a.campaign_id=c.ad_group_id");

leonardBang commented 3 years ago

Sorry, this should not be same with #276

hwzp99 commented 3 years ago

same with #276 2.x #276 1.x #432

ghost commented 3 years ago

same with #276 2.x #276 1.x #432

I also met the same problem, how did you solve it。

lykpass commented 3 years ago

Sorry, this should not be same with #276 Is there a way out?

ghost commented 2 years ago

Sorry, this should not be same with #276 Is there a way out?

I see you replied set global slave_net_timeout = 120 in #61; Have you solved the problem？

lykpass commented 2 years ago

Sorry, this should not be same with #276 Is there a way out?

I see you replied set global slave_net_timeout = 120 in #61; Have you solved the problem？

我出现这个问题的时间点是和我们公司业务库高负载的时间点是对应的，但是我们是rds5.6的不支持这俩参数的修改，这个问题令我很苦恼，,隔一段时间就会出现，然后程序重启，有时候能重启成功，有时候会出现一些其它问题导致超过重启次数失败 @leonardBang

q2515045 commented 2 years ago

问题怎么关闭了呢，我也频繁遇到这种问题，当多个cdc同步同一个库时，数据库升级大量数据基本就会发生这个错误

zhilinli123 commented 2 years ago

@lykpass 我用cdc2.2.1也出现了这个问题 但是程序不会导致无限重启 请问后续您有什么好的解决办法吗？

dailai commented 2 years ago

+1 也有同样的问题

qq443672581 commented 2 years ago

@lykpass 我用cdc2.2.1也出现了这个问题 但是程序不会导致无限重启 请问后续您有什么好的解决办法吗？

我们是在现有任务加上新表的时候出现，我通过debug才发现有一个序列化错误，导致卡在拉取那里，不知道你们现在解决了吗？

voyagertanyao commented 2 years ago

me too. when i restart from a savepoint

leonardBang commented 2 years ago

Thanks for your valuable feedback, I'll look into this issue when I have time

xhmz commented 2 years ago

我们也遇到了， 背压严重的时候, 引起的 binlog client 空闲. 被 svr 踢掉了

leonardBang commented 2 years ago

Hi, guys. I've investigated the root cause and will fix in the next version. But I'm busy now, thus please wait a while.

kylemeow commented 2 years ago

I've met with this issue as well, and I am looking forward to the fix : )

foobarrer commented 2 years ago

I face the problem too, and all the work stop for this reason

kaixinjiuhao123 commented 2 years ago

I also foresee this problem, what is the solution，online etc

kylemeow commented 2 years ago

This issue might be caused by a long pause of TaskManager, such as a prolonged full GC, at which time the MySQL server thinks the client is not responsive so it stops transmitting binlog to the client, resulting in the EOFException.

Usually, a full restart could fix it temporarily. But I wonder if we could adjust the timeout so that the server can wait a little bit longer.

zhilinli123 commented 2 years ago

这个问题可能是由于TaskManager的长时间停顿，比如长时间的full GC，此时MySQL服务器认为客户端没有响应，停止向客户端传输binlog，导致EOFException。

通常，完全重启可以暂时修复它。但是我想知道我们是否可以调整超时，以便服务器可以等待更长的时间。 refer to this https://github.com/ververica/flink-cdc-connectors/issues/61

frozenheartboy commented 2 years ago

问题+1，在修复版本发布前，有什么临时替代方案吗

pycgo commented 2 years ago

+1 求修复

JesseAtSZ commented 1 year ago

附议 @leonardBang

NicksonYap commented 1 year ago

Hi, guys. I've investigated the root cause and will fix in the next version. But I'm busy now, thus please wait a while.

HI @leonardBang, is there a temporary workaround for this? the error happens often with checkpoints on server restart or job recovery

dkondratenkocs commented 1 year ago

@leonardBang could you please describe your fix or let us know a workaround

ruanhang1993 commented 1 year ago

Take a look at the FAQ https://github.com/ververica/flink-cdc-connectors/wiki/FAQ#q13-the-job-reports-an-error-eventdatadeserializationexception-failed-to-deserialize-data-of-eventheaderv4--caused-by-javanetsocketexception-connection-resetwhat-should-i-do .

pycgo commented 1 year ago

This FAQ doesn't seem to solve the problem

buxudu-dev commented 1 year ago

我是在突然数据量变多的时候会出现这个问题，而且会造成数据丢失，导致我无法上生产使用，求修复

pycgo commented 1 year ago

我是在突然数据量变多的时候会出现这个问题，而且会造成数据丢失，导致我无法上生产使用，求修复

数据不会丢失吧 数据是记录在msyql的binlog里面

buxudu-dev commented 1 year ago

@pycgo 有没有好的解决方法

pycgo commented 1 year ago

@pycgo 有没有好的解决方法

增量跑 从失败的位点继续 这个过程可以自己写代码检查自动化