Closed GitHub-19 closed 7 months ago
Where are the logs of both sides?
We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!
Steps to reproduce the behavior (Required)
the propertis of the routine load is: {"partitions":"","partial_update":"false","columnToColumnExpr":"reporttime,adcode,province,city,date,week,dayweather,nightweather,daytemp,nighttemp,daywind,nightwind,daypower,nightpower,daytemp_float,nighttemp_float,low,high,notice,aqi,sunset,sunrise,ganmao,quality,pm10,pm25,collect_time,weather_source","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"","dataFormat":"json","timezone":"Asia/Shanghai","format":"json","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"true","jsonpaths":"","taskConsumeSecond":"15","desireTaskConcurrentNum":"2","maxErrorNum":"10000","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"}
The Kafka error data is: [{"date":"13","high":"高温 37℃","low":"低温 24℃","ymd":"2023-07-13","week":"星期四","sunrise":"05:29","sunset":"19:01","aqi":58,"fx":"南风","fl":"2级","type":"多云","notice":"阴晴之间,谨防紫外线侵扰","reporttime":"20230713 13:31","adcode":"101230601","province":"福建","city":"漳州市","weather_source":"weather api","collect_time":"2023-07-13 15:49:46"},{"date":"14","high":"高温 40℃","low":"低温 27℃","ymd":"2023-07-14","week":"星期五","sunrise":"05:29","sunset":"19:01","aqi":55,"fx":"东风","fl":"1级","type":"晴","notice":"愿你拥有比阳光明媚的心情","reporttime":"20230713 13:31","adcode":"101230601","province":"福建","city":"漳州市","weather_source":"weather api","collect_time":"2023-07-13 15:49:46"}]
Kafka success data is:
{"date":"13","high":"高温 25℃","low":"低温 22℃","ymd":"2023-07-13","week":"星期四","sunrise":"04:57","sunset":"19:43","aqi":71,"fx":"东北风","fl":"1级","type":"小雨","notice":"雨虽小,注意保暖别感冒","reporttime":1689220860000,"adcode":"101010100","province":"北京","city":"北京市","weather_source":"weather api","collect_time":"2023-07-13 15:51:56"}
Expected behavior (Required)
Hope to consume normally and skip the array data with exceptions. If I alter kakfa offset to strip off this error data, it can consume normally.
Real behavior (Required)
But it is not consuming Kafka data.
StarRocks version (Required)