[Bug] Unable to flush data to Doris: unknown result status

[zhoukaiii]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Version

updated master

What's Wrong?

数据在通过hdfsreader ->doriswriter写入数据时报错

初步怀疑是这样的数据导致写入失败 2022-11-11 张三50%/李四50%/王五 2

异常的日志： 2022/11/11 10:51:19 - Shell - (stdout) 经DataX智能分析,该任务最可能的错误原因是: 2022/11/11 10:51:19 - Shell - (stdout) com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-05], Description:[往您配置的写入表中写入数据时失败.]. - java.lang.RuntimeException: Writing records to Doris failed. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:130) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65) 2022/11/11 10:51:19 - Shell - (stdout) at java.lang.Thread.run(Thread.java:748) 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.lang.RuntimeException: Writing records to Doris failed. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:206) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:167) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:115) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:128) 2022/11/11 10:51:19 - Shell - (stdout) ... 3 more 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:187) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:36) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:151) 2022/11/11 10:51:19 - Shell - (stdout) ... 1 more 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:81) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:180) 2022/11/11 10:51:19 - Shell - (stdout) ... 3 more 2022/11/11 10:51:19 - Shell - (stdout) - java.lang.RuntimeException: Writing records to Doris failed. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:130) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65) 2022/11/11 10:51:19 - Shell - (stdout) at java.lang.Thread.run(Thread.java:748) 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.lang.RuntimeException: Writing records to Doris failed. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:206) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:167) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:115) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:128) 2022/11/11 10:51:19 - Shell - (stdout) ... 3 more 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:187) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:36) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:151) 2022/11/11 10:51:19 - Shell - (stdout) ... 1 more 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:81) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:180) 2022/11/11 10:51:19 - Shell - (stdout) ... 3 more 2022/11/11 10:51:19 - Shell - (stdout) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:152) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65) 2022/11/11 10:51:19 - Shell - (stdout) at java.lang.Thread.run(Thread.java:748) 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.lang.RuntimeException: Writing records to Doris failed. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:130) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150) 2022/11/11 10:51:19 - Shell - (stdout) ... 2 more 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.lang.RuntimeException: Writing records to Doris failed. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:206) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:167) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:115) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:128) 2022/11/11 10:51:19 - Shell - (stdout) ... 3 more 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:187) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:36) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:151) 2022/11/11 10:51:19 - Shell - (stdout) ... 1 more 2022/11/11 10:51:19 - Shell - (stdout) Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status. 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:81) 2022/11/11 10:51:19 - Shell - (stdout) at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:180) 2022/11/11 10:51:19 - Shell - (stdout) ... 3 more 2022/11/11 10:51:19 - Shell - (stdout) 2022/11/11 10:51:19 - xx- Finished job entry [Shell] (result=[false]) 2022/11/11 10:51:19 - xx- Job execution finished

What You Expected?

希望能成功处理类似的数据

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[ywhfzy]

MySQL同步至Doris 也报错这个，有懂的大佬吗？

[Allen-zy]

我也遇到了,但是脚本在本地可以执行，如果数据问题话，本地执行脚本的时候应该也会报错

[hf200012]

https://jiafeng-beijing-1308700295.cos.ap-beijing.myqcloud.com/datax.tar.gz 下载这个版本试试

[whmnoe4j]

这个和datax官方的有啥差别啊，是doris读写插件不一样，还是基础插件里面方法也变了啊？

[liujie-yt]

我这里报错跟你类似，内容中又日志地址，下载下来可以看到报错信息

Caused by: java.io.IOException: java.io.IOException: Failed to flush data to Doris.
{
"TxnId":5202759,"Label":"datax_doris_writer_30d43642-677f-48cc-8601-c72139672f32","Comment":"","TwoPhaseCommit":"false","Status":"Fail",
"Message":"[DATA_QUALITY_ERROR]too many filtered rows","NumberTotalRows":292770,"NumberLoadedRows":292769,"NumberFilteredRows":1,"NumberUnselectedRows":0,"LoadBytes":94664703,"LoadTimeMs":1368,"BeginTxnTimeMs":0,"StreamLoadPutTimeMs":1,
"ReadDataTimeMs":23,"WriteDataTimeMs":1364,"CommitAndPublishTimeMs":0,
"ErrorURL":"http://doriscluster-be-0.doriscluster-be-internal.doris.svc.cluster.local:8040/api/_load_error_log?file=__shard_125/error_log_insert_stmt_33461070593dc9d5-1807271f0dc96af_33461070593dc9d5_1807271f0dc96af"
}