search

WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
https://github.com/WeBankFinTech/DataSphereStudio-Doc
Apache License 2.0
3.08k stars 1k forks source link

[Bug] 使用DataSphereStudioV 1.1.2中调用exchangisv1.1.2时候 接口前端会传输两个tranferMode 向 /api/dss/exchangis/main/job/5807/content，导致异常 #1119

Closed xuliancheng563 closed 9 months ago

xuliancheng563 commented 9 months ago

Search before asking

DSS Component

dss-web/workspace

What happened + What you expected to happen

版本：DataSphereStudio V1.1.2 exchangisv1.1.2 在界面中使用 DataSphereStudio exchangis 点击执行时，前端会向后端/api/dss/exchangis/main/job/5807/content发送请求，其中请求参数params存在重复 ,"config_key": "transferMode", 出现了两次，部分请求参数如下所示：

[{
    "subJobName": "taJCpFHGPrij",
    "dataSources": {
        "source_id": "HIVE.3.xuhui.student_score",
        "sink_id": "MYSQL.2.test_xlc.user_info"
    },
    "params": {
        "sources": [{
            "config_key": "transferMode",
            "config_name": "传输方式",
            "config_value": "二进制",
            "sort": 1
        }, {
            "config_key": "nullFormat",
            "config_name": "空值字符",
            "config_value": "",
            "sort": 2
        }, {
            "config_key": "transferMode",
            "config_name": "传输方式",
            "config_value": "二进制",
            "sort": 1
        }, {
            "config_key": "nullFormat",
            "config_name": "空值字符",
            "config_value": "",
            "sort": 2
        }],
        "sinks": [{
            "config_key": "writeMode",
            "config_name": "写入方式",
            "config_value": "INSERT",
            "sort": 1
        }, {
            "config_key": "writeMode",
            "config_name": "写入方式",
            "config_value": "INSERT",
            "sort": 1
        }]
    },
    "transforms": {
        "addEnable": true,
        "type": "MAPPING",
        "sql": null,
        "mapping": [{
            "validator": null,
            "transformer": null,
            "source_field_name": "stu_id",
            "source_field_type": "string",
            "sink_field_name": "username",
            "sink_field_type": "VARCHAR",
            "deleteEnable": true,
            "source_field_index": 0,
            "sink_field_index": 0,
            "source_field_editable": true,
            "sink_field_editable": true
        }, {
            "validator": null,
            "transformer": null,
            "source_field_name": "stu_name",
            "source_field_type": "string",
            "sink_field_name": "age",
            "sink_field_type": "INT",
            "deleteEnable": true,
            "source_field_index": 1,
            "sink_field_index": 1,
            "source_field_editable": true,
            "sink_field_editable": true
        }],
        "code_id": null,
        "transformEnable": true
    },
    "settings": [{
        "config_key": "setting.speed.byte",
        "config_name": "作业速率限制",
        "config_value": "5",
        "sort": 1
    }, {
        "config_key": "setting.speed.record",
        "config_name": "作业记录数限制",
        "config_value": "100",
        "sort": 2
    }, {
        "config_key": "setting.speed.channel",
        "config_name": "作业最大并行度",
        "config_value": "1",
        "sort": 3
    }, {
        "config_key": "setting.max.memory",
        "config_name": "作业最大使用内存",
        "config_value": "1024",
        "sort": 4
    }, {
        "config_key": "setting.errorLimit.record",
        "config_name": "最多错误记录数",
        "config_value": "",
        "sort": 5
    }, {
        "config_key": "setting.speed.byte",
        "config_name": "作业速率限制",
        "config_value": "5",
        "sort": 1
    }, {
        "config_key": "setting.speed.record",
        "config_name": "作业记录数限制",
        "config_value": "100",
        "sort": 2
    }, {
        "config_key": "setting.speed.channel",
        "config_name": "作业最大并行度",
        "config_value": "1",
        "sort": 3
    }, {
        "config_key": "setting.max.memory",
        "config_name": "作业最大使用内存",
        "config_value": "1024",
        "sort": 4
    }, {
        "config_key": "setting.errorLimit.record",
        "config_name": "最多错误记录数",
        "config_value": "",
        "sort": 5
    }]
}]

错误的日志：

1.   2024-01-04 10:34:30.577 INFO  - [hadoop:a40c6017-e17f-4344-bef5-dc924347fb01] Init to create launched job and begin generating
2.   2024-01-04 10:34:30.586 ERROR - [hadoop:a40c6017-e17f-4344-bef5-dc924347fb01] Error occurred in generating com.webank.wedatasphere.exchangis.job.server.exception.ExchangisTaskGenerateException: errCode: 31997 ,desc: Error in generating launchable tasks ,ip: dss.prod.sh ,port: 9080 ,serviceKind: dss-exchangis-server
3.      at com.webank.wedatasphere.exchangis.job.server.execution.generator.DefaultTaskGenerator.execute(DefaultTaskGenerator.java:116) ~[exchangis-job-server-1.1.2.jar:?]
4.      at com.webank.wedatasphere.exchangis.job.server.execution.generator.AbstractTaskGenerator.generate(AbstractTaskGenerator.java:63) ~[exchangis-job-server-1.1.2.jar:?]
5.      at com.webank.wedatasphere.exchangis.job.server.execution.generator.AbstractTaskGenerator.generate(AbstractTaskGenerator.java:24) ~[exchangis-job-server-1.1.2.jar:?]
6.      at com.webank.wedatasphere.exchangis.job.server.execution.scheduler.tasks.GenerationSchedulerTask.schedule(GenerationSchedulerTask.java:37) ~[exchangis-job-server-1.1.2.jar:?]
7.      at com.webank.wedatasphere.exchangis.job.server.execution.scheduler.AbstractExchangisSchedulerTask$DirectExecuteRequest.directExecute(AbstractExchangisSchedulerTask.java:92) ~[exchangis-job-server-1.1.2.jar:?]
8.      at com.webank.wedatasphere.exchangis.job.server.execution.scheduler.ExchangisSchedulerExecutorManager$DefaultDirectExecutor.execute(ExchangisSchedulerExecutorManager.java:166) ~[exchangis-job-server-1.1.2.jar:?]
9.      at org.apache.linkis.scheduler.queue.Job.$anonfun$run$2(Job.scala:315) ~[linkis-scheduler-1.4.0.jar:1.4.0]
10.     at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:49) ~[linkis-common-1.4.0.jar:1.4.0]
11.     at org.apache.linkis.scheduler.queue.Job.run(Job.scala:315) ~[linkis-scheduler-1.4.0.jar:1.4.0]
12.     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_321]
13.     at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_321]
14.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_321]
15.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_321]
16.     at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_321]
17.   Caused by: com.webank.wedatasphere.exchangis.job.exception.ExchangisJobException: errCode: 31998 ,desc: Fail to build transformJob from input job, message: [Duplicate key 二进制] ,ip: dss.prod.sh ,port: 9080 ,serviceKind: dss-exchangis-server
18.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.GenericExchangisTransformJobBuilder.buildJob(GenericExchangisTransformJobBuilder.java:140) ~[exchangis-job-server-1.1.2.jar:?]
19.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.GenericExchangisTransformJobBuilder.buildJob(GenericExchangisTransformJobBuilder.java:31) ~[exchangis-job-server-1.1.2.jar:?]
20.     at com.webank.wedatasphere.exchangis.job.builder.api.AbstractExchangisJobBuilder.build(AbstractExchangisJobBuilder.java:50) ~[exchangis-job-builder-1.1.2.jar:?]
21.     at com.webank.wedatasphere.exchangis.job.builder.api.GenericExchangisJobBuilderChain.build(GenericExchangisJobBuilderChain.java:40) ~[exchangis-job-builder-1.1.2.jar:?]
22.     at com.webank.wedatasphere.exchangis.job.builder.manager.DefaultExchangisJobBuilderManager.doBuild(DefaultExchangisJobBuilderManager.java:39) ~[exchangis-job-builder-1.1.2.jar:?]
23.     at com.webank.wedatasphere.exchangis.job.builder.manager.DefaultExchangisJobBuilderManager.doBuild(DefaultExchangisJobBuilderManager.java:30) ~[exchangis-job-builder-1.1.2.jar:?]
24.     at com.webank.wedatasphere.exchangis.job.server.execution.generator.DefaultTaskGenerator.execute(DefaultTaskGenerator.java:91) ~[exchangis-job-server-1.1.2.jar:?]
25.     ... 13 more
26.   Caused by: java.lang.IllegalStateException: Duplicate key 二进制
27.     at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_321]
28.     at java.util.HashMap.merge(HashMap.java:1256) ~[?:1.8.0_321]
29.     at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_321]
30.     at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_321]
31.     at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_321]
32.     at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_321]
33.     at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_321]
34.     at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_321]
35.     at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_321]
36.     at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_321]
37.     at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_321]
38.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.TransformExchangisJob$TransformSubExchangisJob.lambda$convertContentToParams$5(TransformExchangisJob.java:167) ~[exchangis-job-server-1.1.2.jar:?]
39.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.TransformExchangisJob$TransformSubExchangisJob.setIntoParams(TransformExchangisJob.java:201) ~[exchangis-job-server-1.1.2.jar:?]
40.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.TransformExchangisJob$TransformSubExchangisJob.convertContentToParams(TransformExchangisJob.java:164) ~[exchangis-job-server-1.1.2.jar:?]
41.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.TransformExchangisJob$TransformSubExchangisJob.<init>(TransformExchangisJob.java:79) ~[exchangis-job-server-1.1.2.jar:?]
42.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.GenericExchangisTransformJobBuilder.lambda$buildJob$6(GenericExchangisTransformJobBuilder.java:91) ~[exchangis-job-server-1.1.2.jar:?]
43.     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_321]
44.     at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_321]
45.     at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_321]
46.     at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_321]
47.     at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_321]
48.     at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_321]
49.     at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_321]
50.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.GenericExchangisTransformJobBuilder.buildJob(GenericExchangisTransformJobBuilder.java:97) ~[exchangis-job-server-1.1.2.jar:?]
51.     at com.webank.wedatasphere.exchangis.job.server.builder.transform.GenericExchangisTransformJobBuilder.buildJob(GenericExchangisTransformJobBuilder.java:31) ~[exchangis-job-server-1.1.2.jar:?]
52.     at com.webank.wedatasphere.exchangis.job.builder.api.AbstractExchangisJobBuilder.build(AbstractExchangisJobBuilder.java:50) ~[exchangis-job-builder-1.1.2.jar:?]
53.     at com.webank.wedatasphere.exchangis.job.builder.api.GenericExchangisJobBuilderChain.build(GenericExchangisJobBuilderChain.java:40) ~[exchangis-job-builder-1.1.2.jar:?]
54.     at com.webank.wedatasphere.exchangis.job.builder.manager.DefaultExchangisJobBuilderManager.doBuild(DefaultExchangisJobBuilderManager.java:39) ~[exchangis-job-builder-1.1.2.jar:?]
55.     at com.webank.wedatasphere.exchangis.job.builder.manager.DefaultExchangisJobBuilderManager.doBuild(DefaultExchangisJobBuilderManager.java:30) ~[exchangis-job-builder-1.1.2.jar:?]
56.     at com.webank.wedatasphere.exchangis.job.server.execution.generator.DefaultTaskGenerator.execute(DefaultTaskGenerator.java:91) ~[exchangis-job-server-1.1.2.jar:?]
57.     ... 13 more
58.

传输方式出现两个

Relevent platform

win 11 chrome78

Reproduction script

没有

Anything else

No response

Are you willing to submit a PR?

xuliancheng563 commented 9 months ago

脚本导入错误