search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
11.84k stars 3.12k forks source link

spark-doris-connector connect doris http-server-v2 failed, http-server-v1 is ok. #6288

Open eyewire opened 2 years ago

eyewire commented 2 years ago

Describe the bug spark-doris-connector connect doris http-server-v2 failed

To Reproduce use spark-doris-connector read from doris, http-server-v2 is set true, cannot read anything.but when we set to http-server-v1 is ok.

Expected behavior A clear and concise description of what you expected to happen.

Screenshots image

Additional context please fix this problem, because we also want to use flink-doris-connector to connect doris, but flink-doris-connector cannot use in http-server-v1, so we met a huge problem. please help~ thx~

JNSimba commented 2 years ago

Can you show code or detail log ? it work normally for me

hf200012 commented 2 years ago

https://github.com/apache/incubator-doris/pull/5788 Use doris-0.14.12.4 version

eyewire commented 2 years ago

Can you show code or detail log ? it work normally for me

we upgrade our doris version ,but spark-doris-connector get error message:

21/07/22 08:57:46 ERROR SchemaUtils: Doris FE's response cannot map to schema. res: {"msg":"Unauthorized","code":401,"data":"Need auth information.","count":0} org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "msg" (Class org.apache.doris.spark.rest.models.Schema), not marked as ignorable at [Source: java.io.StringReader@55d9b8f0; line: 1, column: 9] (through reference chain: org.apache.doris.spark.rest.models.Schema["msg"]) at org.codehaus.jackson.map.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:53) at org.codehaus.jackson.map.deser.StdDeserializationContext.unknownFieldException(StdDeserializationContext.java:267) at org.codehaus.jackson.map.deser.std.StdDeserializer.reportUnknownProperty(StdDeserializer.java:673) at org.codehaus.jackson.map.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:659) at org.codehaus.jackson.map.deser.BeanDeserializer.handleUnknownProperty(BeanDeserializer.java:1365) at org.codehaus.jackson.map.deser.BeanDeserializer._handleUnknown(BeanDeserializer.java:725) at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:703) at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580) at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732) at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863) at org.apache.doris.spark.rest.RestService.parseSchema(RestService.java:239) at org.apache.doris.spark.rest.RestService.getSchema(RestService.java:223) at org.apache.doris.spark.sql.SchemaUtils$.discoverSchemaFromFe(SchemaUtils.scala:51) at org.apache.doris.spark.sql.SchemaUtils$.discoverSchema(SchemaUtils.scala:41) at org.apache.doris.spark.sql.DorisRelation.lazySchema$lzycompute(DorisRelation.scala:48) at org.apache.doris.spark.sql.DorisRelation.lazySchema(DorisRelation.scala:48) at org.apache.doris.spark.sql.DorisRelation.schema(DorisRelation.scala:52) at org.apache.spark.sql.execution.datasources.LogicalRelation.(LogicalRelation.scala:40) at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:389) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125) at com.baidu.crowdtest.markdw.dbhelper.DbUtil.getDataFrame(DbUtil.java:128) at com.baidu.crowdtest.markdw.etl.step.stage.support.dev.generateStartDataFrame(dev.java:29) at com.baidu.crowdtest.markdw.etl.step.BaseSparkStep.runSparkStep(BaseSparkStep.java:242) at com.baidu.crowdtest.markdw.MainEntrance.main(MainEntrance.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

code is ： spark.read().format("doris") .option("user", "XXX") .option("password", "XXXX") .option("doris.table.identifier", "data_warehouse.ods_mark_task_config") .option("doris.fenodes", "XXXXXX:8030") // .option("doris.filter.query", "pro_id > 4000000 and pro_id < 4001000") .load();

JNSimba commented 2 years ago

your spark-doris-connector jar may be need update . #5366