hf200012
commented
3 years ago Have you started the flink sql shell to load the flink doris connector jar? This has not been tested with flink sql shell
Closed yangmingjie2018 closed 3 years ago
hf200012
commented
3 years ago Have you started the flink sql shell to load the flink doris connector jar? This has not been tested with flink sql shell
hf200012
commented
3 years ago final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
final StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
List<Tuple2<String, Integer>> data = new ArrayList<>();
data.add(new Tuple2<>("doris",1));
DataStreamSource<Tuple2<String, Integer>> source = env.fromCollection(data);
tEnv.createTemporaryView("doris_test",source,$("name"),$("age"));
tEnv.executeSql(
"CREATE TABLE doris_test_sink (" +
"name STRING," +
"age INT" +
") " +
"WITH (\n" +
" 'connector' = 'doris',\n" +
" 'fenodes' = 'fe_ip:8030',\n" +
" 'table.identifier' = 'db.tbl',\n" +
" 'username' = 'root',\n" +
" 'password' = ''\n" +
")");
tEnv.executeSql("select * from doris_test limit 20").print();
JNSimba
commented
3 years ago fenodes options port is HttpPort 8030
yangmingjie2018
commented
3 years ago 多谢 大神，上面的是端口错了，改成8030后可以使用flink获取数据了。
yangmingjie2018
commented
3 years ago 但是有产生一个新的问题，请大神指点。
JNSimba
commented
3 years ago Failed to get the BE information, can your account access the be information from the web page? like http://ip:8030/System?path=//backends
yangmingjie2018
commented
3 years ago 
yangmingjie2018
commented
3 years ago 
源脚本和目标脚本在flink中是可以取到数据的。
JNSimba
commented
3 years ago I know the problem. This version is not compatible with httpv1. If you want to use it, you need to enable httpv2, enable_http_server_v2=true， and httpv1 will add compatibility as soon as possible
yangmingjie2018
commented
3 years ago 大神 您说的是 配置 Doris 的 FE config，设置enable_http_server_v2=true 是吗？
JNSimba
commented
3 years ago yes
yangmingjie2018
commented
3 years ago 我修改了配置： FE config，设置enable_http_server_v2=true 重启了FE
但是还是报一样的错误。
JNSimba
commented
3 years ago Can you try to access this url？http://ip:8030/System?path=//backends
hf200012
commented
3 years ago The following method is no problem, it is not tested under Flink sql shell, you can try it by program，Later, we will test under the flink sql shell
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
final StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);// List<Tuple2<String, Integer>> data = new ArrayList<>(); // data.add(new Tuple2<>("doris",1)); // DataStreamSource<Tuple2<String, Integer>> source = env.fromCollection(data); // tEnv.createTemporaryView("doris_test",source,$("name"),$("age"));
tEnv.executeSql(
"CREATE TABLE test_aggregation01 (" +
"user_id STRING," +
"user_city STRING," +
"age INT," +
"last_visit_date STRING" +
") " +
"WITH (\n" +
" 'connector' = 'doris',\n" +
" 'fenodes' = 'doris01:8030',\n" +
" 'table.identifier' = 'demo.test_aggregation',\n" +
" 'username' = 'root',\n" +
" 'password' = ''\n" +
")");
tEnv.executeSql(
"CREATE TABLE test_aggregation02 (" +
"user_id STRING," +
"user_city STRING," +
"age INT," +
"last_visit_date STRING" +
") " +
"WITH (\n" +
" 'connector' = 'doris',\n" +
" 'fenodes' = 'doris01:8030',\n" +
" 'table.identifier' = 'demo.test_aggregation_01',\n" +
" 'username' = 'root',\n" +
" 'password' = ''\n" +
")");
tEnv.executeSql("INSERT INTO test_aggregation02 select * from test_aggregation01");"C:\Program Files\Java\jdk1.8.0_221\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2020.3.1\lib\idea_rt.jar=65445:C:\Program Files\JetBrains\IntelliJ IDEA 2020.3.1\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_221\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\rt.jar;C:\work\incubator-doris\extension\flink-doris-connector\target\test-classes;C:\work\incubator-doris\extension\flink-doris-connector\target\classes;C:\Users\zhang.ivy2\cache\org.scala-lang\scala-library\jars\scala-library-2.12.10.jar;C:\Users\zhang.ivy2\cache\org.scala-lang\scala-reflect\jars\scala-reflect-2.12.10.jar;C:\Users\zhang.ivy2\cache\org.scala-lang\scala-library\srcs\scala-library-2.12.10-sources.jar;C:\Users\zhang.m2\repository\io\netty\netty-all\4.1.63.Final\netty-all-4.1.63.Final.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-java\1.11.2\flink-java-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-core\1.11.2\flink-core-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-annotations\1.11.2\flink-annotations-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-metrics-core\1.11.2\flink-metrics-core-1.11.2.jar;C:\Users\zhang.m2\repository\com\esotericsoftware\kryo\kryo\2.24.0\kryo-2.24.0.jar;C:\Users\zhang.m2\repository\com\esotericsoftware\minlog\minlog\1.2\minlog-1.2.jar;C:\Users\zhang.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\zhang.m2\repository\org\apache\commons\commons-compress\1.20\commons-compress-1.20.jar;C:\Users\zhang.m2\repository\org\apache\commons\commons-lang3\3.3.2\commons-lang3-3.3.2.jar;C:\Users\zhang.m2\repository\org\apache\commons\commons-math3\3.5\commons-math3-3.5.jar;C:\Users\zhang.m2\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;C:\Users\zhang.m2\repository\org\apache\flink\force-shading\1.11.2\force-shading-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-streaming-java_2.12\1.11.2\flink-streaming-java_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-runtime_2.12\1.11.2\flink-runtime_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-queryable-state-client-java\1.11.2\flink-queryable-state-client-java-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-hadoop-fs\1.11.2\flink-hadoop-fs-1.11.2.jar;C:\Users\zhang.m2\repository\commons-io\commons-io\2.4\commons-io-2.4.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-shaded-netty\4.1.39.Final-11.0\flink-shaded-netty-4.1.39.Final-11.0.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-shaded-jackson\2.10.1-11.0\flink-shaded-jackson-2.10.1-11.0.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-shaded-zookeeper-3\3.4.14-11.0\flink-shaded-zookeeper-3-3.4.14-11.0.jar;C:\Users\zhang.m2\repository\org\javassist\javassist\3.24.0-GA\javassist-3.24.0-GA.jar;C:\Users\zhang.m2\repository\com\typesafe\akka\akka-actor_2.12\2.5.21\akka-actor_2.12-2.5.21.jar;C:\Users\zhang.m2\repository\com\typesafe\config\1.3.3\config-1.3.3.jar;C:\Users\zhang.m2\repository\org\scala-lang\modules\scala-java8-compat_2.12\0.8.0\scala-java8-compat_2.12-0.8.0.jar;C:\Users\zhang.m2\repository\com\typesafe\akka\akka-stream_2.12\2.5.21\akka-stream_2.12-2.5.21.jar;C:\Users\zhang.m2\repository\org\reactivestreams\reactive-streams\1.0.2\reactive-streams-1.0.2.jar;C:\Users\zhang.m2\repository\com\typesafe\ssl-config-core_2.12\0.3.7\ssl-config-core_2.12-0.3.7.jar;C:\Users\zhang.m2\repository\org\scala-lang\modules\scala-parser-combinators_2.12\1.1.1\scala-parser-combinators_2.12-1.1.1.jar;C:\Users\zhang.m2\repository\com\typesafe\akka\akka-protobuf_2.12\2.5.21\akka-protobuf_2.12-2.5.21.jar;C:\Users\zhang.m2\repository\com\typesafe\akka\akka-slf4j_2.12\2.5.21\akka-slf4j_2.12-2.5.21.jar;C:\Users\zhang.m2\repository\org\clapper\grizzled-slf4j_2.12\1.3.2\grizzled-slf4j_2.12-1.3.2.jar;C:\Users\zhang.m2\repository\com\github\scopt\scopt_2.12\3.5.0\scopt_2.12-3.5.0.jar;C:\Users\zhang.m2\repository\org\xerial\snappy\snappy-java\1.1.4\snappy-java-1.1.4.jar;C:\Users\zhang.m2\repository\com\twitter\chill_2.12\0.7.6\chill_2.12-0.7.6.jar;C:\Users\zhang.m2\repository\com\twitter\chill-java\0.7.6\chill-java-0.7.6.jar;C:\Users\zhang.m2\repository\org\lz4\lz4-java\1.6.0\lz4-java-1.6.0.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-shaded-guava\18.0-11.0\flink-shaded-guava-18.0-11.0.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-clients_2.12\1.11.2\flink-clients_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-optimizer_2.12\1.11.2\flink-optimizer_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\commons-cli\commons-cli\1.3.1\commons-cli-1.3.1.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-table-common\1.11.2\flink-table-common-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-shaded-asm-7\7.1-11.0\flink-shaded-asm-7-7.1-11.0.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-table-api-java-bridge_2.12\1.11.2\flink-table-api-java-bridge_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-table-api-java\1.11.2\flink-table-api-java-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-table-planner-blink_2.12\1.11.2\flink-table-planner-blink_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-table-api-scala_2.12\1.11.2\flink-table-api-scala_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-table-api-scala-bridge_2.12\1.11.2\flink-table-api-scala-bridge_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-scala_2.12\1.11.2\flink-scala_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-streaming-scala_2.12\1.11.2\flink-streaming-scala_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\apache\flink\flink-table-runtime-blink_2.12\1.11.2\flink-table-runtime-blink_2.12-1.11.2.jar;C:\Users\zhang.m2\repository\org\codehaus\janino\janino\3.0.9\janino-3.0.9.jar;C:\Users\zhang.m2\repository\org\codehaus\janino\commons-compiler\3.0.9\commons-compiler-3.0.9.jar;C:\Users\zhang.m2\repository\org\apache\calcite\avatica\avatica-core\1.16.0\avatica-core-1.16.0.jar;C:\Users\zhang.m2\repository\org\reflections\reflections\0.9.10\reflections-0.9.10.jar;C:\Users\zhang.m2\repository\org\scala-lang\scala-library\2.12.8\scala-library-2.12.8.jar;C:\Users\zhang.m2\repository\org\scala-lang\scala-reflect\2.12.8\scala-reflect-2.12.8.jar;C:\Users\zhang.m2\repository\org\scala-lang\scala-compiler\2.12.8\scala-compiler-2.12.8.jar;C:\Users\zhang.m2\repository\org\scala-lang\modules\scala-xml_2.12\1.0.6\scala-xml_2.12-1.0.6.jar;C:\Users\zhang.m2\repository\org\apache\thrift\libthrift\0.9.3\libthrift-0.9.3.jar;C:\Users\zhang.m2\repository\org\apache\httpcomponents\httpclient\4.4.1\httpclient-4.4.1.jar;C:\Users\zhang.m2\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;C:\Users\zhang.m2\repository\org\apache\httpcomponents\httpcore\4.4.1\httpcore-4.4.1.jar;C:\Users\zhang.m2\repository\org\apache\arrow\arrow-vector\0.15.1\arrow-vector-0.15.1.jar;C:\Users\zhang.m2\repository\org\apache\arrow\arrow-format\0.15.1\arrow-format-0.15.1.jar;C:\Users\zhang.m2\repository\org\apache\arrow\arrow-memory\0.15.1\arrow-memory-0.15.1.jar;C:\Users\zhang.m2\repository\io\netty\netty-buffer\4.1.27.Final\netty-buffer-4.1.27.Final.jar;C:\Users\zhang.m2\repository\com\fasterxml\jackson\core\jackson-core\2.9.8\jackson-core-2.9.8.jar;C:\Users\zhang.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.9.8\jackson-annotations-2.9.8.jar;C:\Users\zhang.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.9.8\jackson-databind-2.9.8.jar;C:\Users\zhang.m2\repository\commons-codec\commons-codec\1.10\commons-codec-1.10.jar;C:\Users\zhang.m2\repository\io\netty\netty-common\4.1.27.Final\netty-common-4.1.27.Final.jar;C:\Users\zhang.m2\repository\com\google\flatbuffers\flatbuffers-java\1.9.0\flatbuffers-java-1.9.0.jar;C:\Users\zhang.m2\repository\org\slf4j\slf4j-api\1.7.25\slf4j-api-1.7.25.jar;C:\Users\zhang.m2\repository\org\slf4j\slf4j-log4j12\1.7.25\slf4j-log4j12-1.7.25.jar;C:\Users\zhang.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;C:\Users\zhang.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;C:\Users\zhang.m2\repository\org\mockito\mockito-scala_2.12\1.4.7\mockito-scala_2.12-1.4.7.jar;C:\Users\zhang.m2\repository\org\mockito\mockito-core\2.27.0\mockito-core-2.27.0.jar;C:\Users\zhang.m2\repository\net\bytebuddy\byte-buddy\1.9.10\byte-buddy-1.9.10.jar;C:\Users\zhang.m2\repository\net\bytebuddy\byte-buddy-agent\1.9.10\byte-buddy-agent-1.9.10.jar;C:\Users\zhang.m2\repository\org\objenesis\objenesis\2.6\objenesis-2.6.jar;C:\Users\zhang.m2\repository\org\scalactic\scalactic_2.12\3.0.8-RC4\scalactic_2.12-3.0.8-RC4.jar;C:\Users\zhang.m2\repository\ru\vyarus\generics-resolver\3.0.0\generics-resolver-3.0.0.jar;C:\Users\zhang.m2\repository\junit\junit\4.11\junit-4.11.jar" org.apache.doris.flink.DorisSinkExample 16:30:37,300 INFO org.apache.doris.flink.table.DorisRowDataInputFormat - Send request to Doris FE 'http://doris01:8030/api/demo/test_aggregation/_query_plan' with user 'root'. 16:30:41,834 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.cpu.cores required for local execution is not set, setting it to the maximal possible value. 16:30:41,834 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.task.heap.size required for local execution is not set, setting it to the maximal possible value. 16:30:41,834 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.task.off-heap.size required for local execution is not set, setting it to the maximal possible value. 16:30:41,843 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.network.min required for local execution is not set, setting it to its default value 64 mb. 16:30:41,844 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.network.max required for local execution is not set, setting it to its default value 64 mb. 16:30:41,845 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.managed.size required for local execution is not set, setting it to its default value 128 mb. 16:30:42,005 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting Flink Mini Cluster 16:30:42,025 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting Metrics Registry 16:30:42,425 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 16:30:42,425 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting RPC Service(s) 16:30:42,496 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start local actor system 16:30:44,954 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 16:30:46,043 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka://flink 16:30:46,105 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start local actor system 16:30:46,177 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 16:30:46,574 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka://flink-metrics 16:30:46,623 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService . 16:30:47,025 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting high-availability services 16:30:48,167 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory C:\Users\zhang\AppData\Local\Temp\blobStore-79ae950f-0184-4ff5-ba65-67c4f2dd9f98 16:30:48,194 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:65454 - max concurrent requests: 50 - max backlog: 1000 16:30:48,220 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory C:\Users\zhang\AppData\Local\Temp\blobStore-406bbf0a-1024-40e6-8453-5b3bb93e322b 16:30:48,224 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory C:\Users\zhang\AppData\Local\Temp\blobStore-508f93c4-4c52-465f-a170-3908dc32c811 16:30:48,225 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting 1 TaskManger(s) 16:30:48,238 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager with ResourceID: 6ce238d6-9736-4471-97b4-11a3e78749a8 16:30:48,339 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory 'C:\Users\zhang\AppData\Local\Temp': total 236 GB, usable 72 GB (30.51% usable) 16:30:48,360 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager uses directory C:\Users\zhang\AppData\Local\Temp\flink-io-8d616306-fe31-4a4c-89cc-bee043ff453f for spill files. 16:30:48,422 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager uses directory C:\Users\zhang\AppData\Local\Temp\flink-netty-shuffle-be0c2368-c1b1-4881-8ffd-76cef62884f7 for spill files. 16:30:48,618 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768). 16:30:48,647 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment - Starting the network environment and its components. 16:30:48,654 INFO org.apache.flink.runtime.taskexecutor.KvStateService - Starting the kvState service and its components. 16:30:48,791 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/rpc/taskmanager_0 . 16:30:48,858 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Start job leader service. 16:30:48,877 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory C:\Users\zhang\AppData\Local\Temp\flink-dist-cache-f04b233e-25df-44d4-aa81-b44be8620988 16:30:49,139 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint. 16:30:49,148 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Failed to load web based job submission extension. Probable reason: flink-runtime-web is not in the classpath. 16:30:50,649 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set. 16:30:50,649 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'web.log.path'. 16:30:53,304 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at localhost:65473 16:30:53,304 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender http://localhost:65473 16:30:53,313 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://localhost:65473 was granted leadership with leaderSessionID=643f1980-b1f0-47ab-9c9c-dcf6a371f53d 16:30:53,315 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader http://localhost:65473 , session=643f1980-b1f0-47ab-9c9c-dcf6a371f53d 16:30:53,380 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/rpc/resourcemanager_1 . 16:30:53,477 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender LeaderContender: DefaultDispatcherRunner 16:30:53,480 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender LeaderContender: StandaloneResourceManager 16:30:53,501 INFO org.apache.flink.runtime.minicluster.MiniCluster - Flink Mini Cluster started successfully 16:30:53,506 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka://flink/user/rpc/resourcemanager_1 was granted leadership with fencing token bdcccb6c32c2148a1c32da14150e4ccb 16:30:53,545 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl - Starting the SlotManager. 16:30:53,593 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader akka://flink/user/rpc/resourcemanager_1 , session=1c32da14-150e-4ccb-bdcc-cb6c32c2148a 16:30:53,605 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Start SessionDispatcherLeaderProcess. 16:30:53,613 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(bdcccb6c32c2148a1c32da14150e4ccb). 16:30:53,615 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Recover all persisted job graphs. 16:30:53,618 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Successfully recovered 0 persisted job graphs. 16:30:53,741 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_2 . 16:30:53,912 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader akka://flink/user/rpc/dispatcher_2 , session=36f448e4-7b57-4c04-a19c-f0c03dfe4af8 16:30:53,933 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration 16:30:53,961 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering TaskManager with ResourceID 6ce238d6-9736-4471-97b4-11a3e78749a8 (akka://flink/user/rpc/taskmanager_0) at ResourceManager 16:30:54,146 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka://flink/user/rpc/resourcemanager_1 under registration id ff475314bf085072594363af946f387c. 16:30:54,173 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Received JobGraph submission cf774905ee923911a7ce2d0424f9dbff (insert-into_default_catalog.default_database.test_aggregation02). 16:30:54,175 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job cf774905ee923911a7ce2d0424f9dbff (insert-into_default_catalog.default_database.test_aggregation02). 16:30:54,301 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_3 . 16:30:54,333 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff). 16:30:54,398 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart back off time strategy NoRestartBackoffTimeStrategy for insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff). 16:30:54,561 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff). 16:30:54,574 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 13 ms. 16:30:54,605 INFO org.apache.doris.flink.table.DorisRowDataInputFormat - DorisTableInputSplit Num:4 16:30:54,636 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology - Built 1 pipelined regions in 5 ms 16:30:54,697 INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 16:30:54,761 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@1cf1a70c for insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff). 16:30:54,770 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender akka://flink/user/rpc/jobmanager_3 16:30:54,775 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl - JobManager runner for job insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff) was granted leadership with session id 4b2e4cdd-5de5-4e72-8418-91eae52ee336 at akka://flink/user/rpc/jobmanager_3. 16:30:54,790 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff) under job master id 841891eae52ee3364b2e4cdd5de54e72. 16:30:54,795 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.EagerSchedulingStrategy] 16:30:54,797 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff) switched from state CREATED to RUNNING. 16:30:54,837 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) switched from CREATED to SCHEDULED. 16:30:54,944 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{b3867d9ffca58e12d459355a7e898164}] 16:30:54,988 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader akka://flink/user/rpc/jobmanager_3 , session=4b2e4cdd-5de5-4e72-8418-91eae52ee336 16:30:54,989 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(bdcccb6c32c2148a1c32da14150e4ccb) 16:30:54,997 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration 16:30:55,002 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering job manager 841891eae52ee3364b2e4cdd5de54e72@akka://flink/user/rpc/jobmanager_3 for job cf774905ee923911a7ce2d0424f9dbff. 16:30:55,026 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registered job manager 841891eae52ee3364b2e4cdd5de54e72@akka://flink/user/rpc/jobmanager_3 for job cf774905ee923911a7ce2d0424f9dbff. 16:30:55,036 INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: bdcccb6c32c2148a1c32da14150e4ccb. 16:30:55,037 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{b3867d9ffca58e12d459355a7e898164}] and profile ResourceProfile{UNKNOWN} from resource manager. 16:30:55,203 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Request slot with profile ResourceProfile{UNKNOWN} for job cf774905ee923911a7ce2d0424f9dbff with allocation id 5aef532e1c77ec84c783e683c076e3c2. 16:30:55,222 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Receive slot request 5aef532e1c77ec84c783e683c076e3c2 for job cf774905ee923911a7ce2d0424f9dbff from resource manager with leader id bdcccb6c32c2148a1c32da14150e4ccb. 16:30:55,252 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot for 5aef532e1c77ec84c783e683c076e3c2. 16:30:55,256 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Add job cf774905ee923911a7ce2d0424f9dbff for job leader monitoring. 16:30:55,277 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Try to register at job manager akka://flink/user/rpc/jobmanager_3 with leader id 4b2e4cdd-5de5-4e72-8418-91eae52ee336. 16:30:55,280 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Resolved JobManager address, beginning registration 16:30:55,287 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation - No hostname could be resolved for the IP address 127.0.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 16:30:55,302 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Successful registration at job manager akka://flink/user/rpc/jobmanager_3 for job cf774905ee923911a7ce2d0424f9dbff. 16:30:55,305 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Establish JobManager connection for job cf774905ee923911a7ce2d0424f9dbff. 16:30:55,317 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Offer reserved slots to the leader of job cf774905ee923911a7ce2d0424f9dbff. 16:30:55,332 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) switched from SCHEDULED to DEPLOYING. 16:30:55,336 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (attempt #0) to 6ce238d6-9736-4471-97b4-11a3e78749a8 @ 127.0.0.1 (dataPort=-1) 16:30:55,354 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Activate slot 5aef532e1c77ec84c783e683c076e3c2. 16:30:55,505 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1). 16:30:55,537 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Activate slot 5aef532e1c77ec84c783e683c076e3c2. 16:30:55,550 INFO org.apache.flink.runtime.taskmanager.Task - Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) switched from CREATED to DEPLOYING. 16:30:55,578 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) [DEPLOYING]. 16:30:55,584 INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) [DEPLOYING]. 16:30:55,922 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 16:30:55,954 INFO org.apache.flink.runtime.taskmanager.Task - Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) switched from DEPLOYING to RUNNING. 16:30:55,957 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) switched from DEPLOYING to RUNNING. 16:30:56,033 WARN org.apache.flink.metrics.MetricGroup - The operator name Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) exceeded the 80 characters length limit and was truncated. 16:30:56,066 WARN org.apache.flink.metrics.MetricGroup - The operator name Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) exceeded the 80 characters length limit and was truncated. 16:30:56,163 INFO org.apache.doris.flink.table.DorisDynamicOutputFormat - Send request to Doris FE 'http://doris01:8030/rest/v1/system?path=//backends' with user 'root'. 16:30:56,403 INFO org.apache.doris.flink.table.DorisDynamicOutputFormat - Backend Info:{"href_columns":["BackendId"],"parent_url":"/rest/v1/system?path=/","column_names":["BackendId","Cluster","IP","HostName","HeartbeatPort","BePort","HttpPort","BrpcPort","LastStartTime","LastHeartbeat","Alive","SystemDecommissioned","ClusterDecommissioned","TabletNum","DataUsedCapacity","AvailCapacity","TotalCapacity","UsedPct","MaxDiskUsedPct","ErrMsg","Version","Status"],"rows":[{"HttpPort":"8040","Status":"{\"lastSuccessReportTabletsTime\":\"2021-04-26 16:30:16\"}","SystemDecommissioned":"false","LastHeartbeat":"2021-04-26 16:30:54","DataUsedCapacity":"9.646 GB","ErrMsg":"","IP":"10.220.146.11","UsedPct":"1.76 %","hrefPaths":["/rest/v1/system?path=//backends/10002"],"Cluster":"default_cluster","Alive":"true","MaxDiskUsedPct":"1.76 %","BrpcPort":"8060","BePort":"9060","ClusterDecommissioned":"false","AvailCapacity":"1.962 TB","Version":"0.13.15-Unknown","BackendId":"10002","HeartbeatPort":"9050","LastStartTime":"2021-03-20 16:33:39","TabletNum":"424107","TotalCapacity":"1.997 TB","HostName":"doris02"},{"HttpPort":"8040","Status":"{\"lastSuccessReportTabletsTime\":\"2021-04-26 16:30:17\"}","SystemDecommissioned":"false","LastHeartbeat":"2021-04-26 16:30:54","DataUsedCapacity":"8.954 GB","ErrMsg":"","IP":"10.220.146.12","UsedPct":"1.47 %","__hrefPaths":["/rest/v1/system?path=//backends/10003"],"Cluster":"default_cluster","Alive":"true","MaxDiskUsedPct":"1.47 %","BrpcPort":"8060","BePort":"9060","ClusterDecommissioned":"false","AvailCapacity":"1.968 TB","Version":"0.13.15-Unknown","BackendId":"10003","HeartbeatPort":"9050","LastStartTime":"2021-03-20 16:32:56","TabletNum":"423146","TotalCapacity":"1.997 TB","HostName":"doris03"},{"HttpPort":"8040","Status":"{\"lastSuccessReportTabletsTime\":\"2021-04-26 16:29:57\"}","SystemDecommissioned":"false","LastHeartbeat":"2021-04-26 16:30:54","DataUsedCapacity":"9.733 GB","ErrMsg":"","IP":"10.220.146.13","UsedPct":"1.35 %","hrefPaths":["/rest/v1/system?path=//backends/10004"],"Cluster":"default_cluster","Alive":"true","MaxDiskUsedPct":"1.35 %","BrpcPort":"8060","BePort":"9060","ClusterDecommissioned":"false","AvailCapacity":"1.970 TB","Version":"0.13.15-Unknown","BackendId":"10004","HeartbeatPort":"9050","LastStartTime":"2021-03-24 15:25:19","TabletNum":"421157","TotalCapacity":"1.997 TB","HostName":"doris04"},{"HttpPort":"8040","Status":"{\"lastSuccessReportTabletsTime\":\"2021-04-26 16:30:27\"}","SystemDecommissioned":"false","LastHeartbeat":"2021-04-26 16:30:54","DataUsedCapacity":"11.326 GB","ErrMsg":"","IP":"10.220.146.14","UsedPct":"1.23 %","hrefPaths":["/rest/v1/system?path=//backends/22644890"],"Cluster":"default_cluster","Alive":"true","MaxDiskUsedPct":"1.23 %","BrpcPort":"8060","BePort":"9060","ClusterDecommissioned":"false","AvailCapacity":"1.972 TB","Version":"0.13.15-Unknown","BackendId":"22644890","HeartbeatPort":"9050","LastStartTime":"2021-03-20 16:22:56","TabletNum":"342718","TotalCapacity":"1.997 TB","HostName":"doris05"},{"HttpPort":"8040","Status":"{\"lastSuccessReportTabletsTime\":\"2021-04-26 16:30:27\"}","SystemDecommissioned":"false","LastHeartbeat":"2021-04-26 16:30:54","DataUsedCapacity":"9.756 GB","ErrMsg":"","IP":"10.220.146.15","UsedPct":"1.50 %","__hrefPaths":["/rest/v1/system?path=//backends/10006"],"Cluster":"default_cluster","Alive":"true","MaxDiskUsedPct":"1.50 %","BrpcPort":"8060","BePort":"9060","ClusterDecommissioned":"false","AvailCapacity":"1.968 TB","Version":"0.13.15-Unknown","BackendId":"10006","HeartbeatPort":"9050","LastStartTime":"2021-03-24 15:24:37","TabletNum":"417756","TotalCapacity":"1.998 TB","HostName":"doris06"},{"HttpPort":"8040","Status":"{\"lastSuccessReportTabletsTime\":\"2021-04-26 16:30:12\"}","SystemDecommissioned":"false","LastHeartbeat":"2021-04-26 16:30:54","DataUsedCapacity":"9.307 GB","ErrMsg":"","IP":"10.220.146.16","UsedPct":"1.50 %","hrefPaths":["/rest/v1/system?path=//backends/10007"],"Cluster":"default_cluster","Alive":"true","MaxDiskUsedPct":"1.50 %","BrpcPort":"8060","BePort":"9060","ClusterDecommissioned":"false","AvailCapacity":"1.967 TB","Version":"0.13.15-Unknown","BackendId":"10007","HeartbeatPort":"9050","LastStartTime":"2021-03-20 16:40:20","TabletNum":"416876","TotalCapacity":"1.997 TB","HostName":"doris07"}]} 16:30:56,435 INFO org.apache.doris.flink.table.DorisDynamicOutputFormat - Steamload BE:http://10.220.146.12:8040/api/demo/test_aggregation_01/_stream_load? 16:30:58,651 INFO org.apache.doris.flink.table.DorisStreamLoad - Streamload Response:status: 200, resp msg: OK, resp content: { "TxnId": 3532947, "Label": "audit_20210426_163058_a502149581e24e4a8624ee8977d4cf7c", "Status": "Success", "Message": "OK", "NumberTotalRows": 3, "NumberLoadedRows": 3, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 95, "LoadTimeMs": 115, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 1, "ReadDataTimeMs": 0, "WriteDataTimeMs": 7, "CommitAndPublishTimeMs": 105} 16:30:58,672 INFO org.apache.flink.runtime.taskmanager.Task - Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) switched from RUNNING to FINISHED. 16:30:58,673 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef). 16:30:58,677 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FINISHED to JobManager for task Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) e17d6325c9d470de3a9b727f58e72fef. 16:30:58,713 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: TableSourceScan(table=[[default_catalog, default_database, test_aggregation01]], fields=[user_id, user_city, age, last_visit_date]) -> Sink: Sink(table=[default_catalog.default_database.test_aggregation02], fields=[user_id, user_city, age, last_visit_date]) (1/1) (e17d6325c9d470de3a9b727f58e72fef) switched from RUNNING to FINISHED. 16:30:58,723 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job insert-into_default_catalog.default_database.test_aggregation02 (cf774905ee923911a7ce2d0424f9dbff) switched from state RUNNING to FINISHED. 16:30:58,724 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job cf774905ee923911a7ce2d0424f9dbff. 16:30:58,732 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down 16:30:58,763 INFO org.apache.flink.runtime.minicluster.MiniCluster - Shutting down Flink Mini Cluster 16:30:58,765 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Shutting down rest endpoint. 16:30:58,766 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopping TaskExecutor akka://flink/user/rpc/taskmanager_0. 16:30:58,766 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close ResourceManager connection 79768d51d0191a762f53e579c206dbca. 16:30:58,768 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job cf774905ee923911a7ce2d0424f9dbff reached globally terminal state FINISHED. 16:30:58,773 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close JobManager connection for job cf774905ee923911a7ce2d0424f9dbff. 16:30:58,775 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Closing TaskExecutor connection 6ce238d6-9736-4471-97b4-11a3e78749a8 because: The TaskExecutor is shutting down. 16:30:58,787 INFO org.apache.flink.runtime.jobmaster.JobMaster - Stopping the JobMaster for job insert-into_default_catalog.default_database.test_aggregation02(cf774905ee923911a7ce2d0424f9dbff). 16:30:58,804 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Free slot TaskSlot(index:0, state:ALLOCATED, resource profile: ResourceProfile{managedMemory=128.000mb (134217728 bytes), networkMemory=64.000mb (67108864 bytes)}, allocationId: 5aef532e1c77ec84c783e683c076e3c2, jobId: cf774905ee923911a7ce2d0424f9dbff). 16:30:59,004 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Suspending SlotPool. 16:30:59,004 INFO org.apache.flink.runtime.jobmaster.JobMaster - Close ResourceManager connection 79768d51d0191a762f53e579c206dbca: JobManager is shutting down.. 16:30:59,005 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Stopping SlotPool. 16:30:59,012 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Disconnect job manager 841891eae52ee3364b2e4cdd5de54e72@akka://flink/user/rpc/jobmanager_3 for job cf774905ee923911a7ce2d0424f9dbff from the resource manager. 16:30:59,174 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - JobManager for job cf774905ee923911a7ce2d0424f9dbff with leader id 841891eae52ee3364b2e4cdd5de54e72 lost leadership. 16:30:59,176 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Stop job leader service. 16:30:59,190 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager - Shutting down TaskExecutorLocalStateStoresManager. 16:30:59,198 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Removing cache directory C:\Users\zhang\AppData\Local\Temp\flink-web-ui 16:30:59,200 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Shut down complete. 16:30:59,202 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager removed spill file directory C:\Users\zhang\AppData\Local\Temp\flink-io-8d616306-fe31-4a4c-89cc-bee043ff453f 16:30:59,203 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment - Shutting down the network environment and its components. 16:30:59,208 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Shut down cluster because application is in CANCELED, diagnostics DispatcherResourceManagerComponent has been closed.. 16:30:59,209 INFO org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent - Closing components. 16:30:59,215 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Stopping SessionDispatcherLeaderProcess. 16:30:59,226 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopping dispatcher akka://flink/user/rpc/dispatcher_2. 16:30:59,227 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopping all currently running jobs of dispatcher akka://flink/user/rpc/dispatcher_2. 16:30:59,227 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator - Shutting down back pressure request coordinator. 16:30:59,228 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopped dispatcher akka://flink/user/rpc/dispatcher_2. 16:30:59,229 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl - Closing the SlotManager. 16:30:59,229 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl - Suspending the SlotManager. 16:30:59,288 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager removed spill file directory C:\Users\zhang\AppData\Local\Temp\flink-netty-shuffle-be0c2368-c1b1-4881-8ffd-76cef62884f7 16:30:59,290 INFO org.apache.flink.runtime.taskexecutor.KvStateService - Shutting down the kvState service and its components. 16:30:59,295 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Stop job leader service. 16:30:59,297 INFO org.apache.flink.runtime.filecache.FileCache - removed file cache directory C:\Users\zhang\AppData\Local\Temp\flink-dist-cache-f04b233e-25df-44d4-aa81-b44be8620988 16:30:59,386 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopped TaskExecutor akka://flink/user/rpc/taskmanager_0. 16:30:59,390 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service. 16:30:59,885 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service. 16:30:59,885 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service. 16:30:59,903 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Shutting down BLOB cache 16:30:59,906 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache 16:30:59,911 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:65454 16:30:59,912 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service.
Process finished with exit code 0
hf200012
commented
3 years ago After configuring http v2, FE needs to restart
yangmingjie2018
commented
3 years ago 感谢 两位大神，FE config，设置enable_http_server_v2=true，但是在UI界面中的doris:8030/variable中并没有查到，也就是没有起效。我们改完配置后，重启了FE，甚至重启了一次虚拟机。但是，还是不起效。
yangmingjie2018
commented
3 years ago 我们的doris 0.13版本 好像是没有这个配置项，0.13是不是还有其他小版本，有这个配置项。
yangmingjie2018
commented
3 years ago 我在官网上没有找到0.13的其他版本，但是晚上的一些论坛里有0.13.15的说明。咱们的这个connector 是否支持 doris 0.14？
hf200012
commented
3 years ago 我在官网上没有找到0.13的其他版本，但是晚上的一些论坛里有0.13.15的说明。咱们的这个connector 是否支持 doris 0.14？
Supported
hf200012
commented
3 years ago 我们的doris 0.13版本 好像是没有这个配置项，0.13是不是还有其他小版本，有这个配置项。 There is this configuration item You add the following configuration in fe.conf, restart fe enable_http_server_v2 = true
yangmingjie2018
commented
3 years ago 感谢两位大神 @hf200012 @JNSimba 这么细致的指导！
使用代码提交应该是没有问题。
使用flink client提交，我们将doris升级到0.14后enable_http_server_v2 = true可以设置了，设置后，插入操作也成功了！
maker-dong
commented
3 years ago 您好，我用的0.13版本，跟你遇到了同样的问题，我在FE中添加了enable_http_server_v2 = true，已经可以通过FlinkSQL查询Doris数据库，并且成功提交insert任务，但是并没有数据写入doris表，查看任务也没有报错，请问您知道是怎么回事吗？
非常激动，flink-doris-connector 终于合并到了master上线了。
我们今天尝试了一下，一直报错：
会是什么原因，网络是通的。