DTStack / flinkStreamSQL

基于开源的flink,对其实时sql进行扩展;主要实现了流与维表的join,支持原生flink SQL所有的语法
Apache License 2.0
2.02k stars 926 forks source link

通过Yarn模式提交任务时报错Caused by: java.lang.ClassNotFoundException: com.dtstack.flink.sql.source.kafka.KafkaConsumer010 #265

Open lijianfeng6666526 opened 4 years ago

lijianfeng6666526 commented 4 years ago

### sql文件内容如下: CREATE TABLE sourceKafkaTable( id int as id, name varchar as name )WITH( type ='kafka10', bootstrapServers ='spark01:9092,spark02:9092', zookeeperQuorum ='spark01:2181,spark02:2181/brokers', offsetReset ='latest', topic ='flink1', parallelism ='1' );

CREATE TABLE sideMysqlTable( id int, cname varchar, PRIMARY KEY(id), PERIOD FOR SYSTEM_TIME )WITH( type='mysql', url='jdbc:mysql://192.168.181.15:3306/hive?charset=utf8', userName='root', password='6666526', tableName='side_order', cache ='LRU', cacheSize ='10000', cacheTTLMs ='60000', parallelism ='1', partitionedJoin='false' );

CREATE TABLE sinkMysqlTable( id int, name varchar, cname varchar )WITH( type ='mysql', url ='jdbc:mysql://192.168.181.15:3306/hive?charset=utf8', userName ='root', password ='6666526', tableName ='result_order', parallelism ='1' );

insert into sinkMysqlTable select d.id, d.name, d.cname from (select a.id,a.name,b.cname from sourceKafkaTable a join sideMysqlTable b on a.id=b.id) as d

================================================================ ### 提交的命令如下: sh submit.sh -sql /home/flinksql.txt -name flinkstream -remoteSqlPluginPath /home/FlinkStreamSQL-1.8.0-new/plugins -localSqlPluginPath /home/FlinkStreamSQL-1.8.0-new/plugins -mode yarn -yarnconf /home/hadoop-2.6.0/etc/hadoop -flinkconf /home/flink-1.8.0/conf -confProp {\"time.characteristic\":\"EventTime\",\"sql.checkpoint.interval\":10000} -yarnSessionConf {\"yid\":\"application_1582170401251_0003\"}

================================================================ ### 报如下错误: 图片

todd5167 commented 4 years ago

@lijianfeng6666526 你用yarnPer 和 -pluginLoadMode shipfile 模式试试,yarnsession模式我们的测试的比较少。

lijianfeng6666526 commented 4 years ago

用yarnPer会保其他错误

hengyuan commented 4 years ago

遇到同样的问题,yarnPer无法正常运行报错和上面是一样的。 -pluginLoadMode shipfile