search

scalaspring / akka-spring-boot

Apache License 2.0
48 stars 12 forks source link

can not read input properties from cinfig file #7

Closed lordshenyf closed 5 years ago

lordshenyf commented 5 years ago

Hi:

I'm trying to read config file for waterdrop to dump HDFS data to clickhouse, but here showed some error: Exception StackTrace: com.typesafe.config.ConfigException$WrongType: batch.conf: 7: input has type OBJECT rather than LIST at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:133) at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:145) at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159) at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164) at com.typesafe.config.impl.SimpleConfig.getList(SimpleConfig.java:212) at com.typesafe.config.impl.SimpleConfig.getHomogeneousWrappedList(SimpleConfig.java:336) at com.typesafe.config.impl.SimpleConfig.getObjectList(SimpleConfig.java:354) at com.typesafe.config.impl.SimpleConfig.getConfigList(SimpleConfig.java:359) at io.github.interestinglab.waterdrop.config.ConfigBuilder.createStaticInputs(ConfigBuilder.scala:130) at io.github.interestinglab.waterdrop.Waterdrop$.io$github$interestinglab$waterdrop$Waterdrop$$entrypoint(Waterdrop.scala:104) at io.github.interestinglab.waterdrop.Waterdrop$$anonfun$1.apply$mcV$sp(Waterdrop.scala:35) at io.github.interestinglab.waterdrop.Waterdrop$$anonfun$1.apply(Waterdrop.scala:35) at io.github.interestinglab.waterdrop.Waterdrop$$anonfun$1.apply(Waterdrop.scala:35) at scala.util.Try$.apply(Try.scala:192) at io.github.interestinglab.waterdrop.Waterdrop$.main(Waterdrop.scala:35) at io.github.interestinglab.waterdrop.Waterdrop.main(Waterdrop.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)

this is the batch.config: spark { spark.app.name = "Waterdrop" spark.executor.instances = 20 spark.executor.cores = 10 spark.executor.memory = "4g" } input { hdfs { path = "hdfs://cbgcluster/user/iflyrd/work/yfshen2/ABtestAnal/hotfunction/userLabel/20190808" result_table_name = "user_label" format = "text" } } filter { split { fields = [ "pdate", "uid", "age", "fdf", "fver", "fdate", "ver", "df", "actdays", "lastactdate"] delimiter = "~" } } output { clickhouse { host = "10.100.108.100:8123" database = "imeandroid" table = "userlabel" fields = ["pdate", "uid", "age", "fdf", "fver", "fdate", "ver", "df", "actdays", "lastactdate"] username = "default" } }