Task located in hdfs doesn't work

spring-attic / spring-cloud-dataflow-server-yarn

Spring Cloud Data Flow Implementation for Apache YARN

http://cloud.spring.io/spring-cloud-dataflow-server-yarn/

Apache License 2.0

15 stars 34 forks source link

Task located in hdfs doesn't work #169

Closed jvalkeal closed 7 years ago

jvalkeal commented 7 years ago

Launching a task which is defined using maven uri coordinates works, but if you place same jar in hdfs and reference by hdfs: instead of maven: task launch fails. This is something to do with how launch is orchestrated as stream apps can be placed in hdfs just fine.

This bug is either in yarn related code in a server impl, or in a yarn deployer module. Anyway creating this ticket here.

jvalkeal commented 7 years ago

Bug is actually getting fixed in https://github.com/spring-cloud/spring-cloud-deployer-yarn/issues/62 but we need to build a new yarn deployer version.

At this point is also worth to make a note about https://github.com/spring-cloud/spring-cloud-dataflow/issues/1628 where different h2 versions is causing a hang with task app(if default h2 DB is used in task and yarn).