search

spring-attic / spring-cloud-dataflow-server-yarn

Spring Cloud Data Flow Implementation for Apache YARN
http://cloud.spring.io/spring-cloud-dataflow-server-yarn/
Apache License 2.0
15 stars 34 forks source link

Cannot post data to HTTP source module #27

Closed sabbyanandan closed 8 years ago

sabbyanandan commented 8 years ago

As a user, I have http --server.port 9000 | cassandra and http --server.port 9001 | jdbc streams running in YARN; however, I'm unable to post data to http://localhost:9000 or http://localhost:9001 endpoints. The request fails with 404 errors.

Reason: In a YARN cluster, we do not know where the http modules (boot applications) are hosted. They're randomly distributed by YARN's AppMaster. It is impossible to guess the node where they're hosted.

Possible solution: Leverage spring-cloud components to register where http modules are hosted, so they're discoverable for futures.

jvalkeal commented 8 years ago

POC: https://github.com/jvalkeal/spring-cloud-dataflow-admin-yarn/commits/gh27-poc

more info in this commit: https://github.com/jvalkeal/spring-cloud-dataflow-admin-yarn/commit/d63752d2862dca434508b64c5b9ad6c8d13b8d87

trisberg commented 8 years ago

Ran this and it works as advertised. Concern is additional services to run - admin + zuul + eureka and we'd need two of each for HA. Wonder if they could be combined into single runtime for fewer nodes needed in Ambari cluster config? Also, I think we should name the projects after the functionality rather than implementation - dataflow-zuul -> dataflow-proxy and dataflow-eureka -> dataflow-service-registry

jvalkeal commented 8 years ago

Naming is probably better without zuul/eureka and I think functionality can be combined. Just wanted to have clear features in separated jars in this poc.

If going forward with these ideas, we need to think about a future of a module launcher because the black magic I did with that boot auto-config jar for eureka client kinda relates to how deps are pulled in when modules are run.

jvalkeal commented 8 years ago

closing this poc pr as concept has been reviewed.