search

davidglevy / spark-itcase

Set of tools to run Spark programs remotely during development builds
Apache License 2.0
1 stars 1 forks source link

Do a gap analysis with server module vs Livy. #9

Open davidglevy opened 8 years ago

davidglevy commented 8 years ago

So almost in parallel we see the project Livy component arrive - it has some commonalites to the server component (batch deployment of a single JAR) however has some differences and areas which would need to be featured-up to satisfy requirements here.... however seems very worthwhile to maybe intgrate with a known component if possible.

This task is to do a gap analysis on Livy and list requirements to switch across.

davidglevy commented 8 years ago

Requirement 1: Livy release pre-built and easy to install (without compilation) Having a pre-built version of Livy available is non-functional and really represents where that project is up to - we'll just have to watch this space on progress in that area.

Requirement 2: Artifact resolution The ability for Livy to become "aware" of Java dependencies and retrieve them on behalf of the client is something the creators likely never considered for their use cases. This however is front/center of our requirement.

Requirement 3: Run from artifact pre-downloaded I would like to see Livy be run from a pre-resolved artifact. This could be a pre-commit JAR which has been uploaded or an artifact pulled from a trusted Maven repository.