search

dfdx / Spark.jl

Julia binding for Apache Spark
Other
205 stars 39 forks source link

Do not initialise the JVM by default Fix #28 #44

Closed aviks closed 7 years ago

aviks commented 7 years ago

The workers load the Spark module, and do not need the JVM loaded on there. This PR changes the initialisation from default to user initiated. This improves the memory usage and start time on the workers.

dfdx commented 7 years ago

Since this is a breaking change, we will need to bump package version. I was thinking about removing support for Julia 0.5 in a month or two, but we can do it sooner if it will help with changes (e.g. Task/Channel stuff).

aviks commented 7 years ago

I will need to support 0.5 for a while yet. So I'd hope we can do a new release, say 0.2.0 after #40 is merged. At which point, we can drop support for 0.5 from master, and move 0.5 support to a branch.