No SparkContext defined

[JuanVargas]

OS = Ubuntu 22.04 LTS Julia 1.7.3 java -version openjdk version "11.0.15" 2022-04-19 OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1) OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1, mixed mode, sharing) Apache Maven 3.6.3 Maven home: /usr/share/maven Java version: 11.0.15, vendor: Private Build, runtime: /usr/lib/jvm/java-11-openjdk-amd64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.15.0-41-generic", arch: "amd64", family: "unix"

When I try to follow the example code in the documentation to create a SparkContext:

using Spark Spark.init() sc = SparkContext(master="local")

UndefVarError: SparkContext not defined

NOTE: I searched for SparkContext in the Gitbub code but could not find any file that made a reference to it.

[dfdx]

Can you point to the page with this example? SparkContext has been removed, and I can't find any mentions of it in the docs.

[JuanVargas]

It is right here:

http://dfdx.github.io/Spark.jl/

On Mon, Jul 25, 2022 at 12:29 AM Andrei Zhabinski @.***> wrote:

Can you point to the page with this example? SparkConf has been removed, and I can't find any mentions of it in the docs.

— Reply to this email directly, view it on GitHub https://github.com/dfdx/Spark.jl/issues/109#issuecomment-1193564002, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGK34JWZQFTSVJR3GTRADTVVYJ25ANCNFSM54QTCY3Q . You are receiving this because you authored the thread.Message ID: @.***>

[JuanVargas]

I think I see what you mean. The documentation at Github points to different content depending on the links one follows.

These links coming from Julia Hub mention SparkContext

https://juliahub.com/ui/Packages/Spark/zpJEw/0.6.0

http://dfdx.github.io/Spark.jl/index.html

http://dfdx.github.io/Spark.jl/api.html

... etc...

but these don't mention SparkContext:

http://dfdx.github.io/Spark.jl/dev/

Thanks,

On Mon, Jul 25, 2022 at 6:52 AM Juan Vargas @.***> wrote:

It is right here:

http://dfdx.github.io/Spark.jl/

On Mon, Jul 25, 2022 at 12:29 AM Andrei Zhabinski < @.***> wrote:

Can you point to the page with this example? SparkConf has been removed, and I can't find any mentions of it in the docs.

— Reply to this email directly, view it on GitHub https://github.com/dfdx/Spark.jl/issues/109#issuecomment-1193564002, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGK34JWZQFTSVJR3GTRADTVVYJ25ANCNFSM54QTCY3Q . You are receiving this because you authored the thread.Message ID: @.***>

[dfdx]

For whatever reason JuliaHub doesn't want to update the README of the project and still points to the old documentation. I tried to fix it by re-triggering the TagBot, but it doesn't seem to have any effect. Anyway, the correct link to the documentation is this:

http://dfdx.github.io/Spark.jl/dev/

(You can always find it in the README.md in this repo)

[JuanVargas]

Yes, I noticed the changes. I guess the JuliaHUb people need to be notified.

Given the changes that I can see now in the docs, it looks like the SparkContext was taken out of the project. Is that correct?

Thank you

On Mon, Jul 25, 2022 at 3:38 PM Andrei Zhabinski @.***> wrote:

For whatever reason JuliaHub doesn't want to update the README of the project and still points to the old documentation. I tried to fix it by re-triggering the TagBot, but it doesn't seem to have any effect. Anyway, the correct link to the documentation is this:

http://dfdx.github.io/Spark.jl/dev/ http://dfdx.github.io/Spark.jl/dev/

(You can always find it in the README.md https://github.com/dfdx/Spark.jl/blob/main/README.md in this repo)

— Reply to this email directly, view it on GitHub https://github.com/dfdx/Spark.jl/issues/109#issuecomment-1194535319, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGK34J4XEYRPG6ZMLH4DHDVV3UMTANCNFSM54QTCY3Q . You are receiving this because you authored the thread.Message ID: @.***>

[dfdx]

Given the changes that I can see now in the docs, it looks like the SparkContext was taken out of the project. Is that correct?

Yes.

[dfdx]

Closing because of no activity.