jupyter / kernel_gateway_demos

Demos associated with the kernel gateway incubator project
BSD 3-Clause "New" or "Revised" License
152 stars 76 forks source link

Using an existing SparkContext instance in example.py #28

Closed dockter-84 closed 8 years ago

dockter-84 commented 8 years ago

In some instances of Spark, the SparkContext, "sc", is already instantiated. Thus when the example is run, an error occurs :

'Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=spark://ys1-spark-dal09-env4-0019:7082) created by <module> at /usr/local/src/bluemix_jupyter_bundle.v14/notebook/lib/python2.7/site-packages/IPython/utils/py3compat.py:286

The following changes to example.py fix this issue:

# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
import pyspark

if not globals().has_key('sc'):
    sc = pyspark.SparkContext()
rdd = sc.parallelize(range(1000))
sample = rdd.takeSample(False, 5)
print(sample)

Can that change be made to example.py?

parente commented 8 years ago

Addressed by #29.