fix script by adding call to stop default SparkContext
This commit fixes an error that is observed in the book version
of the code that occurs when a new SparkContext is created.
>>> sc = SparkContext(
... appName = "Agile Data Science: PySpark Streaming 'Hello, world!'", conf=conf
... )
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/home/vagrant/spark/python/pyspark/context.py", line 115, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/home/vagrant/spark/python/pyspark/context.py", line 299, in _ensure_initialized
callsite.function, callsite.file, callsite.linenum))
ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by <module> at /home/vagrant/spark/python/pyspark/shell.py:45
This can be fixed by stopping the default SparkContext that is
created during startup. This is proably a better fix than
just commenting out the instantiation of the SparkContext
because it ensures the SparkConf is used.
fix script by adding call to stop default SparkContext
This commit fixes an error that is observed in the book version of the code that occurs when a new SparkContext is created.
This can be fixed by stopping the default SparkContext that is created during startup. This is proably a better fix than just commenting out the instantiation of the SparkContext because it ensures the SparkConf is used.