search

rjurney / Agile_Data_Code_2

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
http://bit.ly/agile_data_science
MIT License
456 stars 306 forks source link

fix bad versions for mongo-java-driver and elasticsearch-spark jars #92

Closed pjhinton closed 5 years ago

pjhinton commented 5 years ago

This commit fixes a shell echo command that appends JAR paths to the Apache Spark configuration file. The ones for mongo-java-driver and elasticsearch-spark do not match what are present in the lib directory.

The effect of making this change means that the following errors (and their underlying exceptions) will be removed from the startup sequence for pyspark.

18/12/05 21:26:12 ERROR SparkContext: Failed to add file:/home/vagrant/Agile_Data_Code_2/lib/mongo-java-driver-3.4.2.jar to Spark environment

18/12/05 21:26:12 ERROR SparkContext: Failed to add file:/home/vagrant/Agile_Data_Code_2/lib/elasticsearch-spark-20_2.10-5.2.1.jar to Spark environment
rjurney commented 5 years ago

Thanks!