Closed metasim closed 6 years ago
It could also be that the python errors generated when the spark version changes are due to the .eggs
directory getting accidentally processed when it shouldn't.
I might be misreporting the bug. I'm attempting to build/test against Spark 2.3.1, and it looks like the .eggs
directory has the previous version of Spark in it.
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/decision_tree_classification_example.py", line 47
[error] testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/decision_tree_regression_example.py", line 47
[error] testMSE = labelsAndPredictions.map(lambda (v, p): (v - p) * (v - p)).sum() /\
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/gradient_boosting_classification_example.py", line 46
[error] testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/gradient_boosting_regression_example.py", line 46
[error] testMSE = labelsAndPredictions.map(lambda (v, p): (v - p) * (v - p)).sum() /\
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/linear_regression_with_sgd_example.py", line 47
[error] .map(lambda (v, p): (v - p)**2) \
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py", line 47
[error] trainErr = labelsAndPreds.filter(lambda (v, p): v != p).count() / float(parsedData.count())
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/naive_bayes_example.py", line 53
[error] accuracy = 1.0 * predictionAndLabel.filter(lambda (x, v): x == v).count() / test.count()
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/random_forest_classification_example.py", line 48
[error] testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/random_forest_regression_example.py", line 48
[error] testMSE = labelsAndPredictions.map(lambda (v, p): (v - p) * (v - p)).sum() /\
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/svm_with_sgd_example.py", line 41
[error] trainErr = labelsAndPreds.filter(lambda (v, p): v != p).count() / float(parsedData.count())
[error] ^
[error] SyntaxError: invalid syntax
[error] File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/streaming/network_wordjoinsentiments.py", line 70
[error] .map(lambda (word, tuple): (word, float(tuple[0]) * tuple[1])) \
[error] ^
[error] SyntaxError: invalid syntax
Added python build locations to sbt clean files.
After
sbt clean
is run, the directorypyrasterframes/python/.eggs
is still around, making it difficult to test against multiple versions of Spark without manual cleaning of cached files. I'd suggest either having.eggs
created in thepyrasterframes/target
directory, or adding the directory to the list of to-clean files.