search

locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
246 stars 45 forks source link

Have `sbt clean` remove cached Python files #65

Closed metasim closed 6 years ago

metasim commented 6 years ago

After sbt clean is run, the directory pyrasterframes/python/.eggs is still around, making it difficult to test against multiple versions of Spark without manual cleaning of cached files. I'd suggest either having .eggs created in the pyrasterframes/target directory, or adding the directory to the list of to-clean files.

metasim commented 6 years ago

It could also be that the python errors generated when the spark version changes are due to the .eggs directory getting accidentally processed when it shouldn't.

metasim commented 6 years ago

I might be misreporting the bug. I'm attempting to build/test against Spark 2.3.1, and it looks like the .eggs directory has the previous version of Spark in it.

[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/decision_tree_classification_example.py", line 47
[error]     testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
[error]                                                  ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/decision_tree_regression_example.py", line 47
[error]     testMSE = labelsAndPredictions.map(lambda (v, p): (v - p) * (v - p)).sum() /\
[error]                                               ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/gradient_boosting_classification_example.py", line 46
[error]     testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
[error]                                                  ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/gradient_boosting_regression_example.py", line 46
[error]     testMSE = labelsAndPredictions.map(lambda (v, p): (v - p) * (v - p)).sum() /\
[error]                                               ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/linear_regression_with_sgd_example.py", line 47
[error]     .map(lambda (v, p): (v - p)**2) \
[error]                 ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py", line 47
[error]     trainErr = labelsAndPreds.filter(lambda (v, p): v != p).count() / float(parsedData.count())
[error]                                             ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/naive_bayes_example.py", line 53
[error]     accuracy = 1.0 * predictionAndLabel.filter(lambda (x, v): x == v).count() / test.count()
[error]                                                       ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/random_forest_classification_example.py", line 48
[error]     testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
[error]                                                  ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/random_forest_regression_example.py", line 48
[error]     testMSE = labelsAndPredictions.map(lambda (v, p): (v - p) * (v - p)).sum() /\
[error]                                               ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/mllib/svm_with_sgd_example.py", line 41
[error]     trainErr = labelsAndPreds.filter(lambda (v, p): v != p).count() / float(parsedData.count())
[error]                                             ^
[error] SyntaxError: invalid syntax
[error]   File "/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/python/.eggs/pyspark-2.1.2-py3.6.egg/pyspark/examples/src/main/python/streaming/network_wordjoinsentiments.py", line 70
[error]     .map(lambda (word, tuple): (word, float(tuple[0]) * tuple[1])) \
[error]                 ^
[error] SyntaxError: invalid syntax
bguseman commented 6 years ago

Added python build locations to sbt clean files.