Spark course - Githubissues

constructor-igor / TechSugar

Tech. Sugar seminars

7 stars 7 forks source link

Spark course #284

Closed constructor-igor closed 7 years ago

constructor-igor commented 7 years ago

course's page: https://app.pluralsight.com/library/courses/apache-spark-fundamentals/table-of-contents

scala course

Scala: Getting Started

installation notes:

additional references:

constructor-igor commented 7 years ago

Found and fixed a problem with spark-submit: http://stackoverflow.com/questions/39190829/spark-submit-gives-error-as-the-system-cannot-find-the-path-specified

constructor-igor commented 7 years ago

added question http://stackoverflow.com/questions/42068903/spark-error-output-directory-file-already-exists

answers:

Well... it is as clear as it says that output directory already exists and thus your output saveAsTextFile will not work. Most big-data frameworks prefer to avoid the chances of over-writing any existing data. So... they do not allow output in existing directories. Just pick some other directory for your output. – Sarvesh Kumar

What about using an absolute path like "file:///C:/temp/WordCount? Or look at stackoverflow.com/questions/38669206/… about some possible glitches across Spark versions. – Samson Scharfrichter

The problem fixed by using of full path:

sortedCounts.saveAsTextFile("file:///C:/temp/ReadMeWordCountViaApp")

constructor-igor commented 7 years ago

Introduction to Spark for .NET Developers

constructor-igor commented 7 years ago

PyHive is a collection of Python DB-API and SQLAlchemy interfaces for Presto and Hive. https://pypi.python.org/pypi/PyHive

constructor-igor commented 7 years ago

Actions:

collect
take(...)
takeSample(...)
first
count
countApproxDistinct ...

Persistence actions:

saveAsObjectFile
savaAsTextFile
External
foreach

constructor-igor commented 7 years ago

Spark Core Part 1 Resources and Summary

constructor-igor commented 7 years ago