search

yuwtennis / google-data-engineer

Repository includes programs which was created while studying for google data engineer cert.
0 stars 0 forks source link

Enable remote spark cluster access #6

Closed yuwtennis closed 4 years ago

yuwtennis commented 4 years ago

submitting job with cluster mode not available yet in pyspark .

https://spark.apache.org/docs/latest/submitting-applications.html

Alternatively, if your application is submitted from a machine far from the worker machines (e.g. locally on your laptop), it is common to use cluster mode to minimize network latency between the drivers and the executors. Currently, the standalone mode does not support cluster mode for Python applications.