Open ian-whitestone opened 5 years ago
Tweaking pyspark config:
conf = (SparkConf()
.setAppName("implicit_benchmark")
.setMaster('local[*]') # Run Spark locally with as many worker threads as logical cores on your machine.
.set('spark.driver.memory', '16G')
)
Spark repartitioning:
df1.rdd.getNumPartitions()
revisit processes
scheduler with less partitions
processes
is so slow: However moving data to remote processes and back can introduce performance penalties, particularly when the data being transferred between processes is large. The multiprocessing scheduler is an excellent choice when workflows are relatively linear, and so do not involve significant inter-task data transfer, and when inputs and outputs are both small, like filenames and counts. try using distributed...
play with local diagnostics
find way to demonstrate impact of having too many tasks..
resources:
notes from talking to Martin Durant:
Test this out: https://github.com/dask/dask-yarn/issues/28 Make a PR for this: https://github.com/dask/dask/issues/4110
Raise issues for the following:
1) raise a new issue in S3FS for how it just returns no files if your tokens aren't refreshed but doesn't raise errors.
2) Invalid paths returns weird errors. Should return no files or empty bag.
>>> bag = dask.bag.read_avro(urlpath, storage_options = {'profile_name': AWS_PROFILE})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/anaconda3/envs/gandalf/lib/python3.6/site-packages/dask/bag/avro.py", line 103, in read_avro
heads, sizes = zip(*out)
ValueError: not enough values to unpack (expected 2, got 0)
3) Empty bag to dataframe to pandas returns a empty tuple, you can't figure this out until you do a compute. May not be a way to fix this..check if you can join empty dask dataframes, since if you convert to pandas first you can do a manual check there.
Testing Plan
Dummy Credit Card Application Dataset
Test 1
Leave the default schedulers
Modifications
Test 2 - Runing Some Calcs
Modifications
Test 3 - Running some Python UDFs
Test 4 - Scaling on a Single Machine
NYC Taxi Public Dataset
Coming soon..
Sample ETL Workflows