Partitions coalesce

[Jday7879]

Merge request template: please remove the appropriate parts of this template.

Pre-merge request checklist (to be completed by the one making the request):

• [x] I have performed a full review of this code myself.
  • For Python code in PySpark specific sections, all code should have been run in Jupyter notebooks.
  • For code in sections of the book containing both Python and R code, the page of the book should be constructed as described in the contributing guide and converted to a markdown file.
• [x] I have formatted the outputs of code blocks correctly (to match other outputs in the book and in line with the style guide [coming soon])
• [x] I have built the book as outlined in the contributing guide and confirmed that any additional/modified content is displaying as expected.

Details of this request (such as):

• updated partitions page to include content on binding/unioning
• Fixed link issues on Union two DataFrames ... page
• Moved managing partitions content to Reading and Writing Data in Spark page

Requirements for review (such as):

• Review new text content for spelling / grammar mistakes
• Check new code runs.

closes #98 closes #100 closes #119 closes #19

[jkhall06]

Hi Jordan,

Apologies for not getting round to this earlier. It was weird, I remember reading through this notebook so not sure what happened to my notes. I must have slept and then forgotten it!

Anyway, the text is fine, clear and makes sense.

Intermediate partitions in wide partitions Under the section on Run the jobs and investigate the UI there are a couple of images which do not seem to be displaying properly. I have checked and they are in as code md (e.g. ``` as opposed to the format). Not sure if it has to be in this format for when the image is displayed in the book, if it is then ignore this comment! The code runs fine.

Partitions when Unioning or Binding Dataframes All code runs here fine, I was getting something odd going on when running it on my laptop using a local session with the spark.sql.shuffle.partitions config setting not working. Tried it on DevTest now and it is working as it should, so must be something to do with running on a laptop (probably can't partition the data in the same way I am thinking?).

[Jday7879]

Thanks Johanna, I think I noticed some issues with partitions on a local machine when writing the page as well. Might be something to keep an eye on (Not sure how many people will be running spark locally and be changing number of partitions!)