best-practice-and-impact / ons-spark

MIT License
9 stars 5 forks source link

Add in coalesce_small_files notebook from the troubleshooting repo into the book #98

Closed NathanKelly-ONS closed 10 months ago

NathanKelly-ONS commented 1 year ago

This notebook talks about why you shouldn't use coalesce on small files. It should be added to the book, in the reading/writing data section.

This could be combined with #19 as the reading/writing data section doesn't currently talk about partitions.