Introduces `#|expatch` directive

[ew-meetup]

• Documentation is in nbs/explanations/directives.ipynb.
• Implementation is spread across nbs/apis/0{1,4}*.

Use Case:

This directive has been tested in production for a couple of months to develop ETL style PySpark jobs.

Due to how the pyspark jobs are deployed, we end up needing all of the code wrapped in a method annotated with @call_parse. Using #|expatch has been helpful, because the code in the method is linear. It involves transforming complex, messy, and disparate sets of data. We interweave exported code cells (commented with the #|expatch directive) with documentation and example cells. But we also get documentation and examples right alongside the code. There is plenty of code refactoring, helper methods, and code reuse. But at the end of the day, the bulk of the business logic is best kept in a single exported method. It has proven much easier and faster to debug and iterate than declaring the whole of the primary method inside a single hundred-line-plus cell.

A Caveat on Usage:

Please note that the #|expatch directive has very limited utility where it actually makes sense to refactor the code into small methods. It is not a good fit for those cases. But in that narrow set of use-cases where a long drawn-out method makes sense, this directive significantly simplifies documentation and debugging. In design pattern terminology, this is sometimes referred to as a "Transaction Script". Not to be confused with the much more common anti-pattern called "Long Method".

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ew-meetup]

sorry, wrong account!