goldma99 / beavers

Miriam Gold, Beavers writing sample
0 stars 0 forks source link

Reorganize pipeline code into a fully modular structure #16

Closed goldma99 closed 5 months ago

goldma99 commented 5 months ago

Design concept

This structure, which I call Goalkeeper, aims to be highly modular and adaptive to new directions or new sub-projects. The basic elements, in hierarchical order, are a project, blocks and goals.

Within the Scotland code directory, a master script executes everything. The directory also holds folders for each top-level block of the project work (data cleaning, data construction/wrangling, modeling, plotting, table construction). Given the modular structure, more may be added without disturbing the existing workflows in the other project blocks.

Within each block's directory, a master script executes every script, and there is one script per goal (e.g., construct the panel of beaver expansion). To ensure these scripts remain moderately sized, each block's directory contains a functions/ directory, containing one script per custom function. Any function relevant to that block's goals may be included. Because I use source_dir() , which searches its target directory recursively, to source all my custom functions, function scripts may be added at the top-level within the functions/ directory, or separated into folders as necessary.