Data analyses are often complex. Data pipelines are ways of managing that complexity. Our data pipelines have two foundational pieces:
Good organization of code scripts help you quickly find the file you need, whether you or a teammate created it.
Dependency managers such as remake, scipiper, and drake formalize the relationships among the datasets and functions to ensure reproducibility while also minimizing the amount of unnecessary runtime as you're creating or modifying parts of the pipeline.
:keyboard: Activity: Assign yourself to this issue to get started.
:bulb: Tip: Throughout this course, I, the Learning Lab Bot, will reply and direct you to the next step each time you complete an activity. But sometimes I'm too fast when I :hourglass_flowing_sand: give you a reply, and occasionally you'll need to refresh the current GitHub page to see it. Please be patient, and let my humans know (jread-usgs or aappling-usgs) if I seem to have become completely stuck.
I'll sit patiently until you've assigned yourself to this one.
kat3t3
commented
4 years ago
Data analyses are often complex. Data pipelines are ways of managing that complexity. Our data pipelines have two foundational pieces:
Good organization of code scripts help you quickly find the file you need, whether you or a teammate created it.
Dependency managers such as
remake,scipiper, anddrakeformalize the relationships among the datasets and functions to ensure reproducibility while also minimizing the amount of unnecessary runtime as you're creating or modifying parts of the pipeline.:keyboard: Activity: Assign yourself to this issue to get started.
:bulb: Tip: Throughout this course, I, the Learning Lab Bot, will reply and direct you to the next step each time you complete an activity. But sometimes I'm too fast when I :hourglass_flowing_sand: give you a reply, and occasionally you'll need to refresh the current GitHub page to see it. Please be patient, and let my humans know (
jread-usgsoraappling-usgs) if I seem to have become completely stuck.I'll sit patiently until you've assigned yourself to this one.