Refactor: fv3net monorepo structure

[nbren12]

Overall, the fv3net mono-repo has worked pretty well for us. It allows us to concentrate tooling and auth in once place, while maintaining reasonable modularity. The current structure has

• external, a folder for micropackages (with varying degrees of interdependence)
• workflows, a folder of "workflows", which is a somewhat poorly defined concept
• fv3net, the original directory structure. Once #469 is merged, this directory will mostly contain dataflow jobs.
• tests, a random assortment of intergration tests, and tests for code in fv3net.
• docs, some documentation

Some things I like:

• It works!
• Easy to develop related components at same time
  • only need one PR
  • single development environment (mostly)
• Mostly good separation of concerns in external

Some pain points:

• Unclear what a "workflow" is. How does this differ from a script in a micropackage?
• Test and documentation are scattered in many places
• Little/no support for testing code with minimal dependencies
  • almost all tests run from a single global "fv3net" environment.
• Dataflow submission is tricky
  • poetry path dependencies don't work with pip wheel or pip sdist
• Hard to depend on code not in this repo. git submodules aren't ideal.
• Currently, using many strategies to determine what tests are executed. marks, fixtures, manual exclusions, etc
• Many routines make assumptions about this directory structure
  • workflows executed from root
  • docker images built from root
  • scripts depend on data files (e.g. catalog.yaml) with relative paths
• Too many entry points
  • there are so many scripts to keep track of with little categorization and poor discoverability.
• Not easy to define a new docker image, and have it pushed to GCR automatically.

[nbren12]

cc @mcgibbon

[mcgibbon]

I copy-pasted your thoughts into the Paper document.

[nbren12]

Closing this meta issue, since we discussed it. not all of these issues have been resolved (e.g. too many entry points).