divnix / std

A DevOps framework for the SDLC with the power of Nix and Flakes. Good for keeping deadlines!
https://std.divnix.com
396 stars 43 forks source link

Examples for using Standard as a toolkit for multi-project webdev monorepos? #165

Closed montchr closed 1 year ago

montchr commented 2 years ago

Standard positions itself a tool for managing monorepos (I think?). Most of my encounters with monorepos have been within the JavaScript ecosystem. Usually these tend to use Yarn, PNPM, Lerna, or some other newer tools (Vercel’s Turborepo also comes to mind, and there's also the very-confusingly-named "Nx"!).

I’ve been looking through some the examples and tutorials but am struggling a bit to find a middle ground between the simple tutorial examples and the complexity of the Bitte/Cardano examples. JS monorepos are more complex than the single-binary Rust example in the template but generally don't need to concern themselves with the complexities of managing hosting / distributed apps infrastructure.

Some notable real-world examples:

And some template examples I've come across:

So, all that is to say, does Standard fit the bill for these types of mega-projects containing smaller projects containing their own packages, components, utility function libraries, etc.? Or am I attempting to plug a square into a circle?

If Standard does fit such applications, in theory, how would Thee Standardisers map the concepts/structures of cells / cell blocks / runnables / data / etc. to one of those existing projects?

Another even more abstract question: how would Standard be applied as a tool for managing a web agency's client+project structures, or the many desks and newsrooms of a large media publication? If we zoom out and spin around, a larger variety of domain-boundaries appear: cells within organisms within hives within colonies within ... planets...


On the Broader Subject of Standard as Monorepo Tool

Perhaps most directly, let's refer to https://monorepo.tools/ (made by Nrwl, the company responsible for that confusingly-named "Nx" tool):

https://monorepo.tools/#monorepo-features

What monorepo tools should provide

  • Local computation caching -> fast
  • Distributed computation caching -> fast
  • Distributed task execution -> fast
  • Transparent remote execution -> fast
  • Detecting affected projects/packages -> fast
  • Local task orchestration -> fast
  • Workspace analysis -> understandable
  • Dependency graph visualisation -> understandable
  • Code sharing -> manageable
  • Consistent tooling -> manageable
  • Code generation -> manageable
  • Project constraints and visibility -> manageable

They then list several monorepo tools and compare them using these criteria. I do think Standard+Nix accomplish at least some of these requirements. Does Standard belong on that list too? Should it?

I am a little skeptical of the self-promotional aspect of that website, so here is another reference point with quite a lot less gloss:

https://www.robinwieruch.de/javascript-monorepos/

blaggacao commented 2 years ago

@rhyslbw I'm not sure if you actually have or want to have an opinion on this. But I think you might be the only person I know that sits on the JS-side (or in fact somwhere in between) of the intersection.

I remember our discussion how the nix cache can potentially disintermediate quite some of those novel domain-specific shared cache techniques in JS, such as in nx (as if not only the name was a shameless rip :wink:) and potentially solve some of the (re-)build duration issues, that developers reportedly face (or faced).

All that, while modulo "usability", currently not much is left on the nix side worth exploring in practical terms in a productivity setting.

blaggacao commented 2 years ago

Another even more abstract question: how would Standard be applied as a tool for managing a web agency's client+project structures, or the many desks and newsrooms of a large media publication? If we zoom out and spin around, a larger variety of domain-boundaries appear: cells within organisms within hives within colonies within ... planets...

Beyond the repo, there is the interconnecting nix tissue (this project would have been almost named tissue). Standard has a very friendly interface for downstream users (modulo issues with exponential lock file growth that is upon nix upstream to solve).

The fundamentals of this power is undoubtedly nix' fetcher class of functions.

For an agency, I'd probably use one repo per client (due to separation of accounts, while the cross-repo/cross-account integration penalty on the nix 'tissue' is low).

For News Agencies, I'd also try to find repo boundaries around teams amd spans of control (whatever that actually means when looking at it from the technical angle).

std, the CLI aims to a friendly reach-out and hand-holder for the average tech savvy consumer and can reach, with some very limited training on the command line, into non-technical users short of a web frontend for the stuff that DevOps puts out there.

It goes into the direction of a nix-based self service strategy. Yes, you heard that right! Modulo windows/mac support (which @jmgilman has done an excellent job in tackling via recent devcontainer support).

dzmitry-lahoda commented 1 year ago

General guidance on monorepos could be awesome.

I think to have any possible subfolder by any possible CODEOWNERS team to have is own cells to grow.

So root flake.nix is registered with many subfolder which are than flat out into packages/shells/apps/nixops/home/managers/arions/oci-images/.. according some pattern.

blaggacao commented 1 year ago

I want to say that arion (aka docker-compose on nix) support just landed in https://github.com/divnix/std/issues/173 (closed by https://github.com/divnix/std/pull/175). It's one piece in the puzzle of making Standard increasingly work for the average web developer. Not directly related to the mono-repo aspect, though.

blaggacao commented 1 year ago

@montchr in the JS world, I'd point you to https://github.com/canva-public/js2nix

With a little bit of flake-wiring and softening of hard-coded input contracts to the CLI, that looks like an excellent integration candidate for std + js.

montchr commented 1 year ago

I'm going to close this since it's too broad, and I don't really know how one would pick anything actionable from my initial post 🙃