Split up infrastructure page

[h-vetinari]

I'm looking at our infrastructure page, and it's pretty long and unwieldy.

I think it should be split up, possibly into several different pages. At the very least, I'd like to split it into stuff that's recipe-relevant (compilers, sysroot, CDTs), and stuff that regular maintainers need much less often (website, metadata, CI setup, etc.).

Other possible things that might deserve their own pages:

• bot commands
• staged-recipes
• "special package handling" (i.e. repodata patches, admin requests, feedstock-outputs)
• everything regro-related

Thoughts @conda-forge/core?

CC @zklaus for recent efforts on these pages.

[jaimergp]

Last year, I experimented with how Diátaxis would apply to our docs in https://cf-infra-docs.netlify.app/docs. Some of the docs written for that experiment are being migrated over to our new website. @zklaus is almost finished, I think, with that effort.

I fully support the splitting. Here are some unordered thoughts that might make sense along with these efforts:

• I think it would be beneficial to have the "life cycle" published as a standalone documentation entry because it gives a big picture of the whole infra. And then more details can be filled in in specific pages.
• The bot-commands could be autogenerated from the conda-forge-webservices code similar to what we have done with the conda-forge.yml JSON schema.
• If there's code out there emitting documentation, we should not rewrite those in the website. Either make it programmatically renderable in the website, or just point to the external documentation from the place(s) that make(s) most sense.

Also I think we need to distinguish between the infra that supports running a single build job, and the infra that supports orchestrating build jobs and automation. The line gets blurry at times, but I think it's worth trying it.

[jakirkham]

It is worth noting that we link to the admin bot requests all over the place. So if there is some way to forward the old links to whatever new page they live on, that would be quite helpful

[h-vetinari]

Last year, I experimented with how Diátaxis would apply to our docs in https://cf-infra-docs.netlify.app/docs. Some of the docs written for that experiment are being migrated over to our new website. @zklaus is almost finished, I think, with that effort.

That looks very nice! I think we should not only take over the content, but also start structuring things in a similar way.

I think it would be beneficial to have the "life cycle" published as a standalone documentation entry because it gives a big picture of the whole infra. And then more details can be filled in in specific pages.

Sounds great - i.e. a big overview that branches off into separate little topic pages. I think in terms of diataxis quadrants that would be "Explanation" AFAICT, and very useful at that. We're lacking in that category a bit IMO (e.g. people struggle a lot with understanding the different roles of build: & host: environments; I have been wanting to write something up about that for a long time).

[h-vetinari]

Hm, this turned out to be a way deeper rabbit hole than I wanted to dive into 😅

First, a proposal only for Reference:Infrastructure (topic of this issue):

[...]
├ Reference
| ├ Infrastructure
| | ├ Staged recipes            # including teams
| | ├ Compilers
| | ├ Other core packages       # sysroots, CDTs, msys2, ...
| | ├ CI Setup                  # docker images, conda-forge-ci-setup
| | ├ CI providers
| | ├ Global pinning
| | ├ Feedstock evolution       # conda-smithy
| | ├ Feedstock metadata        # graphs, feedstocks feedstock-outputs
| | ├ Metadata corrections      # repodata-patches, admin-requests
| | ├ Bot repos                 # all things regro
| | ├ Website
| | └ ...
| ├ Bot commands
| └ ...
└ ...

In order to orient myself, I however had to look at the bigger picture, so here's a suggested page structure based on the existing docs and the @jaimergp's experiment:

Overall structure proposal

| | vs. [docs](https://conda-forge.org/docs/) | vs. [Infra Exp.](https://cf-infra-docs.netlify.app/docs/) | Comment | |--------|--------|--------|--------| | `Welcome` | == [here](https://conda-forge.org/docs/) | == [Welcome](https://cf-infra-docs.netlify.app/docs/) | | | `For users` | == [here](https://conda-forge.org/docs/user/) | n/a | could surely be improved... | | `└ ...` | | | ... but not in scope here| | `For maintainers` | == [here](https://conda-forge.org/docs/maintainer/) | n/a | | | `├ Tutorials` | partially [here](https://conda-forge.org/docs/maintainer/adding_pkgs) | == [Tutorials](https://cf-infra-docs.netlify.app/docs/tutorials/) | | | `\| ├ Generate a conda recipe` | | | Using `greyskull` or whatever | | `\| ├ Write your first recipe` | | | Segue from first tutorial,
make some changes | | `\| ├ Submit your first recipe` | | ~= [placeholder](https://cf-infra-docs.netlify.app/docs/tutorials/submit-a-recipe/) | Segue again,
process for submission | | `\| └ ...` | | | | | `├ How-to guides` | distributed all over | == [How-to guides](https://cf-infra-docs.netlify.app/docs/howto/) | | | `\| ├ Easy` | | | | | `\| \| ├ Read a recipe` | | | Component overview,
refer to in-depth explanation | | `\| \| ├ Rerender feedstock` | | | | | `\| \| ├ Test packages` | | | | | `\| \| ├ Deal with numpy` | roughly [here](https://conda-forge.org/docs/maintainer/knowledge_base/#building-against-numpy) | | | | `\| \| ├ Enable osx-arm64` | roughly [here](https://conda-forge.org/docs/maintainer/knowledge_base/#apple-silicon-builds) | | Same for [arches](https://conda-forge.org/docs/maintainer/knowledge_base/#using-arch_rebuildtxt) | | `\| \| ├ Talk to the bots` | | | refer to bot command
reference page | | `\| \| └ ...` | | | | | `\| ├ Medium` | | | | | `\| \| ├ Cross-compile` | ~= [here](https://conda-forge.org/docs/maintainer/knowledge_base/#cross-compilation) | | | | `\| \| ├ Enable CUDA` | ~= [here](https://conda-forge.org/docs/maintainer/knowledge_base/#cuda-builds) | | | | `\| \| ├ Multi-output recipes` | ~= [here](https://conda-forge.org/docs/maintainer/knowledge_base/#multi-output-recipes) | | | | `\| \| ├ LTS branches` | ~= [here](https://conda-forge.org/docs/maintainer/updating_pkgs/#maintaining-several-versions) | | | | `\| \| └ ...` | | | | | `\| ├ Emergencies` | | | | | `\| \| ├ Patch repodata` | | | | | `\| \| └ Mark a package broken` | ~= [here](https://conda-forge.org/docs/maintainer/updating_pkgs/#removing-broken-packages) | | | | `├ Understanding c-f` | not much | ~= [Fundamentals](https://cf-infra-docs.netlify.app/docs/fundamentals/) | "Explanation" part of [diataxis](https://diataxis.fr/) | | `\| ├ Life-cycle of a pkg` | | == [here](https://cf-infra-docs.netlify.app/docs/fundamentals/life-cycle/) | | | `\| \| └ ...` | | | | | `\| ├ How a pkg gets built` | | | | | `\| \| ├ Compilation concepts` | | | shared/static/etc. | | `\| \| ├ Env. roles` | short [comment](https://conda-forge.org/docs/maintainer/knowledge_base/#placing-requirements-in-build-or-host) | | e.g. `build:` vs. `host:` | | `\| \| ├ Compilers` | partially [here](https://conda-forge.org/docs/maintainer/infrastructure/#compilers-and-runtimes) | | e.g. role of activation | | `\| \| ├ Global pinning` | ~= [here](https://conda-forge.org/docs/maintainer/pinning_deps/) | | | | `\| \| ├ Run exports` | ~= [here](https://conda-forge.org/docs/maintainer/pinning_deps/#specifying-run_exports) | | | | `\| \| ├ Jinja functions` | | | | | `\| \| └ ...` | | | | | `\| ├ How our bots work` | | see [here](https://cf-infra-docs.netlify.app/docs/advanced/automation/) | | | `\| \| ├ Migrations` | ~= [here](https://conda-forge.org/docs/maintainer/knowledge_base/#migrators-and-migrations) | | | | `\| \| ├ regro infra` | | | | | `\| \| └ ...` | | | | | `\| ├ Security` | ~= [here](https://conda-forge.org/docs/maintainer/knowledge_base/#security-considerations-for-conda-forge-builds) | see [placeholder](https://cf-infra-docs.netlify.app/docs/advanced/security/) | | | `\| \| ├ Output validation` | | | | | `\| \| └ ...` | | | | | `\| ├ Maintainer roles` | roughly [here](https://conda-forge.org/docs/maintainer/guidelines/) & [here](https://conda-forge.org/docs/maintainer/adding_pkgs/#maintainer-role) | | | | `\| └ ...` | | | | | `├ Reference` | all over | == [Reference](https://cf-infra-docs.netlify.app/docs/reference/) | | | `\| ├ Infrastructure` | | ~same content split
somewhat differently | | | `\| \| ├ Staged recipes` | | roughly [here](https://cf-infra-docs.netlify.app/docs/reference/infrastructure/staged-recipes/) | incl. teams | | `\| \| ├ Compilers` | roughly [here](https://conda-forge.org/docs/maintainer/infrastructure/#compilers-and-runtimes) | | | | `\| \| ├ Other core packages` | | | sysroots, CDTs, msys2, ... | | `\| \| ├ CI Setup` | | | docker images,
conda-forge-ci-setup | | `\| \| ├ CI providers` | | roughly [here](https://cf-infra-docs.netlify.app/docs/reference/infrastructure/services/) | | | `\| \| ├ Global pinning` | ~= [here](https://conda-forge.org/docs/maintainer/pinning_deps/) | | +explain migrators | | `\| \| ├ Feedstock evolution` | | | conda-smithy | | `\| \| ├ Feedstock metadata` | | | graphs, feedstocks,
feedstock-outputs | | `\| \| ├ Metadata corrections` | | | repodata-patches, admin-requests | | `\| \| ├ Bot repos` | | | all things regro | | `\| \| ├ Website` | | == [here](https://cf-infra-docs.netlify.app/docs/reference/infrastructure/website/) | | | `\| \| └ ...` | | | | | `\| ├ Bot commands` | ~= [here](https://conda-forge.org/docs/maintainer/infrastructure/#admin-web-services) | == [here](https://cf-infra-docs.netlify.app/docs/reference/bot-commands/) | | | `\| ├ Feedstock settings` | mostly [here](https://conda-forge.org/docs/maintainer/conda_forge_yml/) | == [here](https://cf-infra-docs.netlify.app/docs/reference/feedstock-settings/) | `conda-forge.yml`, CBC, migrators | | `\| ├ Glossary` | == [here](https://conda-forge.org/docs/glossary/) | == [here](https://cf-infra-docs.netlify.app/docs/reference/glossary/) | | | `\| ├ Current pins` | | == [here](https://cf-infra-docs.netlify.app/docs/reference/pinnings/) | Auto-generated / subset? | | `\| ├ Ongoing migrations` | | | refer to status page | | `\| └ ...` | | | | | `├ Miniforge` | | == [here](https://cf-infra-docs.netlify.app/docs/miniforge/) | | | `└ FAQ` | ~= [here](https://conda-forge.org/docs/maintainer/maintainer_faq/) | | separate because
there's no clean split
into how-to & reference...? |

[zklaus]

I really like this! One of the great things about Diátaxis is that it's explicitly build to not force you to do everything in one fell swoop. So moving in this direction with a plan as posted by @h-vetinari as a general guideline to follow sounds great to me.

With #2158, my transfer of content into the infrastructure page will be concluded, so I am all for moving forward in the proposed direction.

[jakirkham]

With #2158, my transfer of content into the infrastructure page will be concluded, so I am all for moving forward in the proposed direction.

This was merged earlier today. So it sounds like this issue is wide open

[h-vetinari]

I opened a more broadly-scoped issue about the overall structure proposed above in #2164. I'll bring it up in the core call this week to see if people are on board with the overall direction. Once we have a Reference section, it would be easier to move things, I think.