Sync waves/phases doc needs more clarity

[lexey-evergreen]

Summary

The Sync Phases and Waves doc does not offer clarity around a couple of key ideas, namely:

• The concept of "sync contexts" (mentioned in #8358 here).
• The fact that sync waves can be applied to some resources and cannot be applied to others (mentioned in #7437, see short summary here).

Motivation

When a user is trying to determine how to apply sync waves/phases, it would be helpful to have a better understanding of which use cases can be solved by them, how they actually work, and some common gotchas.

Proposal

Add a few sections to the Sync Phases and Waves doc.

A section about sync contexts

This paragraph should talk about the scope of sync contexts, how they're created, and how they're related/unrelated to each other. I don't really understand this myself, so I can't give more specific guidance about the contents of this paragraph.

A section about which resources work with sync waves/phases

The purpose of this paragraph is to clarify that sync waves/phases only function properly with some resources. This is essentially a dedicated gotcha paragraph.

Sync waves/phases examples

Possibly the most helpful addition to the doc would be a couple of examples of using sync waves/phases. Ideally, each of these examples include a few things:

• A problem statement describing the desired behavior.
• A "solution statement" that describes in short prose how we can get the desired behavior using sync waves/phases.
• A set of example manifests that implement the desired behavior (or at least a link to them).
• When applicable, a set of example manifests that represent a reasonable attempt at implementing the desired behavior, but in reality do not. This would make any gotchas clearer and more understandable.

Here are a few ideas for examples:

• One example could show the simplest use case of sync waves/phases. This could be a backend deployment and a frontend deployment, where the frontend is synced after the backend.
• Another example could show a more complicated use case. The example in the video that is currently on the page makes use of sync phases/hooks as well.
• Another example could show how sync waves/phases cannot be applied to Application resources. See #8358 for reasonable-attempt example manifests. The solution statement here would say "there is currently no sync waves/phases solution to this problem".
• Another example could show the use of multiple sync contexts, and specifically how they operate independently. A reasonable-attempt manifest set could show a frontend deployment in one sync context and a backend deployment in a different sync context. This would refer back to the first example in this list, while highlighting that it is not possible to solve the same problem statement given that these two resources are in different sync contexts.

[nothingofuse]

heres a gotcha: argocd will not wait for or report on the completion of resource hooks.

In the linked video they state that argocd will "wait until they are all green before proceeding" when discussing resource hooks. what they mean by this is that it will create the job resource, but it will not wait for the completion of that job, and if the job fails that is in no way reflected in the apps health or synced statuses.

this fire-and-forget behavior makes resource hooks completely useless for the use cases the docs outline here: https://argo-cd.readthedocs.io/en/latest/user-guide/resource_hooks/