Update pod readiness condition

[solmonk]

What type of PR is this?

feature

Which issue does this PR fix:

560

What does this PR do / Why do we need it:

Please refer to the issue description of #560 for the background.

• Model build: now ready and non-ready targets are both registered (unless they are terminating)
• Target synthesizer: updates pod readiness condition. Only operates when the pod has the readiness gate. It is mostly injected by webhook, but people may add gate by themselves (that's how I tested)
  • Might be slightly off placed but couldn't find better place to fit in.
  • Status logic is mostly in the synthesizer as well, I tried to comment as much as possible.
  • I put status update logic in the PostSynthesize step, as otherwise health check status can be misleading w/ unused status, etc.
• Target manager: stale target identification logic slightly improved

If an issue # is not available please add repro steps and logs from aws-gateway-controller showing the issue:

Testing done on this change:

Tested in a simple scenario where I have 2 pods in the deployment, and adding pods. (w/ readiness gate)

• Confirmed pods are starting with non-ready state
• Confirmed targets are added while pods are non ready
• Confirmed pods are becoming ready on the next reconciliation loop

Will update after further test.

Automation added to e2e: e2e test cases coming soon

Will this PR introduce any new dependencies?:

Will this break upgrades or downgrades. Has updating a running cluster been tested?:

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[liwenwu-amazon]

Can you add some detail design note on how often do controller reconcile on all lattice TG targets? e.g. Is it on the timer-interval?

Does it pull lattice health state for all targets/pods?

[solmonk]

No - the controller only listens on endpoint event, and pulls lattice health state if any of the endpoint is not ready. This means the controller does not go in opposite direction (making ready pods non-ready)

The purpose of readiness is about "disabling traffic while keeping pods alive" which is already handled by VPC Lattice HC, so I believe this should not block any use cases, although we can support this in the future.

[liwenwu-amazon]

No - the controller only listens on endpoint event, and pulls lattice health state if any of the endpoint is not ready. This means the controller does not go in opposite direction (making ready pods non-ready)

What happens if targets HC are not ready ? Will controller retry later or just busy waiting

[liwenwu-amazon]

To make the PR smaller, Is it possible to have a separate CR just on endpoint slice change?

[solmonk]

controller just goes to another reconcile loop, not busy waiting. actually a separated PR was already up if you're interested - https://github.com/aws/aws-application-networking-k8s/pull/604 If this gets merged first I can rebase this PR to reduce the diff.

[solmonk]

• Now findStaleTargets skips draining targets, as there is no need to deregister them.
• Added e2e test.
• I wasn't able to make multi-TG use case. It turned out more complex than I expected. Since only one route is processed at a time, we can't calculate the readiness if a pod is shared between multiple TG across different HTTPRoutes. If we want to deal with this I think we should approach differently, such as injecting a separate readiness condition for each route.