search

kubernetes / test-infra

Test infrastructure for the Kubernetes project.
Apache License 2.0
3.82k stars 2.63k forks source link

The test-infra architecture diagram is out of date #13063

Open spiffxp opened 5 years ago

spiffxp commented 5 years ago

/kind documentation /kind bug

I visited https://github.com/spiffxp/test-infra/blob/write-some-docs/docs/architecture.svg and saw an architecture that is wildly out of date, eg:

I count this as a bug since this diagram is one of the first links in our repo, and it's definitely misleading.

stevekuznetsov commented 5 years ago

:yikes:

spiffxp commented 5 years ago

While this bug is still open, I recommend checking out the SIG Testing Intro presented a KubeCon NA 2018, find video and slides at https://github.com/kubernetes/community/tree/master/sig-testing#presentations. The final diagram is a bit dense so I tried to walk through how we got to the rube goldberg thing we have going on now.

spiffxp commented 5 years ago

/sig testing /priority important-longterm

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/test-infra/issues/13063#issuecomment-569551719): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
spiffxp commented 4 years ago

/reopen /remove-lifecycle rotten /help-wanted

k8s-ci-robot commented 4 years ago

@spiffxp: Reopened this issue.

In response to [this](https://github.com/kubernetes/test-infra/issues/13063#issuecomment-570077208): >/reopen >/remove-lifecycle rotten >/help-wanted Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

spiffxp commented 4 years ago

/remove-lifecycle stale /help

k8s-ci-robot commented 4 years ago

@spiffxp: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes/test-infra/issues/13063): >/remove-lifecycle stale >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
hasheddan commented 4 years ago

@spiffxp I'd like to help out here both in creating a new diagram and documenting other parts of the repo. For instance, I was recently looking at the kubekins-e2e image because it is so prevalent across jobs and there is pretty minimal documentation on it and other Docker images used. I wonder what would be a good format for a sort of "Intro to test-infra" guide. This may be outside the scope of this issue, but I wanted to see if you had thoughts on the matter?

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/test-infra/issues/13063#issuecomment-685025757): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
spiffxp commented 3 years ago

/remove-lifecycle rotten /reopen Maybe this quarter? Or super open to any help anyone else wants to provide?

k8s-ci-robot commented 3 years ago

@spiffxp: Reopened this issue.

In response to [this](https://github.com/kubernetes/test-infra/issues/13063#issuecomment-690624437): >/remove-lifecycle rotten >/reopen >Maybe this quarter? Or super open to any help anyone else wants to provide? Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
RobertKielty commented 3 years ago

Hi @spiffxp Do you have a source file for SVG file or drawing tool preference?

I quickly mocked up a replica of the out-of-date SVG using dia which I would be happy to share with you.

leeyspaul commented 3 years ago

Nothing helpful to comment on but simply adding that I would love to see this as someone who would like to learn how it all works.

spiffxp commented 3 years ago

@RobertKielty No drawing tool preferences as long as it can be updated by community members

https://docs.google.com/presentation/d/1HOQ2df_AT-vIuz-JNaJol2oiGq84m50h9T49_5WgEaI/edit#slide=id.g4a3c5dc660_0_539 are the slides I walked through back in dec 2018

spiffxp commented 3 years ago

This doc is now also missing concepts such as:

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

spiffxp commented 3 years ago

/remove-lifecycle stale /lifecycle frozen /milestone v1.21

hh commented 3 years ago

/assign @BobyMCbobs

spiffxp commented 3 years ago

Was presented at k8s-infra meeting today. Thank you for picking this up

Good followup: depict the clusters, projects and orgs involved. Service cluster, build clusters, Google.com, kubernetes.io, etc

pacoxu commented 3 years ago

Kubernetes Test Infrastructure (Last updated 2020-03-04)

https://raw.githubusercontent.com/kubernetes/test-infra/master/docs/architecture.svg

2020 -> 2021 😄

BenTheElder commented 3 years ago

this was updated pretty recently, should the follow-up be a follow-up or should we keep tracking here? if the latter we should update the milestone

listx commented 3 years ago

Velodrome is still in the SVG.

spiffxp commented 3 years ago

/milestone v1.23 What I discussed during the most recent meeting is:

  1. there are likely still some stale things (good catch @listx)
  2. being able to draw boxes around what lives within google.com and what lives within kubernetes.io would be helpful for wg-k8s-infra

I'll try to clarify 1 since that's a little too vague to be actionable right now

spiffxp commented 3 years ago

I've taken an alternate crack at this over in https://github.com/kubernetes/k8s.io/pull/2635

It doesn't get a much into the mechanics of Prow, but more all of the other things that prow interacts with

spiffxp commented 3 years ago

/assign I'm not removing help as I could still use plenty of help in suggesting edits and better tools to use to document this stuff

spiffxp commented 2 years ago

I browsed through https://xosh.org/text-to-diagram/ to see if I could find a non-proprietary tool that could accomplish the same sort of thing I did with https://github.com/kubernetes/k8s.io/pull/2635

spiffxp commented 2 years ago

cc @hh

MeNsaaH commented 2 years ago

@spiffxp have we considered https://github.com/mingrammer/diagrams? Looks great.

ameukam commented 2 years ago

GCP released a new diagramming tool : https://cloud.google.com/blog/topics/developers-practitioners/introducing-google-cloud-architecture-diagramming-tool

MeNsaaH commented 2 years ago

This is heavily bundled with GCP. What of AWS workloads?

BenTheElder commented 2 years ago

This is heavily bundled with GCP. What of AWS workloads?

test-infra runs on GCP (due to https://www.cncf.io/announcements/2020/12/17/cloud-native-computing-foundation-receives-renewed-3-million-cloud-credit-grant-from-google-cloud/), FWIW.

When it isn't reasonable to DIY infra all the way down, we use off the shelf offerings sometimes. Sometimes not often enough 🙃

spiffxp is out for a while right now, I think mostly it would be good to have a rendered result like we have now.

vatsalparekh commented 1 year ago

if this is still relevant, would like to be a part /assign

listx commented 1 year ago

At least for the Prow components, there is an architecture diagram at https://docs.prow.k8s.io/docs/overview/architecture/.