search

kubernetes / test-infra

Test infrastructure for the Kubernetes project.
Apache License 2.0
3.83k stars 2.64k forks source link

bootstrapping service-catalog use of prow for CI/CD #8291

Closed MHBauer closed 5 years ago

MHBauer commented 6 years ago

Goals

Current State

Wants

Ambiguous

first do's

/cc @BenTheElder This is probably missing key info you would like to see, so let me know what else is helpful.

Second Round of Questions from July 2018 - W1

Parallel Tracks of work:

My attempt at an overall document

BenTheElder commented 6 years ago

This all sounds reasonable to me, thanks for all the info!

I think the next step is probably configuring those things and then looking at what sorts of tests (unit, build, e2e?) you would want to migrate to Prow, if any.

BenTheElder commented 6 years ago

/area prow /assign /cc @cjwagner

BenTheElder commented 6 years ago

also, for a complete catalog of plugins and their usage see:

you may want to enable more of these in general for various labeling / automation tasks

MHBauer commented 6 years ago

@BenTheElder does it make sense to turn tide on without configuring it? I don't want it to accidentally start merging things that aren't ready by our selection.

MHBauer commented 6 years ago

right now to do a build we rely on docker existing. Then it's a make build or some such. Does this infra support a clone+make type operation? Or do we have to package the repo up into a ready to run image before it's possible to run the build & tests?

BenTheElder commented 6 years ago

@MHBauer:

MHBauer commented 6 years ago

I will look at pod utilities. Thank you. Probably not the time to enable tide yet. We use docker build two ways. A 'buildimage' for development and compiling of the output binaries, and also for release images.

MHBauer commented 6 years ago

stream of consciousness while reading...

It looks like I want to write an entry for presubmits in config.yaml https://github.com/kubernetes/test-infra/blob/master/prow/config.yaml#L408 .

I see https://github.com/kubernetes/test-infra/tree/master/prow#how-to-add-new-jobs afterwards.

I think I need a new prowjob. Reading more, I'm not sure if I want a presubmit, postsubmit, periodic, or batch. What is the context of "submit" in presubmit/postsubmit? What's being submitted where?

So as not to waste resources, maybe we can start with a clone job that does not do anything after the clone, or prints out some env-vars? This seems like it would be a good entry point for people starting in the future.

BenTheElder commented 6 years ago

@MHBauer submit might not be the best term, in this context submit == PR merge. presubmit -> testing PRs before they merge, post-submit -> testing triggered by merges, periodic == just runs on a schedule, used for EG, monitoring that the master branch is healthy with tests that are a bit prohibitive to be presubmits.

So as not to waste resources, maybe we can start with a clone job that does not do anything after the clone, or prints out some env-vars? This seems like it would be a good entry point for people starting in the future.

That sounds doubly excellent. @cjwagner we should put this in the docs. I plan to flesh out the docs around jobs some more once @krzyzacy finishes sorting out support for many job files (so you can just configure your jobs in your file instead of one 16k config...)

cjwagner commented 6 years ago

Cloning is provided by the pod utilities so your job container doesn't need to do any cloning. You could have a container that just echos some env, but I don't think that would really be all that helpful for future use.

@BenTheElder What exactly should we add to the docs? Are you suggesting adding an echo example? I have some simple examples already here: https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md#how-to-configure I suppose an echo example could be even clearer.

BenTheElder commented 6 years ago

Yeah an echo env (maybe also ls?) would be a pretty harmless example. We should also eventually add more directly useful ones like "use the go image and go test".

On Tue, Jun 19, 2018, 09:14 Cole Wagner notifications@github.com wrote:

Cloning is provided by the pod utilities so your job container doesn't need to do any cloning. You could have a container that just echos some env, but I don't think that would really be all that helpful for future use.

@BenTheElder https://github.com/BenTheElder What exactly should we add to the docs? Are you suggesting adding an echo example? I have some simple examples already here: https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md#how-to-configure I suppose an echo example could be even clearer.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/test-infra/issues/8291#issuecomment-398457409, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4Bq_-Z-up1yLfxM_yHERKiQWDAragvks5t-SNmgaJpZM4UfIkj .

MHBauer commented 6 years ago

@cjwagner https://github.com/kubernetes/test-infra/issues/8291#issuecomment-398457409 What is my job container?

Looking at that referenced config, I see image: gcr.io/test-images/bug-finder

Where does that come from? The top of the doc references the utilities as init-containers or sidecars, but I don't see that spec configuration in an example. Are they invisibly added?

@BenTheElder https://github.com/kubernetes/test-infra/issues/8291#issuecomment-398465145 As an example those would be nice. At the current moment, I do not have any idea what I get for free vs what I need to do.

Thanks for the definitions, we should start a test/dev glossary.

Not in any rush yet to start creating a job, so we can wait if necessary for the split support. I do not know what I would put into a job definition right now.

cjwagner commented 6 years ago

Prow runs jobs as kubernetes pods. The specification for a job includes a pod spec with a single container that runs the test. This is the job container that I am referring to, but test container is probably a more appropriate name. The bug-finder image doesn't actually exist. That is just an example test container image name.

You don't see specs for the pod utility containers because they are transparently added when plank creates the pod. Configuration for the pod-utility behavior is exposed through other fields in the job config. The pod utilities are explained here: https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md#pod-utilities This section details the available config fields that you may need or want: https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md#how-to-configure

cjwagner commented 6 years ago

At the current moment, I do not have any idea what I get for free vs what I need to do.

This section describes exactly that: https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md#what-the-test-container-can-expect If something is missing please let me know and I'll add it.

MHBauer commented 6 years ago

What is the test container? I am providing an image to run? I do not have one.

cjwagner commented 6 years ago

Yes, the test container is the main thing that you'll need to provide. It is what actually runs your tests. You may be able to directly use the DinD image that @BenTheElder mentioned or you may need to use that image as a base for your own custom image if you have additional dependencies that are not vendored in the repo.

MHBauer commented 6 years ago

Okay, all of the reading of this documentation has not been very clear about what I provide and how I provide it. Given that it's got a container/pod-spec, I can make assumptions, but it is much better to be very explicit.

Some statements I now think are true facts:

What is the name of the docker-in-docker image? Are we talking about the standard dind image? Is there a common repo of all test images?

BenTheElder commented 6 years ago

We have some common images in the images/ directory, the bootstrap image supports docker in docker amongst other things. We haven't migrated most jobs to the podutils just yet so it includes some other stuff in the entrypoint.. We'll need to create new images for use with the podutils specifically soon.

On Tue, Jun 19, 2018 at 2:20 PM Morgan Bauer notifications@github.com wrote:

Okay, all of the reading of this documentation has not been very clear about what I provide and how I provide it. Given that it's got a container/pod-spec, I can make assumptions, but it is much better to be very explicit.

Some statements I now think are true facts:

  • I give prow a image, prow runs my image in a pod.
  • All the standard pod rules apply, but prow mounts some extra stuff, invisibly to me.
  • My image is run as test-container
  • the working directory of test-container is the git clone directory by default

What is the name of the docker-in-docker image? Are we talking about the standard dind image? Is there a common repo of all test images?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/test-infra/issues/8291#issuecomment-398550033, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4Bqwojdm9XB3D_JZF6hFS8z__Gnc4bks5t-WsngaJpZM4UfIkj .

cjwagner commented 6 years ago

I give prow a image, prow runs my image in a pod I give my image to prow by making a prowjob My image is run as test-container

You give Prow an entire Kubernetes PodSpec as part of the job specification. Kubernetes Pods aren't Prow specific so we don't go into detail about how to write them. The only Prow specific rule is that you can only specify one container in the pod (this does need to be better documented).

These docs may also be helpful @MHBauer: https://github.com/kubernetes/test-infra/tree/master/prow#how-to-add-new-jobs When pod-utilities.md says In addition to normal ProwJob configuration... this is the normal configuration that it is referring to. That should be linked from pod-utilities.md and could be rephrased now that we intend for this to be the normal way to configure jobs.

All the standard pod rules apply, but prow mounts some extra stuff, invisibly to me. the working directory of test-container is the git clone directory by convention (due to some of those invisible things) prowjobs are defined in config.yaml (for now, to be split up in the future)

These points are already covered in these two documents. If there is something unclear about how they are written I can try and improve the wording. The organization of the main prow README and linking between documents is something I've been meaning to improve.

BenTheElder commented 6 years ago

Actually in addition, the idea is that you create a PodSpec with one container, which must specify the command / entrypoint, and that command should exit 0 on success and anything else on failure.

The podutils stuff adds the sidecar/init containers to the podspec at runtime (prior to scheduling the pod) to handle git checkout, log upload, artifact upload, etc.

I think we could make it clearer that the spec field is a podspec, and the requirements on number of containers etc... The pod utilites are still new. Most old jobs are using the horribly hacky jenkins/bootstrap.py script for logging etc. still as we are transitioning.

On Tue, Jun 19, 2018 at 4:14 PM Cole Wagner notifications@github.com wrote:

I give prow a image, prow runs my image in a pod I give my image to prow by making a prowjob My image is run as test-container

You give Prow an entire Kubernetes PodSpec as part of the job specification. Kubernetes Pods aren't Prow specific so we don't go into detail about how to write them. The only Prow specific rule is that you can only specify one container in the pod (this does need to be better documented).

These docs may also be helpful @MHBauer https://github.com/MHBauer: https://github.com/kubernetes/test-infra/tree/master/prow#how-to-add-new-jobs When pod-utilities.md https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md says In addition to normal ProwJob configuration... this is the normal configuration that it is referring to. That should be linked from pod-utilities.md and could be rephrased now that we intend for this to be the normal way to configure jobs.

All the standard pod rules apply, but prow mounts some extra stuff, invisibly to me. the working directory of test-container is the git clone directory by convention (due to some of those invisible things) prowjobs are defined in config.yaml (for now, to be split up in the future)

These points are already covered in these two documents. If there is something unclear about how they are written I can try and improve the wording. The organization of the main prow README and linking between documents is something I've been meaning to improve.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/test-infra/issues/8291#issuecomment-398574624, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4BqzocswrzjxJQvSzgCXJW2xYamd-Nks5t-YW9gaJpZM4UfIkj .

stevekuznetsov commented 6 years ago

There seems to be a lot of confusion here. @cjwagner @BenTheElder do we feel like we have a high-level document that describes how Prow runs k8s-native tests using plank? I wasn't expecting there to be uncertainty in that area but perhaps the different available execution backends (Jenkins, k8s simple, k8s decorated) are not laid out anywhere?

BenTheElder commented 6 years ago

Yes, we definitely need more docs regarding that.

On Wed, Jun 20, 2018 at 7:22 AM Steve Kuznetsov notifications@github.com wrote:

There seems to be a lot of confusion here. @cjwagner https://github.com/cjwagner @BenTheElder https://github.com/BenTheElder do we feel like we have a high-level document that describes how Prow runs k8s-native tests using plank? I wasn't expecting there to be uncertainty in that area but perhaps the different available execution backends (Jenkins, k8s simple, k8s decorated) are not laid out anywhere?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/test-infra/issues/8291#issuecomment-398767702, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4Bq6p8Asca7uWKXeRxBQGkmwURvai4ks5t-lq0gaJpZM4UfIkj .

MHBauer commented 6 years ago

I think part of this is "How do I know I want a prowjob at all? Why is it called a job? What even is a job?"

The organization of the main prow README and linking between documents is something I've been meaning to improve.

I think maybe I'm confused by the organization. I seem to end up jumping around a lot. A short tutorial with:

I'm psuedo writing this up as a big checklist myself, but still learning how things are expected to be. I'll read through those two docs again.

you create a PodSpec with one container, which must specify the command / entrypoint, and that command should exit 0 on success and anything else on failure

^ this is very clear and direct statement.

cjwagner commented 6 years ago

do we feel like we have a high-level document that describes how Prow runs k8s-native tests using plank?

I think the closest thing that we have is the "Life of a ProwJob" document: https://github.com/kubernetes/test-infra/blob/master/prow/architecture.md Its isn't a high level overview of the things that are important for users to know though, its more of a technical overview of how ProwJobs are triggered. It also isn't discoverable.

BenTheElder commented 6 years ago

so the first prowjob is setup, but we need to enable the trigger plugin to enable triggering presubmit jobs from pull requests. making a PR to do this now...

MHBauer commented 6 years ago

Added some more questions and discussion topics for the future in the OP. No need to answer, just want to have my thoughts writen down.

MHBauer commented 6 years ago

/cc @jeremyrickard @nikhita as other interested parties

MHBauer commented 6 years ago

Recommendation today was configuring tide is our immediate next step.

MHBauer commented 6 years ago

the result is thus #8710

MHBauer commented 6 years ago

we're going smoothy. prow/tide/whoever is running nicely. :fire:

BenTheElder commented 6 years ago

Whoo! 🎉

stevekuznetsov commented 6 years ago

@MHBauer if everything is still running smoothly, can we close this out?

spiffxp commented 5 years ago

/close I'm going to close this out, please /reopen if there's anything missing

k8s-ci-robot commented 5 years ago

@spiffxp: Closing this issue.

In response to [this](https://github.com/kubernetes/test-infra/issues/8291#issuecomment-432086162): >/close >I'm going to close this out, please /reopen if there's anything missing Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.