EPIC: Evaluate Engines for Build, Orchestration and Deployment for containers

[krishnapaparaju]

One of key goals of Build.Next initiative is to choose the right build engine for performing CI&CD of container images. Evaluation of currently available tools is key to understand current offerings and evaluation can be carried out with these criteria..

1) Convert Source to artifact 2) Deployable on Openshift 3) Support Multi tenancy 4) Able Retrieve status of jobs, logs 5) Orchestration of parallel, serial pipeline steps 6) API support for integrations 7) Ecosystem of Plugin support 8) Canary deployments 9) Blue Green deployments 10) Community adoption 11) Health of upstream 12) License 13) Developer ease for pipeline DSL 14) Authentication with OSIO Auth

Engineering / architecture specific criteria..

    1. Secret isolation between master and slave nodes

2.  NO Resource starving from a specific CI/CD job

3.   Load testing for around 100 concurrent users

4.  Resource / cost metrics for the deployment

5.   How to stream logs from master node ?

6.  How reliable scheduling inside the engine ?

7. How plugins developed, get tested at multiple environments ?

8. How master handle huge list of pipeline jobs ? 

9. Reliability of required plugins  (eg: for Jenkins engine, Jenkins-Kuberenetes plugin) ?

These are CI/CD tools require evaluation:

a) Jenkins #3581 b) JenkinsX #3602 c) Drone #3603 d) Prow #3606 e) Zuul #3604 f) Concourse #3577 g) TravisCI #3578 h) Spinnaker #3617 i) Quay (CoreOS) #3729 j) https://brigade.sh/ #3811 k) Argo #3805

[lordofthejars]

Prow is not a CI/CD tool

[jaseemabid]

There is a lot going on here!

One of the big lessons from the f2f meeting was to consider the service as 3 different verticals Build, Orchestrator and Deploy. Unless we look at it so, none of the tools above will even remotely match our needs. I've been thinking about the problem lately and I'd like to add a 4th vertical - Engineering Sanity. Even if Jenkins did all 3 well, its not something that is sane enough to build a saas on.

If for example, A has the best plugin model but is terrible at Canary deployments while B has great upstream, but a terrible API what would we do?

Realistically, we are looking at multiple micro services and they need to be evaluated as such. Not all those parameters are equal either, so we need a model for assigning priority and weights.

Scoping this into a few services, evaluating each in isolation, knowing what to pick and what to build will take some serious disciplined effort. I'll try to come with an initial list which others can build upon sometime this week.

[jaseemabid]

@krishnapaparaju This initial list needs to be thought through carefully before it can be worked upon.

Jenkins is not even a remote option, let's not have the same discussion we've had for last 6 months one more time. Jenkinsx considering its origins is not something I want to touch with a 100m pole.

Prow has no multi tenancy and they are at the early stages of it, while its execution engine looks kind of neat (from the little time I spent reading their code). Its not a complete CI/CD solution like @lordofthejars mentioned, something that came out of k8s community to test k8s well. Gitlab is literal direct competition, I'm not sure we will be able to work with it even though there are good bits there. Bamboo is proprietary software.

[lordofthejars]

I pick up Concourse CI to see how it works inside Kubernetes because it seems there is no official support but I want to see deeply.

[rupalibehera]

@jaseemabid @lordofthejars , please create a issue with the tool you guys are evaluating and link to this epic

[GeorgeActon]

Is there a link into planner for this?

[maxandersen]

Scoping this into a few services, evaluating each in isolation, knowing what to pick and what to build will take some serious disciplined effort. I'll try to come with an initial list which others can build upon sometime this week.

Getting a structured approach is key here. +100

[ldimaggi]

Scoping this into a few services, evaluating each in isolation, knowing what to pick and what to build

+1 - smaller pieces, easier to maintain, etc.

[joshuawilson]

@krishnapaparaju this should be listed under https://openshift.io/openshiftio/openshiftio/plan/detail/2034 as an Experience.

[jaseemabid]

@joshuawilson The link says "Continuous delivery may increase success" and a text that is almost lorem ipsum. How is this relevant?

[lordofthejars]

Concourse CI: https://github.com/openshiftio/openshift.io/issues/3577

[rupalibehera]

Travis Ci: https://github.com/openshiftio/openshift.io/issues/3578

[jaseemabid]

Moving the evaluation criteria to a markdown file here because its easier to keep updating than a comment here. https://github.com/fabric8io/buildnext/blob/master/Evaluation.md

[krishnapaparaju]

These criteria kinda more specific than list mentioned in the issue, but missing couple of important criteria. Would be great if you can expand the original list of criteria mentioned. I guess we take out Bamboo, GitLab from the original list (Deleted). Any compatibility , we might wanna stick with OpenShift than K8S.. a bit more specific as we would be deploying our infra. on OpenShift. We surely wanna include Jenkins as part of evaluation..

[lordofthejars]

I also miss some subpoints in section 2 like: is cloud-native? External dependencies of the system. If it is scalable. And how difficult is to scale up.

Also how multitenant is covered.

[gastaldi]

Semaphore CI (https://semaphoreci.com) is a much better alternative than CircleCI and Travis for Java projects, in case you're considering other options ;)

[jaseemabid]

@krishnapaparaju @lordofthejars As mentioned in mattermost, the draft above is definitely not complete, I just left it here for an early preview. I'll try to work on it over the next couple of days.

[sthaha]

@jaseemabid @krishnapaparaju

Build a language specific package and push to a repository.

Is this something the build tool must address? i.e. should it be able to build and publish npm, gems, jars etc ? If so, what is the minimal set of languages that build.next must support?

[sthaha]

@jaseemabid

How does the k8s integration work?

Why k8s instead of openshift?

[sthaha]

@jaseemabid @krishnapaparaju

• Shouldn't we also look at how much work is involved in integrating OSIO auth (authentication and authorisation) into the engine?

• What about multi-tenancy?

[krishnapaparaju]

@sthaha integration into OSIO Auth is key, the front ending build service (API endpoint) would be integrated into OSIO Auth, we have couple of working API end points already been integrated with OSIO Auth. Multi-tenancy is already present in the list of criteria mentioned.

[jaseemabid]

Thanks for the comments folks, I've updated the draft with hopefully all of the concerns raised here answered.

@krishnapaparaju Even though I'm strongly against the idea of going ahead with Jenkins, please go ahead if you think it might be an option. I could share the notes from the discussion I had with the upstream openshift folks regarding their experience with Jenkins without any of the f8 bits - that wasn't very pleasant either. @gastaldi here might be able to give you a ton of insight into the issues.

@lordofthejars Added cloud native as second point under the Build section. Thanks.

@gastaldi, Added semaphore to a build as a candidate. If you are familiar with the service, will you be able to help us by answering the questions in the evaluation metrics once we (sort of) finalize it? This will help us speed up the evaluation :zap: Hopefully its not Java specific and can build other tools as well.

@sthaha, thanks for poking holes into my arguments as usual :heart:

Build a language specific package and push to a repository.

I've reworded the section to clarify the idea. It was indeed ambiguous.

Why k8s instead of openshift?

I used k8s and openshift as synonyms in the first draft. I've fixed it. I meant os.

:heavy_check_mark: Addressed auth and multi tenancy.

[gastaldi]

Added semaphore to a build as a candidate. If you are familiar with the service, will you be able to help us by answering the questions in the evaluation metrics once we (sort of) finalize it?

@jaseemabid sure, I think I may be able to help by answering that, I've had a good experience with it while configuring the Fabric8 Launcher builds.

[stevengutz]

@krishnapaparaju This should be in Planner and tracked in the I-Train plan.

[jaseemabid]

@Steve, are you suggesting we use planner for the actual planning or use it as a GitHub mirror?

On Thu, 24 May, 2018, 4:00 PM Steve Gutz, notifications@github.com wrote:

This should be in Planner and tracked in the I-Train plan.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openshiftio/openshift.io/issues/3570#issuecomment-391666726, or mute the thread https://github.com/notifications/unsubscribe-auth/AAkuckbgD_KsorKYSaRPR-JqqOwbVp9Mks5t1ouwgaJpZM4UAu-j .

[stevengutz]

@jaseemabid We have moved all of our planning to our own Planner tool. An Epic (aka an Experience in SDD) is something that should be in Planner rather than GitHub. In fact even new Stories (aka Features) should be in Planner because that's the only way they can be targeted at a train/sprint and be tracked in a plan now.

Feel free to use GH if you want but you will be responsible for keeping the work item here and it's clone in Planner in sync (e.g. If you put it in progress here you will also need to put it in progress over in Planner). Sound crazy? Yup, so just use Planner for all new things. :-).

For now though you will still need to use GH for task level things since Planner doesn't know how to handle PRs yet

[jaseemabid]

@stevengutz all during the development of planner (and OSIO as a whole), it was stated several times that we are not the intended audience of the product. As an engineer, I really don't want anything more complicated than a simple issue list with tags (I think Github issues is almost perfect for the job) definitely not an over complicated mess that is going to be a huge productivity hole. The app doesn't even work on mobile or Firefox desktop. This is probably a very bad idea.

[stevengutz]

@jaseemabid So the original messaging was probably not correct since we are an engineering team and we also plan things :-). We also want to use our own tool to work out the obvious UX issues before we drop it on our users.

The good new from your perspective is that other than going in on occasion to set something in progress or complete, it won't really impact engineers. We trying to initially contain most of the work to leads and paper pushers like managers and PMs

[lordofthejars]

I have created this issue for evaluating Flux https://github.com/openshiftio/openshift.io/issues/3667 I really really love the idea behind it, as more videos I see about it the more I like it. I am not saying Build.Next should be based on that (I am trying to see if it can be used outside weavecloud) but a lot of concepts and simplifications they are taking are just awesome.

[sspeiche]

Please add https://brigade.sh/ to the list

[krishnapaparaju]

@sspeiche Have added brigade to the evaluation list. Fits into orchestration of high level CICD.next services, not specific to the core engine of CICD.next

[lordofthejars]

Anyone knows argo?https://github.com/argoproj/argo/blob/master/demo.md

El mar., 12 jun. 2018 19:24, SriKrishna notifications@github.com escribió:

@sspeiche https://github.com/sspeiche Have added brigade to the evaluation list. Fits into orchestration of high level CICD.next services, not specific to the core engine of CICD.next

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openshiftio/openshift.io/issues/3570#issuecomment-396669153, or mute the thread https://github.com/notifications/unsubscribe-auth/ABcmYUASTwNLmfiBl98Rd_XbTiL4yxKCks5t7_lYgaJpZM4UAu-j .

[arilivigni]

@krishnapaparaju is there a reason we can't use knative build for this? https://github.com/knative/docs/tree/master/build#key-features-of-knative-builds https://github.com/knative/docs/blob/master/build/builds.md

[sspeiche]

@arilivigni whatever is done should be done using or contributing to openshift for supporting this. It would note make sense for osio to take a different route other than to validate an approach. I'll bring you into a proposal I'm putting together

[arilivigni]

@sspeiche agreed that is exactly what I am going for rather than some of the options listed here that are not a part of OpenShift.

[sthaha]

@arilivigni

is there a reason we can't use knative build for this?

By this, do you mean: all of the below?

• [ ] Support Multi tenancy
• [ ] Able Retrieve status of jobs, logs
• [ ] Orchestration of parallel, serial pipeline steps
• [ ] API support for integrations
• [ ] Ecosystem of Plugin support
• [ ] Canary deployments
• [ ] Blue Green deployments
• [ ] Community adoption
• [ ] Health of upstream
• [ ] License
• [ ] Developer ease for pipeline DSL
• [ ] Authentication with OSIO Auth

The goal of the epic was to find a better replacement for Jenkins. I have very little knowledge about Knative. Do you think it ticks all or most of the above?

[stevengutz]

@sthaha This needs to be moved to Planner and this one should be closed. It looks big so please create sized stories as well.

[arilivigni]

@sthaha maybe I don't understand this but I was considering how Knative build could be used for the build service not the CI/CD part. I look at them as separate. Correct me if I am wrong but you are using Jenkins/fabric8 for both currently. So to me if we can leverage the Knative build to replace the build service part https://github.com/knative/docs/tree/master/build#key-features-of-knative-builds. Obviously this is something that is not available now, but long term would be a solution.

[arilivigni]

@sspeiche any update on bringing me into that proposal?

[brujoand]

Any progress on this or has Jenkins been chosen for the foreseeable future?