search

crosscloudci / cross-cloud

Cross-Cloud - multi-cloud K8s provisioner for CNCF CI Project
https://cncf.ci
Apache License 2.0
167 stars 59 forks source link

add Rook to cross-cloud CI testing #137

Open jbw976 opened 6 years ago

jbw976 commented 6 years ago

As per discussion on the CI mailing list, we would love for Rook to be integrated with the cross-cloud CI testing provided by the CNCF.

Our current Jenkins setup is hosted on an instance in Amazon EC2 and can be accessed at https://jenkins.rook.io.

That master instance dynamically brings up slaves in EC2 and GCE to deploy various versions of Kubernetes to those public cloud environments and run end to end integration testing there. Here's a recent build from master that shows the small matrix we currently deploy and test on: https://jenkins.rook.io/blue/organizations/jenkins/rook%2Frook/detail/master/559/pipeline

We would love to expand this matrix and integrate with the same CI services/environment that other CNCF projects are using. Thanks!

/cc @bassam @travisn

travisn commented 5 years ago

We would love to get going on this. Where can we find more info? @denverwilliams @lixuna any pointers? thanks!

taylor commented 5 years ago

Hi, @travisn.

We'd welcome help in getting Rook on the CNCF CI dashboard as well as collaborations on CI/CD in the community as well.

It sounds like Rook is already doing some CI which maybe someone would like to talk about at the next CNCF CI WG meeting (Jan 22nd).

Regarding the cncf.ci dashboard we have short and longer term options. We don't have a quick answer to help you get started unfortunately. While we are working on new documentation and platform updates to allow projects to help directly to add themselves, that work is still in-progress and not yet ready. In the short term adding a project to the cncf.ci dashboard requires re-implementing part of a projects build/deploy/test process in Gitlab and that is not a self-service process unfortunately. I'll go more into that below.

Longer term, we are planning a new iteration of the dashboard which includes integration+collaboration with projects after Kubecon Seattle 2018. Maybe we can meet in-person in Seattle at Kubecon to discuss both the short and longer term options.

An overview of what how the system currently works should be helpful.

Currently the dashboard is populated based on data pulled from the underlying CI platform (Gitlab) or pulled from external CI systems (if we have created an integration). At the moment the only integrations to external CI systems are for build artifacts.

K8s is currently built from source (release + HEAD on master). Then k8s is deployed (using the (cross-cloud provisioner) to all of the supported cloud providers.

After the builds artifacts are collected for the projects deployments to all of the k8s clusters happens. At which point integration tests can be run if any exists.

Some options for integration of Rook deploy+tests on the dashboard:

  1. Gitlab-only: Understand + implement the build and deployment stages for Rook in gitlab.cncf.ci
  2. Gitlab+Jenkins: Integrate Jenkins artifacts for builds. Implement deployment in Gitlab

Option 1 has been the default choice. It would be similar to what you have on Jenkins. Gitlab has a configuration file that has the commands to create a build and run whatever other commands are needed. We would need to see how Rook would be deployed and tested with k8s. Implementation in the various gitlab stages including running the integration tests would be similar to Jenkins.

Option 2 is similar, expect the build artifacts are pulled from Rook's Jenkins server (or some other artifact repository).

Either option will require taking build and test process used in the Rook Jenkins server and re-implementing in the Gitlab system.

Fyi, A future option we hope to provide is pulling all status information from a projects CI system and not running anything in a different system. It would be something like, Rook Jenkins + CNCF.ci Dashboad. Help Jenkins to use cross-cloud for provisioning k8s clusters and deploy Rook to those clusters. Integrate Rook Jenkins results with Cross-Cloud CI status repository which provides data for the Dashboard. The different Cross-Cloud CI components can be used outside of the implementation for the CNCF dashboard.

So implementing one of the current options...

The first thing would be determining where/how Rook fits in with the current system and dashboard. Since Rook provides a storage orchestrator for K8s I'm not sure where it would fit in and what integration tests we want to highlight. The other projects run on the clusters themselves in containers. We need to determine that first. Then implementation of the steps in Gitlab for build + integration tests. Otherwise that would be something someone on the Cross-Cloud CI team would work on.

/cc @denverwilliams @lixuna

travisn commented 5 years ago

@taylor Thanks for all the background on what it might look like to integrate with the CNCF CI. We would really prefer to have the builds and integration tests all running as you described in option 1. Unfortunately I didn't think to think to discuss this at kubecon. We can plan on joining the community meeting next month to discuss.

jbw976 commented 5 years ago

Thank you for the discussion today in the CI working group meeting. I think the CNCF CI v2.0 slides that were shown are going in a good direction that will result in more CNCF projects integrating with the CNCF CI efforts. The Rook project is definitely interested in integrating as well. From the roadmap, it looks like the March timeframe will add documentation for how to add a project to this effort, so we can engage further then.

We initially had hopes for the CNCF hosting the full build/integration environment for us and getting rid of the burden of our own Jenkins server that we are hosting and managing on our own. But it sounds like that is not the direction that will be supported going forward, so we will need to look into how best our existing CI efforts with Jenkins can be integrated into the v2 efforts.

dankohn commented 5 years ago

Jared, thanks for participating in the call. The cross-cloud group is definitely not aiming to take over CI responsibilities for any of the 20 graduated or incubating CNCF projects.

However, that does not mean the Rook project is on it's own for CI. Please open a request at https://github.com/cncf/servicedesk and we would be happy to work with you, perhaps by helping transition Rook to a hosted Jenkins environment, or perhaps an alternative CI platform.

As long as you are producing container images as artifacts in your CI process, cross-cloud should be able to integrate you into their testing matrix fairly quickly.

Dan Kohn dan@linuxfoundation.org Executive Director, Cloud Native Computing Foundation https://www.cncf.io +1-415-233-1000 https://www.dankohn.com

On Tue, Jan 22, 2019 at 3:43 PM Jared Watts notifications@github.com wrote:

Thank you for the discussion today in the CI working group meeting https://docs.google.com/document/d/1NA4N6PvNEkHX1yzaDFr19Xlru-amRxNi2pliqudmYNA/edit#heading=h.f4yxlzcq0ckl. I think the CNCF CI v2.0 slides https://docs.google.com/presentation/d/1NbRstXKJU7y7rOV60gV4Hg2TY2sM6l3rche73XpiRRk/edit#slide=id.g4c4fbf5639_0_143 that were shown are going in a good direction that will result in more CNCF projects integrating with the CNCF CI efforts. The Rook project is definitely interested in integrating as well. From the roadmap https://docs.google.com/presentation/d/1NbRstXKJU7y7rOV60gV4Hg2TY2sM6l3rche73XpiRRk/edit#slide=id.g4cae776d61_3_44, it looks like the March timeframe will add documentation for how to add a project to this effort, so we can engage further then.

We initially had hopes for the CNCF hosting the full build/integration environment for us and getting rid of the burden of our own Jenkins server https://jenkins.rook.io/ that we are hosting and managing on our own. But it sounds like that is not the direction that will be supported going forward, so we will need to look into how best our existing CI efforts with Jenkins https://jenkins.rook.io/ can be integrated into the v2 efforts.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/crosscloudci/cross-cloud/issues/137#issuecomment-456555900, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8MBuLDh9dZBzDalEWtvfHNegQIucTTks5vF3fvgaJpZM4TCJtF .

taylor commented 5 years ago

Hi, @jbw976.

It sounds like we have two efforts that can be worked on independently:

  1. adding Rook to the cncf.ci dashboard
  2. general CI hosting/help for the rook project

As we already have a Jenkins integration (for ONAP) I think integration with Rook existing Jenkins will lower the effort for adding the project to the dashboard. As we are closer to adding Rook we'll reach out to discuss items like published artifacts for different stages, smoke tests deployments, and e2e tests.

Regarding CI for the Rook project I suggest looking at hosted options rather than maintaining your own CI system (as that is not something desired within the project). There are many choices which provide free hosting for open source projects, including:

Hosted CircleCI and TravisCI are being used by many projects in the CNCF and K8s community.

If it's decided to continue self-hosting, I suggest looking at alternatives to Jenkins which should be easier to maintain such as Buildbot or CircleCI.

dankohn commented 5 years ago

We do not recommend LF Infra management as we are trying to reduce the workloads that run in that environment and move more stuff to public services.

Dan Kohn dan@linuxfoundation.org Executive Director, Cloud Native Computing Foundation https://www.cncf.io +1-415-233-1000 https://www.dankohn.com

On Thu, Jan 24, 2019 at 7:32 PM Taylor Carpenter notifications@github.com wrote:

Hi, @jbw976 https://github.com/jbw976.

It sounds like we have two efforts that can be worked on independently:

  1. adding Rook to the cncf.ci dashboard
  2. general CI hosting/help for the rook project

As we already have a Jenkins integration (for ONAP) I think integration with Rook existing Jenkins will lower the effort for adding the project to the dashboard. As we are closer to adding Rook we'll reach out to discuss items like published artifacts for different stages, smoke tests deployments, and e2e tests.

Regarding CI for the Rook project I suggest looking at hosted options rather than maintaining your own CI system (as that is not something desired within the project). There are many choices which provide free hosting for open source projects, including:

Hosted CircleCI and TravisCI are being used by many projects in the CNCF and K8s community.

If it's decided to continue self-hosting, I suggest looking at alternatives to Jenkins which should be easier to maintain such as Buildbot https://buildbot.net/ or CircleCI.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/crosscloudci/cross-cloud/issues/137#issuecomment-457411006, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8MBhBMbRTFiFMh_lKlR_cwqAY1PYbpks5vGlC0gaJpZM4TCJtF .

jbw976 commented 5 years ago

Thanks for the pointers and discussion. It sounds like the short term and easier goal of adding rook to the cncf.ci v2 dashboard will be a great first step. Please let us know when the documentation or process for adding projects (march on the roadmap) is ready and we'll be happy to engage.