Cluster access for Red Hat (part 1, 100 physical nodes)

[jeremyeder]

First Name

Jeremy

Last Name

Eder

Email

jeder@redhat.com

Company/Organization

Red Hat

Job Title

Engineer

Project Title

Deploying 1000 nodes of OpenShift on the CNCF Cluster (Part 2)

What existing problem or community challenge does this work address? ( Please include any past experience or lessons learned )

We are interested in: Working through the operational concepts necessary to handle a large bare metal scale-out environment. Comparing the behavior of Kubernetes on OpenStack with Kubernetes on bare metal. Run our newly developed workload generators and test suite Utilizing newer features in Kubernetes to make use of bare metal hardware features.

Briefly describe the project

To compliment our earlier work on the CNCF lab (https://cncf.io/news/blogs/2016/08/deploying-1000-nodes-openshift-cncf-cluster-part-1) we would like to propose a full-lab scale test scenario once the CNCF lab is at full capacity. We will look to quantify improved performance when running on bare metal instead of virtualized. We will conduct some specific HTTP load testing and storage (persistent volume) performance testing.

Do you intend to measure specific metrics during the work? Please describe briefly

Yes, we will use our pbench framework https://github.com/distributed-system-analysis/pbench to capture metrics on each run. We expect this to involve Prometheus, a CNCF project, to the extent that we use it for gathering Kubernetes API server metrics.

Which members of the CNCF community and/or end-users would benefit from your work?

Kubernetes, Prometheus, end users who are looking to run high performance workloads on bare metal environments. Also fluentd if that is accepted (OpenShift uses fluentd for logging).

Is the code that you’re going be running 100% open source? If so, what is the URL or URLs where it is located?

Yes: https://github.com/openshift

Do you commit to publishing your results and upstreaming the open source code resulting from your work? Do you agree to this within 2 months of cluster use?

Yes, we have already open-sourced everything we write and we have shared significant amounts of data via blog and public-speaking engagements at industry conferences.

Will your testing involve containers? If not, could it? What would be entailed in changing your processes to containerize your workload?

Yes.

Are there identified risks which would prevent you from achieving significant results in the project ?

Not that we are aware of. We have good experience handling OpenShift at scale and we are proposing a two-phase approach where we prototype on 100 nodes (this proposal) with an adjacently-scheduled phase at full-lab scale of 1000 nodes.

Have you requested CNCF cluster resources or access in the past? If ‘no’, please skip the next three questions.

Yes.

Please list project titles associated with prior CNCF cluster usage.

Deploying 1000 nodes of OpenShift on the CNCF Cluster (Part 1)

Please list contributions to open source initiatives for projects listed in the last question. If you did not upstream the results of the open source initiative in any of the projects, please explain why.

Over 30 bugs were filed across projects such as Kubernetes, OpenShift and Ansible.

Have you ever been denied usage of the cluster in the past? If so, please explain why.

No.

Please state your contributions to the open source community and any other relevant initiatives

Red Hat is a fully open-source company. Red Hat is a platinum founding member of CNCF, a contributor to docker, kubernetes, openshift origin, and many more.

Number of nodes requested (minimum 20 nodes, maximum 500 nodes). In Q3, maximum increases to 1000 nodes.

Phase 1 (this request) 100 compute nodes and 10 storage nodes. Phase 2 (upcoming request) 1000 compute nodes.

Duration of request (minimum 24 hours, maximum 2 weeks)

Phase 1 two weeks. Phase 2 two weeks. Both phases must be adjacent in order for us to get what we need done in the 2 week periods.

With or Without an operating system (Restricted to CNCF pre-defined OS and versions)?

With, RHEL7.3

How will this testing advance cloud native computing (specifically containerization, orchestration, microservices or some combination).

We are working to push beyond control plane scalability to simulate realistic bare metal scenarios. This will include loading applications that represent an accurate mix of what we have seen in the wild. Being able to do this at higher scale levels will help us to discover best practices from an architecture standpoint as well as to help validate capacity planning formulas to see if they hold up at higher scale and load levels.

Any other relevant details we should know about while preparing the infrastructure?

[bprestonlf]

+1

[cncfclusterteam]

As mentioned in https://github.com/cncf/cluster/issues/22, we cannot provide you with anything close to 1000 nodes until expansion which takes place in Q1'17. Do you prefer to wait until then or would you rather have a first run with a smaller amount of nodes?

[jeremyeder]

Hi, it's more important that the two allocations are adjacent because of the 2 week time limit per allocation. Please hold this request until the 1000 nodes become available. Let me know if that makes sense?

[cncfclusterteam]

Sure, the request can wait.

[jeremyeder]

Hi @cncfclusterteam -- could you please let us know the timing/likelihood of this allotment please?

[cncfclusterteam]

Hi @jeremyeder, sorry for the silence. There are two things to consider which impact your request.

Firstly, the expansion will last 6 weeks more at minimum so the closest (though not certain) date I could point you to is around February 20. Our aim is to make the new nodes available till the end of Q1 so it might be pushed forward a bit.

Secondly, there might be trouble expanding the total number to 1000. The latest revision of README specifies that the maximum will be around 900 and that it is based on long-running allocations. We will try to provision the maximum available for you at that time but it might not reach 1000. Will this be ok for you?

[jeremyeder]

Hi @cncfclusterteam thanks. 900 nodes is fine, thank you. Question about the dates as your wording was a bit unclear to me:

We could have these approximately February 20th. Understood, that works for us. But what was meant by "available till the end of Q1 so it might be pushed forward a bit" ?

Also: our intent is not to use any virtualization, so two other questions have come up: will we be able to reprovision the OS on these nodes ourselves on-demand? Will we have out-of-band access (IPMI or similar) so that we can reach the serial console of the nodes for debugging or fixing crashed nodes?

[cncfclusterteam]

I meant that finishing expansion by ~February 20 is possible but our target is to do it in Q1'17 so, depending on the amount of problems we have with integrating the hardware, it could last a bit longer and end in March. The date above is an optimistic one :)

Reprovisioning the OS will be possible assuming you prepare your own PXE-based solution in your environment.

Out-of-band access will not be possible (no IPMI) but you will have access to the serial console and power off/on operations.

[jeremyeder]

OK. Thank you.

[jeremyeder]

Hi @cncfclusterteam could you please let us know how long we will have access to the gear?

[cncfclusterteam]

@jeremyeder the gear will be yours for as long as you provided in the details of this request, i.e. 2 weeks for phase 1 + 2 weeks for phase 2. Currently, this is the maximum time per request with the exception happening right now due to the expansion maintenance window. FYI the CNCF TOC is thinking of extending it but the decision has not been made yet.

Is that what you wanted to know?

[jeremyeder]

Yep, thank you. We're just trying to schedule our internal resources to be ready when the lab is, so any heads-up/lead time would be greatly appreciated!

[cncfclusterteam]

At the moment we cannot be sure how much of the infrastructure will be 100% ready after the expansion takes place which is crucial since your allocation will take all the resources available. Nevertheless, we will update everyone on the progress as soon as we have the data, I am sure there will be enough time to prepare for the tests.

[jeremyeder]

Hi, we will be deploying Ceph in this cluster and were wondering if we will be able to set the RAID controller config on 10 of the storage nodes to be JBOD?

[cncfclusterteam]

Raid controller present in storage nodes in PERC H730P which can be controlled via megacli tool. If you will need any assistance with the tool, we'll be happy to help.

---

From: Jeremy Eder [notifications@github.com] Sent: Tuesday, February 07, 2017 10:25 PM To: cncf/cluster Cc: CNCF Cluster; Mention Subject: Re: [cncf/cluster] Cluster access for Red Hat (part 1, 100 physical nodes) (#21)

Hi, we will be deploying Ceph in this cluster and were wondering if we will be able to set the RAID controller config on 10 of the storage nodes to be JBOD?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/cncf/cluster/issues/21#issuecomment-278144923, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AUOR4cVuVSo4V8WgP3CKZ8Nkah0B81kFks5raOEvgaJpZM4KoiD1.

Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.

[bengland2]

PERC H730P is really a LSI 3108 underneath, right? As long as all disks are not part of LUN containing operating system, then I think we're ok and can use MegaCLI if necessary. What is typical baseline storage configuration for RAID controller? @jeremyeder

[jeremyeder]

@cncfclusterteam Hi, any update on more specific availability date of gear?

[cncfclusterteam]

@jeremyeder if the request is limited to the current cluster capacity then the nodes can be provided even this week. However, the 1000 nodes - as requested in part 2 - are not yet reachable and due to delays we are experiencing we are not able to be more precise about the dates. I am sorry for the inconvenience this might cause.

[jeremyeder]

@cncfclusterteam sure, I was referring to the part 1 request. The sooner the better, this week would be great. Is there a chance we could have a quick call with the lab team once the gear is turned over? (something a little more high bandwidth than this github issue).

[cncfclusterteam]

Sure, we can have a call - we'll exchange the details on e-mail during the provisioning process which should take place by tomorrow.

[jeremyeder]

@cncfclusterteam Hi all -- any chance we can have an update/call today (Friday)?

[cncfclusterteam]

@jeremyeder we wrote you an email an hour ago, you'll find more details there!

[jeremyeder]

@cncfclusterteam got it -- if we have questions/issues, will reply to mail. Thanks!

[cncfclusterteam]

Hi @jeremyeder,

We hope the time spent with the cluster has been productive. I am writing to inform you that we would like to clean up the nodes for next tenants. Please let us know when we can take them back to the free pool.

Thank you, CNCF Cluster Team

[jeremyeder]

Hi @cncfclusterteam We've gotten all our data off. Thank you very much for access to the gear!