Develop a plan to migrate from Aporeto to OCP 4 SDN

[mitovskaol]

As the Platform Services team supporting the Openshift Platform, we would like to develop a plan for the possible migration from Aporeto SDN technology to the Openshift 4 build-in SDN capability.

The following has been confirmed:

• The NetworkPolicy Mode is the configuration option in OCP 4 that enables the networkPolicy objects. This configuration is not explicitly managed in our current OCP4 cluster deployments, so the default setting is being applied. The OCP 4 clustere config will need to be "tweaked and tested" to make our network choices and configurations explicit vs implied by default values.
  • OCP 4 SDN does not support hierarchical network security policy organization "out-of-the-box" , however, custom automation can be developed to implement this capability.
  • OCP 4 SDN does not work outside of Openshft and therefore cannot be used for Zone B pilots. However, the Zone B pilots came up with workarounds (using a certificate based authentication, database credentials and using a unqiue egress IP for the namespace for connecting into Zone B that they have been using while waiting for Aporeto to be available in Zone B. Also, there are solutions that are being developed on the market that will add support for non-Kubernetes environments and can integrate with OCP 4 SDN
  • Overall, with the exception of the missing capabilities mentioned above, OCP 4 SDN provides capabilities that cover most of use cases that are currently covered by Aporeto on the Openshift 4 Platform
  • we need to develop a migration plan for product teams to switch namespaces from using Aporeto NSP to using OCP 4 SDN’s network policies (NP): teams will be asked to to work through their environments bottom-up (dev, test, tools and then prod) and replace Aporeto’s NSPs with OCP NPs in each environment. This means deleting the NSP CRD and replace it with OCP 4 SDN NP CRD in each environment. We do not anticipate any problems with It is having some environments that use NPs and others that use NSPs during the migration period.

DoD: Document a plan that addresses the following:

• [x] OCP 4 config changes and estimates for making and testing them in KLAB and Silver (should we add CLAB too?)
• [x] Make a list of Platform Services apps in Silver that will be migrated first
• • [ ] Rocket.Chat
• • [ ] Reggie
• • [ ] bcdevexchange.org
• • [ ] Mobile Signing Service (MSS)
• • [ ] Repomountie
• • [ ] FOI Request
• [x] Self-help documentation for product teams that guides them through the NSP migration process
• [x] How the current Zero-Trust documentation developed for Aporeto will be updated to reflect the change to OCP 4 SDN
• [x] Document how OCP 4 SDN meets the requirements of the SDN Compartment zone (same that was written for Aporeto)
• [x] Comms: TIBs, emails and RC to announce the pivot from Aporeto

[mitovskaol]

1. Install OCP 4.5 in KLAB and CLAB on a clean slate (Jan 27), ( @sbarre-esit )
2. Install Aporeto in in KLAB and CLAB (Jan 28) ( @j-pye )
3. Test the Aporeto fix in KLAB ( @sbarre-esit )
4. Finish Vault work in CLAB by Feb 8 (Justin). AQUA work will start in CLAB. ( @j-pye )
5. Feb 4 - Enable OCP 4 SDN in configuration in KLAB ( @jefkel , @sbarre-esit) - unknown steps, reach out to Matt (@mitovskaol to book a meeting). Notes from the meeting with Matt:

Matt confirmed that OCP 4 SDN is already enabled and ready to use. As soon as a Network Policy is created in a namespace, OCP 4 SDN will pick it up and enforce.

OVS Kubernetes - current network implementation in 4.5 (OVN will be available in 4.6) OCP 4 egress firewalls can limit what namespaces can connect to Zone B IP address and block others from accessing that IP address (needs a cluster admin access). Project X can access a specific IP and deny access for all other projects

Create a default deny-all rule to prevent any projects from connecting to Zone B Allow individual projects that need it to specific IP addresses in Zone B

• Book time (a "firewall" conversation) with Dan D, David E, Steen B, Jeff K to discuss if OCP4 SDN covers all potential holes in the firewalls that Aporeto was covering before - any gaps and how to close them. (@mitovskaol to book a meeting)
  4. Test plan (@jefkel , @stewartshea )
     • add Aporeto NSP "allow-all",
     • remove other Aporeto NSPs
     • add OCP Network policies with specific rules
     • test that the allowed connections work and those not allowed are blocked
  5. SDN Migration plan
• Migrate our own apps first and document the steps ( @jleach )
• Replace Aporeto How-To docs with OCP 4 SDN How-To docs (see the previous step) ( @jleach )
  • make a list of all NSPs in Silver, divide into 2 groups (default and custom NSPs) -> develop a NetworkPolicy to replace the default NSPs. ( @j-pye )
• book time with POs/Tech Leads (Jason, PAs, Nick) and help them implement NetworkPolicies (default or help them develop custom) in DEV (Feb 20-March 31). Give them the timeline to apply the NPs in TEST and PROD (Aporeto shut-off date March 31, 2021). ( @mitovskaol )
  6. Release the comms - Aporeto in Zone B alternatives: service mesh with mutual TLS certification and other workarounds. ( @mitovskaol ). @stewartshea provide the bullet points for the key messages.

Note: Communicate that the upgrade to OCP 4.6 will happen after we onboarded off Aporeto (Q1 21/22)

[mitovskaol]

The OCP 4 SDN capability of the OCP 4 Platform has been assessed on its ability to meet the following requirements for systems hosted in the SDN Compartment Zone as per the Technical Note from the Hosting Branch. Aporeto has been found to satisfy these requiements. ● Must leverage the standard Government Identity Provider services OCP 4 Platform uses KeyCloak SSO and GitHub IdP to authenticate users when accessing namespaces. No changes there as we switch to OCP 4 SDN.

● Must leverage Government standard Multi-Factor Authentication for all user-based administrative interfaces No changes here.

● Must have SDN functionality as part of the core solution OCP 4 Platform will switch to using its built-in SDN capabilities when Aporeto is removed.

● SDN will have the capability to apply hierarchical policies that allow for global policies that cannot be contravened _ The ability to support hierarchincal policies allows for developmen of global policies that override or supersede what developers can develop for their namespaces e.g. cluster level policies. OCP 4 SDN does not provide hierarchical policies out of the box, however, global policies can be developed on the cluster level through code. We need to find out if any global policies are implemented now with Aporeto and re-create them for OCP 4 SDN - Day 1 functionality. Any additional policies such as deniying connections into Zone B for all namespaces in OCP 4 except those that need to connect to their databases in Zone b - can be added as new requirements are discovered - Day 2 functionality.

● SDN will have the ability for end clients to self-service their network policies OCP 4 SDN enforcement for a namespace is controlled by the rules in the namespace's NetworkPolicies. Developers can create OCP 4 Network Policies for their namespaces.

● SDN will utilize Data Classification metadata to determine workload placement in appropriate Zone This capabilty can be implemented with OCP 4 SDN within the Openshift environment (a critical requirement) but not outside of the Platform (a non-critical requirement that will be implemented using a different solution/workaround)

● All SDN technologies must use a default zero-trust model While OCP 4 SDN does not implement the Zero Trust Model by default, the Zero Trust Model enforcement is triggered as soon as at least one Network Policy is added to a namespace. A default Network Policy that triggers the Zero Trust Model in a namespace will be added at the time of provisioning of the namespace and can be later overridden by the product team if needed. A good candidate for the default Network Policy would be a rule that will match all pods within the namespace and allow their access to the Artifactory service in OCP 4.

● Connection between any SDN or Classic networks to other BC Government network zone services must use authenticated connections The security for the connections outside of the OCP 4 Platform will be added via a different solution/workaround as this functionality is not available through OCP 4 SDN

[mitovskaol]

TIB has been submitted on Feb 5, 2021 Platform

Managed Container Service

Product

Aporeto Software Defined Network Service

Implementation Date(s)

March 29tth, 2021 Activity The Software Defined Network (SDN) Service on Openshift 4 Platform will be transitioning from Aporeto which is the current SDN solution to the Openshift 4 Built-In SDN in the OCP 4 Silver cluster. The transition will allow the product teams working on the Platform to benefit from using the enterprise-grade solution and the enterprise support that the BC Gov has in place with RedHat. Description All namespaces in the OCP 4 Silver Cluster will need to replace their Aporeto NetworkSecurityPolicy objects (NSPs) with RedHat NetworkPolicy objects. The Aporeto SDN and the OCP 4 Built-In SDN solution will be run in parallel until March 25th to allow sufficient time for the policy replacement in all projects hosted in the OCP 4 Silver cluster. Customer Impact The SDN service migration will occur in stages with respect to the project environments with the first migration being that of the DEV instances followed by TEST/TOOLS and PROD in one week increments. Feb 15th, 2021: OCP 4 SDN Samples, templates and migration support will be made available to product teams to start the policy migration Feb 22nd 2021: Support for Aporeto NSPs is disabled in all DEV namespaces in the Silver cluster Mar 1st 2021: Support for Aporeto NSPs is disabled in all TEST namespaces in the Silver cluster Mar 8th 2021: Support for Aporeto NSPs is disabled in all TOOLS namespaces in the Silver cluster Mar 25th 2021: Support for Aporeto NSPs is disabled in all PROD namespaces in the Silver cluster Aporeto SDN will be disabled in all namespaces of the Silver cluster on March 25, 2021 and after this time, Aporeto NSP will not be in effect anymore. Teams can start replacing their Aporeto NSPs with RedHat NetworkPolicy objects in all their namespaces any time. They will be provided instruction for how to disable the enforcement of Aporeto NSPs in their namespaces by themselves at any time in order to start testing the RedHat Network Policies as soon as possible. Aporeto NSPs must be replaced with RedHat Network Policies before the support for Aporeto NSP is disabled for a specific namespace type (DEV/TEST/TOOLs/PROD) on the Platform level as per the schedule above. The Platform Services Team will guide the product teams through the SDN service migration offering the following support:

SDN Service Migration Kick-off meeting that will provide an overview of the changes and will include a demo of Aporeto NSPs replacement in a namespace. Instructions for the Aporeto Network Security Policy replacement with RedHat Network Policy including a video tutorial will be uploaded to DevHub at http://developer.gov.bc.ca Several live-help sessions will be held between Feb 15th and March 25th to assist the teams The Platform Services Team will provide on-demand one-on-one support to the teams that need it. Testing Procedures The SDN service migration will occur in stages with respect to the project environments starting from non-production environments which will allow the teams to test the network security enforced by the new SDN Solution before enabling it in their production environment. There will be a period of time when both SDN solutions are available which will allow the teams to migrate from Aporeto SDN to OCP 4 Built-In SDN at their convenience.

---

SDN 101: Background information

Software Defined Networking (SDN) for the Data Centres is a technology similar to network firewalls, using software and standard servers to allow Clients to specify network security policies. SDN technology differs from classical firewalls in that it is implemented to allow for scalability using commodity hardware, Infrastructure-as-Code (IaC), full self-service, improved tagging and rulesets, and the ability to implement microsegmentation and zero trust network security. As such. an SDN solution allows teams and administrators to have more flexibility and speed at which they can provision network connectivity for their projects (yes, it’s self-served!). It also provides granular definition and control of communications between application components that would be much harder to implement with traditional firewall technology.

The new Openshift 4 Platform is hosted in the new SDN compartment network zone that was built in the Kamloops Data Center in 2020.

The SDN compartment network zone is intended to host systems that require integrations with cloud services and, as such, has less physical security (e.g., firewalls) to protect the systems; instead, it requires each hosted system to implement its own SDN solution to provide network security.

Thanks!

[mitovskaol]

Updated Timelines: March 1: All dev namespaces are cut-over to KNP

March 8: All tools namespaces are cut-over to KNP

March 15: All test namespaces are cut-over to KNP

March 29: All prod namespaces are cut-over to KNP

March 31 - Aporeto will be disabled but still be installed on the cluster

April 5: Aporeto will be uninstalled from the cluster