Openshift install with VMware NSX-T

[kratos81]

Version details

Openshift version - 3.11 NSX-T version - 2.4 Vsphere version - 6.5

Issue

After installing Openshift with NSX-T , applications deployed in the cluster are not able to connect outside NSX-T

Description

We installed openshift , integrated with the T0 router , initially the T0 router was active-active , so we changed it to active standby. But whenever we create a namespace in openshift, the associated T1 router , switches and ports are created but applications deployed in the namespaces cannot access anything outside NSX-T We have associated the SNAT pool and passed in the T0 details to the T1 router We find that when when we explicitly associate the T1 with the edge nodes , we have egress . But the association should happen implicitly because they are being created by the NCP application.

There is no explicit association with the edge cluster from the variables we pass in to NCP. Is this association hinged on the T0 router's association ?

Is there anything that you can point to that we may be missing ? we have followed the documentation.

Kindly let me know if what additional information you need and I will be happy to provide it.

[dav1x]

@kratos81 Do the OCP nodes have 2 interfaces?

Can you post your inventory here?

[kratos81]

Hi @dav1x

This is our inventory file

[OSEv3:children]
ansible
masters
infras
apps
etcd
nodes
lb

[OSEv3:vars]
## Ansible Vars
become=yes
ansible_become=yes
ansible_ssh_user=root

## Additional General
osm_use_cockpit=true
oreg_url=xxxxxx
default_subdomain=xxxx

## Openshift General
openshift_deployment_type=openshift-enterprise
openshift_disable_check=memory_availability,disk_availability,docker_storage
openshift_docker_insecure_registries=xxxx
openshift_node_local_quota_per_fsgroup=512Mi

## OpenShift Master General
openshift_master_cluster_method=native
openshift_master_cluster_public_hostname=xxxxxx
openshift_master_default_subdomain=xxxxx

## Image Stream Mods
openshift_examples_modify_imagestreams=true
registry_host=xxxx

## Identity
<redacted> 

## Network Settings
openshift_use_nsx=true
os_sdn_network_plugin_name=cni
openshift_use_openshift_sdn=false
openshift_node_sdn_mtu=1500
os_firewall_use_firewalld=true

## vSphere Cloud provider
openshift_cloudprovider_kind=vsphere
openshift_cloudprovider_vsphere_username="foo"
openshift_cloudprovider_vsphere_password="bar"
openshift_cloudprovider_vsphere_host="xxxx"
openshift_cloudprovider_vsphere_datacenter="xxxx"
openshift_cloudprovider_vsphere_cluster="xxxx"
#openshift_cloudprovider_vsphere_resource_pool=ResourcePool
openshift_cloudprovider_vsphere_datastore="xxxxx"
openshift_cloudprovider_vsphere_folder="Managment VMs"

# NSX specific configuration
#nsx_use_loadbalancer=false
nsx_insecure_ssl=true
nsx_api_managers='xxxxx'
nsx_api_user='xxxx'
nsx_api_password='xxxxx'
nsx_openshift_cluster_name='xxxxx'
nsx_tier0_router='ub-ocp-t0-1'
nsx_overlay_transport_zone='ub01-ocp-c01-w-overlay'
nsx_container_ip_block='ocp-cluster-block'
nsx_no_snat_ip_block='OCP_DEV_NO_NAT'
nsx_external_ip_pool='OCP_DEV_NAT'
nsx_ovs_uplink_port='eth1'
#nsx_edge_cluster_name='Edge Cluster Transport'
nsx_ncp_image=xxxxx
nsx_ovs_url='xxxxx'
nsx_cni_url='xxxx'
nsx_kmod_ovs_url='xxxx'

## Hosted ETCD
openshift_hosted_etcd_storage_kind=dynamic
openshift_hosted_etcd_storage_volume_name=etcd-vol
openshift_hosted_etcd_storage_access_modes=["ReadWriteOnce"]
openshift_hosted_etcd_storage_volume_size=1G
openshift_hosted_etcd_storage_labels={'storage': 'etcd'}

## Identity Providers
#openshift_master_ldap_ca_file=/home/cloud-user/mycert.crt

# OpenShift Registry
openshift_hosted_registry_storage_kind=vsphere
openshift_hosted_registry_storage_access_modes=['ReadWriteOnce']
openshift_hosted_registry_storage_annotations=['volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume']
openshift_hosted_registry_replicas=1

## OpenShift Routers
openshift_hosted_router_replicas=3

## Metrics
openshift_cluster_monitoring_operator_install=false
#openshift_metrics_install_metrics=true
#openshift_metrics_storage_kind=dynamic
#openshift_metrics_storage_volume_size=25Gi

## Logging
#openshift_logging_install_logging=true
#openshift_logging_es_pvc_dynamic=true
#openshift_logging_es_pvc_size=30Gi
#openshift_logging_elasticsearch_storage_type=pvc
#openshift_logging_es_cluster_size=1
#openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"}
#openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"}
#openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"}
#openshift_logging_fluentd_nodeselector={"node-role.kubernetes.io/infra": "true"}
#openshift_logging_storage_kind=dynamic

## Hosts Section ##
[ansible]
localhost

[masters]
<master nodes> 

[infras]
<infra nodes>

[apps]
<app nodes> 
[etcd]
<references masters> 
[lb]

[nodes]
<all nodes>

[kratos81]

Thanks , I have provided a hosts file (redacted )

So for background , we tagged one of the NICs , based on the the documentation , So the entire worklflow works , only that when we create a project , we found out that the T1 routers are not associated with the edge cluster. Previous versions of NSX passed it in as a variable as "nsx_edge_cluster_name='" but this is not available in 2.4 as an option.

Do we need to explicitly associate the created T1 router to the edge cluster ? can we pass it in as a variable ? how does NCP associate the t1 router it creates with the edge cluster ?

Let me know if you need more clarification on this or more information

[dav1x]

Let me engage an NSX wizard @kratos81 .

[kratos81]

Ok thanks !

[yasensim]

@kratos81 this issue was just brought to my attention! I believe we have fixed it yesterday on webex, right? We had to re-configure T0 and Segments thru the advanced UI in NSX as simplified UI will be supported with next release.

If yes, we can close this issue.

[kratos81]

Yes , it can be closed

[dav1x]

Awesome! Thanks all