oracle-terraform-modules / terraform-oci-oke

The Terraform OKE Module Installer for Oracle Cloud Infrastructure provides a Terraform module that provisions the necessary resources for Oracle Container Engine.
https://oracle-terraform-modules.github.io/terraform-oci-oke/
Universal Permissive License v1.0
153 stars 206 forks source link

Generated NSG Security Rules for FSS do not match the documentation #916

Closed ppiechoc closed 6 months ago

ppiechoc commented 6 months ago

Community Note

Terraform Version and Provider Version

Affected Resource(s)

module.oke.module.network.oci_core_network_security_group_security_rule.oke["Allow TCP egress from workers for NFS portmapper to FSS mounts"] module.oke.module.network.oci_core_network_security_group_security_rule.oke["Allow TCP egress from workers for NFS to FSS mounts"] module.oke.module.network.oci_core_network_security_group_security_rule.oke["Allow TCP ingress to workers for NFS from FSS mounts"] module.oke.module.network.oci_core_network_security_group_security_rule.oke["Allow TCP ingress to workers for NFS portmapper from FSS mounts"] module.oke.module.network.oci_core_network_security_group_security_rule.oke["Allow UDP egress from workers for NFS portmapper to FSS mounts"] module.oke.module.network.oci_core_network_security_group_security_rule.oke["Allow UDP egress from workers for NFS to FSS mounts"] module.oke.module.network.oci_core_network_security_group_security_rule.oke["Allow UDP ingress to workers for NFS portmapper from FSS mounts"]

Terraform Configuration Files

locals {
  ## Networking
  subnets = {
    bastion  = { create = "never" }
    operator = { create = "never" }
    cp       = { create = "never", id = var.cp_subnet_id }
    fss      = { create = "never" }
    int_lb   = { create = "never", id = var.int_lb_subnet_id }
    pub_lb   = { create = "never" }
    workers  = { create = "never", id = var.workers_subnet_id }
    pods     = { create = "never", id = var.pods_subnet_id }
  }
  nsgs = {
    bastion  = { create = "never" }
    operator = { create = "never" }
    cp       = { create = "always" }
    fss      = { create = "always" }
    int_lb   = { create = "always" }
    pub_lb   = { create = "never" }
    workers  = { create = "always" }
    pods     = { create = var.enable_npn ? "always" : "never" }
  }

  ## Workers
  worker_pools = {
    "oke-nodepool-${var.instance}-workers" = {
      size = 3
    }
  }

}

module "oke" {
  source    = "git::https://github.com/oracle-terraform-modules/terraform-oci-oke.git?ref=v5.1.4"
  providers =  { oci.home = oci }

  # Identity
  state_id = var.instance
  cluster_name = "oke-${var.instance}"

  # Placement
  tenancy_id = var.tenancy_ocid
  compartment_id = var.compartment_id
  network_compartment_id = var.network_compartment_id

  # Actions
  create_vcn = false
  create_drg = false
  create_bastion = false
  create_cluster = true
  create_operator = false

  # Networking
  vcn_id = var.vcn_id
  subnets = local.subnets
  nsgs = local.nsgs
  allow_worker_internet_access = false
  allow_pod_internet_access = false

  # Cluster
  kubernetes_version = var.kubernetes_version
  cluster_type = "enhanced"
  control_plane_is_public = false
  preferred_load_balancer = "internal"
  cni_type = var.enable_npn ? "npn" : "flannel"
  control_plane_nsg_ids = var.control_plane_nsg_ids

  # Workers
  worker_pools = local.worker_pools
  ssh_public_key = join("\n", var.ssh_public_keys)
  worker_nsg_ids = var.worker_nsg_ids

}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

Generated NSG rules should match the documentation for FSS.

Actual Behavior

There are two differences:

  1. the FSS NSG opens TCP ingress and egress for port range 30000-32767 instead of documented 2048-2050 - looking in the code it uses OKE nodeport range (locals: node_port_min, node_port_max) in place of FSS port range (locals: fss_nfs_port_min, fss_nfs_port_max). The correct range is used in the Workers NSG on the other side of the connection

  2. the documentation specifies to open the reverse path from source ports 111, 2048, 2049, 2050 to ALL destination client ports (in this case worker's). The FSS and Worker NSGs created by the the terraform instead open path from ALL FSS Mount Target source ports to Worker's destination ports 111, 2048, 2049, 2050

Steps to Reproduce

Create FSS and Workers NSG with FSS enabled.

Important Factoids

References

Documentation: https://docs.oracle.com/en-us/iaas/Content/File/Tasks/securitylistsfilestorage.htm Node Port range used in FSS NSG TF Code (ingress): https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/9d22e9b8104cd4777ee21b864e0fb263020baca1/modules/network/nsg-fss.tf#L30 Node Port range used in FSS NSG TF Code: (egress): https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/9d22e9b8104cd4777ee21b864e0fb263020baca1/modules/network/nsg-fss.tf#L41 Portmapper port used in FSS NSG TF Code Egress as destination instead of source (issue exists in other places as well): https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/9d22e9b8104cd4777ee21b864e0fb263020baca1/modules/network/nsg-fss.tf#L35

ppiechoc commented 5 months ago

Thanks for the fix. I think there are still cases in the Workers NSG accepting ingress from FSS NSG which use destination port instead of source port: https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/b9a02e56d41dca34dce6c8eeb691aada2d42bfe0/modules/network/nsg-workers.tf#L93 https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/b9a02e56d41dca34dce6c8eeb691aada2d42bfe0/modules/network/nsg-workers.tf#L96 https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/b9a02e56d41dca34dce6c8eeb691aada2d42bfe0/modules/network/nsg-workers.tf#L99