AS3 not posted to specific tenant API endpoint

[m-kratochvil]

Setup Details

CIS Version : 2.15.0 Build: f5networks/k8s-bigip-ctlr:2:15:0 BIGIP Version: Big IP 15.1.3 AS3 Version: 3.36.1 Agent Mode: AS3 Orchestration: K8S Orchestration Version: 1.28.0 Pool Mode: Nodeport

Description

We are using CIS in Kubernetes with ingress. It is working fine with single tenant. CIS is deployed with bigip_partition: qa-de-1-metal3 set in the values.yaml and AS3 declarations are correctly posted to http://localhost:8100/mgmt/shared/appsvcs/declare/qa-de-1-metal3 tenant endpoint.

When we set annotation virtual-server.f5.com/partition: "testpartition" in the ingress, CIS deletes the default CIS partition qa-de-1-metal3 and configures the new partition testpartition, posting it to the general AS3 endpoint http://localhost:8100/mgmt/shared/appsvcs/declare/ instead of the tenant-specific endpoint.

Steps To Reproduce

1) Deploy CIS with bigip_partition: some_partition 2) Create Ingress without the partition annotation, observe AS3 is posted to tenant-specific endpoint 3) Add partition annotation to ingress, e.g. virtual-server.f5.com/partition: "testpartition", observe AS3 is posted to general declare/ endpoint.

Expected Result

AS3 declaration is posted to tenant-specific API endpoint

Actual Result

AS3 declaration is posted to general declare/ API endpoint

Diagnostic Information

Big-IP restjavad-audit log when ingress does NOT have the partition annotation set:

[I][24314][11 Jan 2024 16:49:51 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/shared/appsvcs/declare/qa-de-1-metal3","status":200,"from":"10.4.8.2"}

Big-IP restjavad-audit log when ingress has the partition annotation set:

[I][25694][12 Jan 2024 12:42:34 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/shared/appsvcs/declare/","status":200,"from":"10.4.8.2"}

Questions

1. With ingress use-case for CIS, do we / should we set bigip_partition: some_partition in the CIS deployment, or should we rather use the ingress annotation for partition
2. Are there any known use-cases for using the partition annotation in ingress? What is the goal?

[vklohiya]

@m-kratochvil , you can decide in which bigip partition you would put like to the virtual server created by CIS when using ingress. If you choose to share the same virtual ip address and partition for all the ingress you can define that in the CIS deployment and no need to define it in the annotations. However if your use-case requires to group the ingress to a different partition and ip addresses in that case you can define the annotations for those ingresses and remaining ingresses will use the default partition and virtual-server address.

Note: Partition annotation change for ingress intermittently causes AS3 422 error. When you receive an error, delete the old ingress and recreate the ingress with a new partition.

[m-kratochvil]

@vklohiya thanks for the review and explanation. However, in reality this doesn't work as expected, at least I think.

Let's say I want to have two ingresses, each in separate partition on the F5. I configure the first ingress "ingress-1" with partition annotation specifying "bigip-tenant-1" CIS configures corresponding virtual server, policy, pool, pool members and nodes in F5 partition "bigip-tenant-1" and all is fine. Then I configure second ingress "ingress-2" with partition annotation specifying "bigip-tenant-2" CIS tries to configure the objects on the F5 but the declaration fails with IP conflict error, because the nodes already exist in partition "bigip-tenant-1"

2024/01/12 14:26:31 [ERROR] [AS3] Raw response from Big-IP: map[code:422 declarationFullId: 
errors:[/bigip-tenant-2/Shared/ingress_f5_bigip_ctlr_michalcis_svc_2/members: pool member 
/bigip-tenant-2/Shared/ingress_f5_bigip_ctlr_michalcis_svc_2/members/0 static address 10.246.84.246 conflicts
 with bigip node /bigip-tenant-1/10.246.84.246] message:declaration is invalid] 

That leaves me with question what is the use-case for using the partition annotation on ingress in a single cluster, if the nodes can't be shared on the Big-IP..?

[vklohiya]

@m-kratochvil , Please add the --share-nodes=true in CIS deployment. It should resolve the issue. if you still see the issue just delete the ingresses and recreate ingress.

[m-kratochvil]

@vklohiya thanks for the suggestion, I must have overlooked the setting. I'll test it out ASAP and will confirm here, give me a bit more time before closing the issue, please.

[m-kratochvil]

I finally tested the proposed solution and I confirm it works as expected, the nodes are installed into "Common" partition. Thank you again for the assistance, much appreciated. The issue can be closed.

[vklohiya]

Thanks @m-kratochvil . Closing the issue now.