Kubernetes Mock Server not dealing correctly with LabelSelector when using matchLabels and matchExpressions

[agustin-munoz]

Describe the bug

When doing unit tests using the Kubernetes/Openshift server use of matchExpressions is not working properly

Expected behavior

Get the mathing namespaces

Actual behavior

Seems like it is just ignoring the matchExpressions

How to Reproduce?

Use this test, there will be 2 fails because MatchExpressions are not handled well by the mock server, in the code is commented right after the failing test the query sent to api. Using this same query against a real kubernetes or openshift API returns the right namespaces.

import static org.junit.jupiter.api.Assertions.assertEquals;

import java.util.List;

import org.junit.jupiter.api.Test;

import io.fabric8.kubernetes.api.model.LabelSelector;
import io.fabric8.kubernetes.api.model.LabelSelectorBuilder;
import io.fabric8.kubernetes.api.model.NamespaceBuilder;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.kubernetes.client.WithKubernetesTestServer;
import jakarta.inject.Inject;

@WithKubernetesTestServer
@QuarkusTest
public class SimpleTest {

    @Inject
    KubernetesClient oc;

    @Test
    void selectorWorksWell() {
        var organization1 = "org1";
        var organization2 = "org2";
        var organizationLabel = "example.com/organization";
        var namespace1 = new NamespaceBuilder().withNewMetadata().withName("namespace1-" + organization1)
                .addToLabels(organizationLabel, organization1)
                .addToLabels("kubernetes.io/metadata.name", "namespace1-" + organization1).endMetadata()
                .build();
        var namespace2 = new NamespaceBuilder().withNewMetadata().withName("namespace2-" + organization1)
                .addToLabels(organizationLabel, organization1)
                .addToLabels("kubernetes.io/metadata.name", "namespace2-" + organization1).endMetadata()
                .build();
        var namespace3 = new NamespaceBuilder().withNewMetadata().withName("namespace3-" + organization1)
                .addToLabels(organizationLabel, organization1)
                .addToLabels("kubernetes.io/metadata.name", "namespace3-" + organization1).endMetadata()
                .build();
        var namespace4 = new NamespaceBuilder().withNewMetadata().withName("namespace4-" + organization1)
                .addToLabels(organizationLabel, organization2)
                .addToLabels("kubernetes.io/metadata.name", "namespace4-" + organization1).endMetadata()
                .build();
        oc.namespaces().resource(namespace1).create();
        oc.namespaces().resource(namespace2).create();
        oc.namespaces().resource(namespace3).create();
        oc.namespaces().resource(namespace4).create();
        LabelSelector selector = new LabelSelectorBuilder().addNewMatchExpression()
                .withKey("kubernetes.io/metadata.name")
                .withOperator("In")
                .withValues(List.of("namespace1-" + organization1, "namespace2-" + organization1))
                .endMatchExpression()
                .addToMatchLabels(organizationLabel, organization1)
                .build();
        var namespacesWithMatchLabelsAndMatchExpressions = oc.namespaces().withLabelSelector(selector).list().getItems();
        selector = new LabelSelectorBuilder().addToMatchLabels(organizationLabel, organization1).build();
        var namespacesWithMatchLabelsOrg1 = oc.namespaces().withLabelSelector(selector).list().getItems();
        selector = new LabelSelectorBuilder().addToMatchLabels(organizationLabel, organization2).build();
        var namespacesWithMatchLabelsOrg2 = oc.namespaces().withLabelSelector(selector).list().getItems();
        selector = new LabelSelectorBuilder().addNewMatchExpression()
                .withKey(organizationLabel)
                .withOperator("NotIn")
                .withValues(organization1)
                .endMatchExpression()
                .build();
        var namespacesUsingOnlyMatchExpression = oc.namespaces().withLabelSelector(selector).list().getItems();
        assertEquals(3, namespacesWithMatchLabelsOrg1.size());
        assertEquals(1, namespacesWithMatchLabelsOrg2.size());
        assertEquals(1, namespacesUsingOnlyMatchExpression.size()); //-> actual is 4
        // GET /api/v1/namespaces?labelSelector=example.com%2Forganization%20notin%20%28org1%29
        assertEquals(2, namespacesWithMatchLabelsAndMatchExpressions.size()); //-> actual is 3
        // GET /api/v1/namespaces?labelSelector=example.com%2Forganization%3Dorg1%2Ckubernetes.io%2Fmetadata.name%20in%20%28namespace1-org1%2Cnamespace2-org1%29
    }

}

Output of uname -a or ver

Linux workhorse 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

java version "17.0.5" 2022-10-18 LTS Java(TM) SE Runtime Environment (build 17.0.5+9-LTS-191) Java HotSpot(TM) 64-Bit Server VM (build 17.0.5+9-LTS-191, mixed mode, sharing)

Quarkus version or git rev

3.8.3

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)

Additional information

I've tried also with quarkus 3.2.10.Final, and found same issue.

[quarkus-bot[bot]]

/cc @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

[geoand]

cc @manusa

[Dairdevil]

I've had a browse around this issue, unless I've got the wrong end of the stick it is coming from the fabric8 library and this particular feature has been requested but not yet implemented: https://github.com/fabric8io/kubernetes-client/issues/4113

In that thread its suggested that this alternative would cover the use case: Kube API test module @agustin-munoz would that help you get your desired tests working?

[manusa]

We've discussed this multiple times on the Fabric8 side. There's a limit to the number of features we can add to the Kubernetes Moc kServer until we've completely re-implemented the Kuberentes API and its workers. The recommendation for more advanced use caes is to either use Dev Services with a Kind cluster or similar, or the newly added Kube API test module as @Dairdevil suggests.

Unfortunately there's been an issue with the release pipelines and the Kube API server hasn't been released yet. However, we're working on its release (https://github.com/fabric8io/kubernetes-client/issues/5898) ASAP.

[agustin-munoz]

Thanks @Dairdevil and @manusa, that for sure will help.