search

openshift / api

Canonical location of the OpenShift API definition.
http://www.openshift.org
Apache License 2.0
95 stars 510 forks source link

NE-1774: Promote `IngressControllerLBSubnetsAWS` to GA #1966

Closed gcs278 closed 1 month ago

gcs278 commented 2 months ago

Promotes IngressControllerLBSubnetsAWS to from TechPreview/DevPreview to become available by default (GA).

E2E Test Data:

openshift-ci-robot commented 2 months ago

@gcs278: This pull request references NE-1774 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/api/pull/1966): >Promotes `IngressControllerLBSubnetsAWS` to from TechPreview/DevPreview to become available by default (GA). > >E2E Test Data: TBD Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fapi). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci[bot] commented 2 months ago

Hello @gcs278! Some important instructions when contributing to openshift/api: API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

JoelSpeed commented 2 months ago

Both verify jobs can be overridden once we are content that the testing is in place, the first is failing due to lack of tests, the second because of NoMaps on a CRD rename, which is not something we can fix right now

openshift-ci-robot commented 2 months ago

@gcs278: This pull request references NE-1774 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/api/pull/1966): >Promotes `IngressControllerLBSubnetsAWS` to from TechPreview/DevPreview to become available by default (GA). > >### E2E Test Data: >- This feature gets tested by the [PreSubmit job](https://github.com/openshift/release/blob/d1efafda808dd6d346aa65bd86e01bfda2022485/ci-operator/config/openshift/cluster-ingress-operator/openshift-cluster-ingress-operator-release-4.17.yaml#L154) `e2e-aws-operator-techpreview` in the [cluster-ingress-operator](https://github.com/openshift/cluster-ingress-operator) repo. >- Specifically, [TestAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L37) and [TestUnmanagedAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L172) both test the functionality of the `IngressControllerLBSubnetsAWS` feature gate. >- There isn't an origin E2E test for this feature, so I expect the `verify` job automation to fail. We will require manual [QE Signoff](https://github.com/openshift/enhancements/blob/master/dev-guide/featuresets.md#obtaining-qe-sign-off) >- Example Successful Runs (PreSubmit): > - TBD Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fapi). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
lihongan commented 2 months ago

Checked with pre-merge testing, the featuregate IngressControllerLBSubnetsAWS is enabled by default now, but cannot check the function because the ingresscontroller config spec.endpointPublishingStrategy.loadBalancer.providerParameters.aws.classicLoadBalancer.subnets is ignored silently, Also tested in another TP enabled cluster and it works.

$ oc get featuregates.config.openshift.io cluster -oyaml
<......>
spec: {}
status:
  featureGates:
  - disabled:
<......>
    enabled:
    - name: IngressControllerLBSubnetsAWS
    version: 4.17.0-0.ci.test-2024-08-01-084745-ci-ln-z29h39b-latest

@gcs278 please let me know if this is expected or missing something?

openshift-ci-robot commented 1 month ago

@gcs278: This pull request references NE-1774 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/api/pull/1966): >Promotes `IngressControllerLBSubnetsAWS` to from TechPreview/DevPreview to become available by default (GA). > >### E2E Test Data: >- This feature gets tested by the [PreSubmit job](https://github.com/openshift/release/blob/d1efafda808dd6d346aa65bd86e01bfda2022485/ci-operator/config/openshift/cluster-ingress-operator/openshift-cluster-ingress-operator-release-4.17.yaml#L154) `e2e-aws-operator-techpreview` in the [cluster-ingress-operator](https://github.com/openshift/cluster-ingress-operator) repo. >- Specifically, [TestAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L37) and [TestUnmanagedAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L172) both test the functionality of the `IngressControllerLBSubnetsAWS` feature gate. >- There isn't an origin E2E test for this feature, so I expect the `verify` job automation to fail. We will require manual [QE Signoff](https://github.com/openshift/enhancements/blob/master/dev-guide/featuresets.md#obtaining-qe-sign-off) >- Example Successful PreSubmit Runs (I don't know if this matters if we are getting QE sign-off anyways): > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818807049249624064 > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818845238211383296 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fapi). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
gcs278 commented 1 month ago

@lihongan I believe it's due to the same errors we see in the integration test.

I pushed a fix in this PR. CEL introduced in https://github.com/openshift/api/pull/1826/files#diff-891d2cdcb01603ef2a2c1bfb2f6ef32cf2acf8b1471871a5a6e27308d50050f9R703-R705 should have specified a feature gate. I tried it out in a cluster too. It's a bit odd, but seems like the CEL silently failed, rejected the Subnets in the spec, and created the IC anyways. If you checked your IngressController, the spec Subnets would look empty:

spec:
  endpointPublishingStrategy:
    loadBalancer:
      dnsManagementPolicy: Managed
      providerParameters:
        aws:
          networkLoadBalancer: {}
          type: NLB
        type: AWS
gcs278 commented 1 month ago

Weird, prow didn't run new tests after my push: /test all

openshift-ci-robot commented 1 month ago

@gcs278: This pull request references NE-1774 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/api/pull/1966): >Promotes `IngressControllerLBSubnetsAWS` to from TechPreview/DevPreview to become available by default (GA). > >### E2E Test Data: >- This feature gets tested by the [PreSubmit job](https://github.com/openshift/release/blob/d1efafda808dd6d346aa65bd86e01bfda2022485/ci-operator/config/openshift/cluster-ingress-operator/openshift-cluster-ingress-operator-release-4.17.yaml#L154) `e2e-aws-operator-techpreview` in the [cluster-ingress-operator](https://github.com/openshift/cluster-ingress-operator) repo. >- Specifically, [TestAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L37) and [TestUnmanagedAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L172) both test the functionality of the `IngressControllerLBSubnetsAWS` feature gate. >- There isn't an origin E2E test for this feature, so I expect the `verify` job automation to fail. We will require manual [QE Signoff](https://github.com/openshift/enhancements/blob/master/dev-guide/featuresets.md#obtaining-qe-sign-off) >- Example Successful PreSubmit Runs: > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818807049249624064 > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818845238211383296 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fapi). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci-robot commented 1 month ago

@gcs278: This pull request references NE-1774 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/api/pull/1966): >Promotes `IngressControllerLBSubnetsAWS` to from TechPreview/DevPreview to become available by default (GA). > >### E2E Test Data: >- This feature gets tested by the [PreSubmit job](https://github.com/openshift/release/blob/d1efafda808dd6d346aa65bd86e01bfda2022485/ci-operator/config/openshift/cluster-ingress-operator/openshift-cluster-ingress-operator-release-4.17.yaml#L154) `e2e-aws-operator-techpreview` in the [cluster-ingress-operator](https://github.com/openshift/cluster-ingress-operator) repo. >- Specifically, [TestAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L37) and [TestUnmanagedAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L172) both test the functionality of the `IngressControllerLBSubnetsAWS` feature gate. >- There isn't an origin E2E test for this feature, so I expect the `verify` job automation to fail. We will require manual [QE Signoff](https://github.com/openshift/enhancements/blob/master/dev-guide/featuresets.md#obtaining-qe-sign-off) >- Example Successful PreSubmit Runs: > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818807049249624064 > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818845238211383296 > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1115/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1819004886000340992 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fapi). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
gcs278 commented 1 month ago

@lihongan I believe it's due to the same errors we see in the integration test.

I pushed a fix in this PR. CEL introduced in https://github.com/openshift/api/pull/1826/files#diff-891d2cdcb01603ef2a2c1bfb2f6ef32cf2acf8b1471871a5a6e27308d50050f9R703-R705 should have specified a feature gate. I tried it out in a cluster too. It's a bit odd, but seems like the CEL silently failed, rejected the Subnets in the spec, and created the IC anyways. If you checked your IngressController, the spec Subnets would look empty:

@lihongan Sorry, I think I confused two problems. The issue you see isn't related to the CEL Validation. I realized we do need to re-vendor this change in the IngressOperator to get the updated IngressController CRD.

So when you are doing testing, you must also update apply the IngressController CRD. I think that's why it's not working for you. You will have to apply the one from this PR: operator/v1/zz_generated.crd-manifests/0000_50_ingress_00_ingresscontrollers-Default.crd.yaml

The integration failure exists due to bug where the CEL is restricted to a feature gate, introduced in https://github.com/openshift/api/pull/1826. That's now fixed in this PR.

gcs278 commented 1 month ago

/retest

lihongan commented 1 month ago

/label qe-approved

Thanks @gcs278 , after doing oc apply -f ./0000_50_ingress_00_ingresscontrollers-Default.crd.yaml then the .subnets setting in ingresscontroller works, the annotation is added to LB service. 

$ oc -n openshift-ingress get svc router-default -oyaml | grep subnet
    service.beta.kubernetes.io/aws-load-balancer-subnets: ci-ln-9vgmglb-76ef8-sl8sc-subnet-public-us-west-1a

$ oc get co ingress console
NAME      VERSION                                                   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
ingress   4.17.0-0.ci.test-2024-08-02-010240-ci-ln-9vgmglb-latest   True        False         False      21m     
console   4.17.0-0.ci.test-2024-08-02-010240-ci-ln-9vgmglb-latest   True        False         False      10m
openshift-ci-robot commented 1 month ago

@gcs278: This pull request references NE-1774 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/api/pull/1966): >Promotes `IngressControllerLBSubnetsAWS` to from TechPreview/DevPreview to become available by default (GA). > >### E2E Test Data: >- This feature gets tested by the [PreSubmit job](https://github.com/openshift/release/blob/d1efafda808dd6d346aa65bd86e01bfda2022485/ci-operator/config/openshift/cluster-ingress-operator/openshift-cluster-ingress-operator-release-4.17.yaml#L154) `e2e-aws-operator-techpreview` in the [cluster-ingress-operator](https://github.com/openshift/cluster-ingress-operator) repo. >- Specifically, [TestAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L37) and [TestUnmanagedAWSLBSubnets](https://github.com/openshift/cluster-ingress-operator/blob/64eafaef2a4e02056bb1976ce8c8ad25a1509d23/test/e2e/lb_subnets_test.go#L172) both test the functionality of the `IngressControllerLBSubnetsAWS` feature gate. >- There isn't an origin E2E test for this feature, so I expect the `verify` job automation to fail. We will require manual [QE Signoff](https://github.com/openshift/enhancements/blob/master/dev-guide/featuresets.md#obtaining-qe-sign-off) >- Example Successful PreSubmit Runs: > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818807049249624064 > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1114/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1818845238211383296 > - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1115/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator-techpreview/1819004886000340992 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fapi). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
miheer commented 1 month ago

/retest

miheer commented 1 month ago

@JoelSpeed it looks like you might need to over-ride this test ?

JoelSpeed commented 1 month ago

/lgtm /override ci/prow/verify

openshift-ci[bot] commented 1 month ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gcs278, JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/api/blob/master/OWNERS)~~ [JoelSpeed] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
openshift-ci[bot] commented 1 month ago

@JoelSpeed: Overrode contexts on behalf of JoelSpeed: ci/prow/verify

In response to [this](https://github.com/openshift/api/pull/1966#issuecomment-2265072700): >/lgtm >/override ci/prow/verify Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
openshift-ci[bot] commented 1 month ago

@gcs278: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
openshift-bot commented 1 month ago

[ART PR BUILD NOTIFIER]

Distgit: ose-cluster-config-api This PR has been included in build ose-cluster-config-api-container-v4.18.0-202408021814.p0.g8b2b377.assembly.stream.el9. All builds following this will include this PR.