search

argoproj-labs / rollouts-plugin-trafficrouter-gatewayapi

The Argo Rollouts plugin implementing the Kubernetes Gateway API specification for using different traffic providers in progressive delivery scenarios
https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/
Apache License 2.0
100 stars 19 forks source link

Error occurs when deploying canary in istio environment using gateway. #89

Open oliverpark999 opened 1 month ago

oliverpark999 commented 1 month ago

Checklist:

Describe the bug

To Reproduce

Expected behavior

Read Document https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/quick-start/

Environment

Contents

NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ◌ Progressing 5m46s └──# revision:1 └──⧉ rollouts-demo-abc ReplicaSet • ScaledDown 5m46s canary


<!-- A clear and concise description of what you expected to happen. -->

**Screenshots**

<!-- If applicable, add screenshots to help explain your problem. -->

**Version**
plugin version => 0.4.0
argo rollouts controller version => 1.7.2

<!-- What version of argo rollouts controller are you running? -->

**Logs**

Paste the logs from the rollout controller

Logs for the entire controller:

kubectl logs -n argo-rollouts deployment/argo-rollouts

│ {"level":"info","msg":"delaying service switch from to abc: ReplicaSet not fully available","namespace":"foobar","rollout":"rollout-canary-demo","service":"rollout-canary-demo-canary","time":"2024-10-29T00:20:40Z"} │ │ {"generation":1,"level":"error","msg":"roCtx.reconcile err failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","resourceVersio │ │ {"generation":1,"level":"info","msg":"Reconciliation completed","namespace":"foobar","resourceVersion":"44704875","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:40Z","time_ms":1.3126} │ │ {"level":"error","msg":"rollout syncHandler error: failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","rollout":"rollout-cana │ │ {"level":"info","msg":"rollout syncHandler queue retries: 33 : key \"foobar/rollout-canary-demo\"","namespace":"foobar","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:40Z"} │ │ {"error":null,"level":"error","msg":"failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","time":"2024-10-29T00:20:40Z"} │ │ {"generation":1,"level":"info","msg":"Started syncing rollout","namespace":"foobar","resourceVersion":"44704875","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:50Z"} │ │ {"level":"info","msg":"delaying service switch from to abc: ReplicaSet not fully available","namespace":"foobar","rollout":"rollout-canary-demo","service":"rollout-canary-demo-canary","time":"2024-10-29T00:20:50Z"} │ │ {"generation":1,"level":"error","msg":"roCtx.reconcile err failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","resourceVersio │ │ {"generation":1,"level":"info","msg":"Reconciliation completed","namespace":"foobar","resourceVersion":"44704875","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:50Z","time_ms":1.721427} │ │ {"level":"error","msg":"rollout syncHandler error: failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","namespace":"foobar","rollout":"rollout-cana │ │ {"level":"info","msg":"rollout syncHandler queue retries: 34 : key \"foobar/rollout-canary-demo\"","namespace":"foobar","rollout":"rollout-canary-demo","time":"2024-10-29T00:20:50Z"} │ │ {"error":null,"level":"error","msg":"failed to get traffic router plugin argoproj-labs/gatewayAPI: unable to start plugin system: unable to get plugin client (argoproj-labs/gatewayAPI) for ping: exec: Stdout already set","time":"2024-10-29T00:20:50Z"} │ │

Logs for a specific rollout:

kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME



---
<!-- Issue Author: Don't delete this message to encourage other users to support your issue! -->
**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
oliverpark999 commented 1 month ago

The problem was the line below. When I removed trafficRouting, the pod was deployed. Why is that?

trafficRouting:
  plugins:
    argoproj-labs/gatewayAPI:
      httpRoute: argo-rollouts-http-route # our created httproute
      namespace: foobar
kostis-codefresh commented 1 month ago

Hello

The part that you mention is the part that loads the plugin. And the error you posted shows that the plugin is not loaded.

Possible reasons:

  1. The clusters doesn't have access to the URL you have defined for the plugin
  2. You are trying to load the plugin of a different architecture.

Have you also followed https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/installation/ ?

Can you post the argo-rollouts-config please and also mention what type of machine is running Argo Rollouts?

Also the plugin you are trying to load is the latest one for Kubernetes API gateway 1.0+ Are you sure the version of Istio you are running also supports this version?

oliverpark999 commented 1 month ago

Hi!

When I configured the plugin with v0.3.0, the plugin was loaded normally and Replicas also worked.

But here's my question.

My k8s Gateway API is version 1.0.

So all resources are already using gateway.networking.k8s.io/v1.

However, in version v0.4.0, an error message was printed and it did not work properly,

but in version v0.3.0, it worked normally.

Do you know anything about this?

oliverpark999 commented 1 month ago

I read the release notes and it seems that k8s Gateway API v1.0 is supported starting from v0.3.0. Then, is this a bug in v0.4.0?

Philipp-Plotnikov commented 1 month ago

Can you say machine architecture and the plugin that you are trying to download, please ?

oliverpark999 commented 4 weeks ago

Yes, The hardware architecture of my Kubernetes Node server is x86, and the plugin that caused the error message is gatewayapi-plugin-linux-amd64(v0.4.0).

kostis-codefresh commented 4 weeks ago

My k8s Gateway API is version 1.0.

What is your Istio version?

When I configured the plugin with v0.3.0, the plugin was loaded normally and Replicas also worked.

But did you enable again the plugin in the Rollout? Because previously you said you removed that part

trafficRouting:
  plugins:
    argoproj-labs/gatewayAPI:
      httpRoute: argo-rollouts-http-route # our created httproute
      namespace: foobar

To the best of my knowledge there are no major API changes between plugin 0.3.0 and 0.4.0

oliverpark999 commented 4 weeks ago

Istio is version 1.22.3. As you said, I removed trafficRouting and checked that it works. However, I tried several things without stopping the game. That was v0.3.0, and I added trafficRouting again. I also restarted Argo-Rollout Deployment. The result was normal operation as below.

Normal  RolloutAddedToInformer  10s   rollouts-controller  Rollout resource added to informer: foo/bar
Normal  RolloutNotCompleted     10s   rollouts-controller  Rollout not completed, started update to revision 1
Normal  RolloutUpdated          10s   rollouts-controller  Rollout updated to revision 1
Normal  NewReplicaSetCreated    10s   rollouts-controller  Created ReplicaSet bar-xxx (revision 1)
Normal  TrafficWeightUpdated    9s    rollouts-controller  Traffic weight updated to 0
Normal  ScalingReplicaSet       9s    rollouts-controller  Scaled up ReplicaSet bar-xxx (revision 1) from 0 to 1
Normal  RolloutCompleted        9s    rollouts-controller  Rollout completed update to revision 1 : Initial deploy
Normal  TrafficWeightUpdated    9s    rollouts-controller  Traffic weight updated
Normal  SwitchService           1s    rollouts-controller  Switched selector for service 'bar' from '' to 'xx'
Normal  SwitchService           1s    rollouts-controller  Switched selector for service 'bar-canary' from '' to 'xx'
kostis-codefresh commented 2 weeks ago

Are you using argocd as well? Or just argo rollouts on its own?