The application deployed is not reachable to Meshery

[gyohuangxin]

Description

The application deployed by SMP github action is not reachable to Meshery.

Expected Behavior

Root cause:

1. The default endpoint_url is Istio ingress url + ingress port, which returns 404 status code to Meshery.
2. minikube tunnel is easy to be killed.

Screenshots/Logs

Environment:

• Meshery Version:
• Kubernetes Version:
• Host OS:
• Browser:

[leecalcote]

@piyushsingariya @s1ntaxe770r fyi

[gyohuangxin]

@leecalcote Please assign this to me, I'll raise a PR to fix it, thanks.

[leecalcote]

@gyohuangxin excellent. 😄

[pottekkat]

@gyohuangxin This is also an issue for deploying Linkerd and an application on top of it.

Another area we can expand on is to have the sample application configurable as different sample applications can act as representative workloads. For a single test comparison, we can run tests on different service meshes with the same sample application.

[gyohuangxin]

@navendu-pottekkat Thanks, I will have a look on this issue. And I aggress with you about extending a configuration of sample applications. We can create an issue on this, which repo is suitable, SMP or here?

[gyohuangxin]

@navendu-pottekkat It failed when executing command meshery perf apply --file load-test.yaml ...

Configuration file: load-test.yaml
Endpoint URL: http://localhost:8080
Service Mesh: LINKERD
Test Name: linkerd-fortio-load-test.yaml
Load Generator: fortio
Running test with test configuration file load-test.yaml
Error: please enter a profile-name.
See https://docs.meshery.io/reference/mesheryctl/perf/apply for usage details

I found that after your PR https://github.com/meshery/meshery/pull/4710 merged, profile-name is a must though --file load-test.yaml is provided. The same error occurs when running benchmarking on Istio.

[pottekkat]

@gyohuangxin Yes, we need to pass in a profile name when we are using a test configuration file as well.

The test configuration currently does not capture the profile name. Prior to https://github.com/meshery/meshery/pull/4710, the logic was to use the test name as the profile name. This could work well in most use cases but for the action, it would be difficult to automatically collect and organise the test results.

To use this change in mesheryctl, the meshery-smp-action was updated as well to fix this error of not having a profile-name.

https://github.com/layer5io/meshery-smp-action/blob/2f296c567ae53696d8c578bdd9bf517a7a916a83/mesheryctl.sh#L67

The Istio benchmark tests does work with Fortio load generator but there seems to be some issue in upstream for wrk2 which is throwing errors (https://github.com/meshery/meshery/issues/4199).

The Linkerd tests however does not work. And we need to look into debugging this and fixing this as well as another failure when trying to deploy OSM.

Will open up a new issue and link it here for adding the sample application to the job matrix.

[gyohuangxin]

@navendu-pottekkat Thanks, got it. I'll raise a PR to fix Linkerd issue and add some check for profile name.

[pottekkat]

@gyohuangxin Great! Thank you 🚀

[gyohuangxin]

In the process of deploying application on Linkerd, there are some errors when excuting kubectl get -n emojivoto deploy -o yaml | linkerd inject - | kubectl apply -f -

Error from server (Conflict): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"deployment.kubernetes.io/revision\":\"1\"},\"generation\":1,\"labels\":{\"app.kubernetes.io/name\":\"emoji\",\"app.kubernetes.io/part-of\":\"emojivoto\",\"app.kubernetes.io/version\":\"v11\"},\"name\":\"emoji\",\"namespace\":\"emojivoto\",\"resourceVersion\":\"1176\",\"uid\":\"2b4d353c-1f3a-4280-8f48-0598779543a5\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"app\":\"emoji-svc\",\"version\":\"v11\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"annotations\":{\"linkerd.io/inject\":\"enabled\"},\"labels\":{\"app\":\"emoji-svc\",\"version\":\"v11\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"GRPC_PORT\",\"value\":\"8080\"},{\"name\":\"PROM_PORT\",\"value\":\"8801\"}],\"image\":\"docker.l5d.io/buoyantio/emojivoto-emoji-svc:v11\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"emoji-svc\",\"ports\":[{\"containerPort\":8080,\"name\":\"grpc\",\"protocol\":\"TCP\"},{\"containerPort\":8801,\"name\":\"prom\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"100m\"}},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\"}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"serviceAccount\":\"emoji\",\"serviceAccountName\":\"emoji\",\"terminationGracePeriodSeconds\":30}}},\"status\":{\"conditions\":[{\"message\":\"Created new replica set \\\"emoji-66ccdb4d86\\\"\",\"reason\":\"NewReplicaSetCreated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"message\":\"Deployment does not have minimum availability.\",\"reason\":\"MinimumReplicasUnavailable\",\"status\":\"False\",\"type\":\"Available\"}],\"observedGeneration\":1,\"unavailableReplicas\":1}}\n"},"resourceVersion":"1176"},"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"enabled"}}}},"status":{"$setElementOrder/conditions":[{"type":"Progressing"},{"type":"Available"}],"conditions":[{"message":"Created new replica set \"emoji-66ccdb4d86\"","reason":"NewReplicaSetCreated","type":"Progressing"}]}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "emoji", Namespace: "emojivoto"
for: "STDIN": Operation cannot be fulfilled on deployments.apps "emoji": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"deployment.kubernetes.io/revision\":\"1\"},\"generation\":1,\"labels\":{\"app.kubernetes.io/name\":\"vote-bot\",\"app.kubernetes.io/part-of\":\"emojivoto\",\"app.kubernetes.io/version\":\"v11\"},\"name\":\"vote-bot\",\"namespace\":\"emojivoto\",\"resourceVersion\":\"1175\",\"uid\":\"512b9e16-70b0-44ed-a91d-6466f69cfd78\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"app\":\"vote-bot\",\"version\":\"v11\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"annotations\":{\"linkerd.io/inject\":\"enabled\"},\"labels\":{\"app\":\"vote-bot\",\"version\":\"v11\"}},\"spec\":{\"containers\":[{\"command\":[\"emojivoto-vote-bot\"],\"env\":[{\"name\":\"WEB_HOST\",\"value\":\"web-svc.emojivoto:80\"}],\"image\":\"docker.l5d.io/buoyantio/emojivoto-web:v11\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"vote-bot\",\"resources\":{\"requests\":{\"cpu\":\"10m\"}},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\"}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"terminationGracePeriodSeconds\":30}}},\"status\":{\"conditions\":[{\"message\":\"Created new replica set \\\"vote-bot-69754c864f\\\"\",\"reason\":\"NewReplicaSetCreated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"message\":\"Deployment does not have minimum availability.\",\"reason\":\"MinimumReplicasUnavailable\",\"status\":\"False\",\"type\":\"Available\"}],\"observedGeneration\":1,\"unavailableReplicas\":1}}\n"},"resourceVersion":"1175"},"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"enabled"}}}},"status":{"$setElementOrder/conditions":[{"type":"Progressing"},{"type":"Available"}],"conditions":[{"message":"Created new replica set \"vote-bot-69754c864f\"","reason":"NewReplicaSetCreated","type":"Progressing"}]}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "vote-bot", Namespace: "emojivoto"
for: "STDIN": Operation cannot be fulfilled on deployments.apps "vote-bot": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"deployment.kubernetes.io/revision\":\"1\"},\"generation\":1,\"labels\":{\"app.kubernetes.io/name\":\"voting\",\"app.kubernetes.io/part-of\":\"emojivoto\",\"app.kubernetes.io/version\":\"v11\"},\"name\":\"voting\",\"namespace\":\"emojivoto\",\"resourceVersion\":\"1183\",\"uid\":\"65719b55-f960-43b9-8a02-b0176540071c\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"app\":\"voting-svc\",\"version\":\"v11\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"annotations\":{\"linkerd.io/inject\":\"enabled\"},\"labels\":{\"app\":\"voting-svc\",\"version\":\"v11\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"GRPC_PORT\",\"value\":\"8080\"},{\"name\":\"PROM_PORT\",\"value\":\"8801\"}],\"image\":\"docker.l5d.io/buoyantio/emojivoto-voting-svc:v11\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"voting-svc\",\"ports\":[{\"containerPort\":8080,\"name\":\"grpc\",\"protocol\":\"TCP\"},{\"containerPort\":8801,\"name\":\"prom\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"100m\"}},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\"}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"serviceAccount\":\"voting\",\"serviceAccountName\":\"voting\",\"terminationGracePeriodSeconds\":30}}},\"status\":{\"conditions\":[{\"message\":\"Created new replica set \\\"voting-f999bd4d7\\\"\",\"reason\":\"NewReplicaSetCreated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"message\":\"Deployment does not have minimum availability.\",\"reason\":\"MinimumReplicasUnavailable\",\"status\":\"False\",\"type\":\"Available\"}],\"observedGeneration\":1,\"unavailableReplicas\":1}}\n"},"resourceVersion":"1183"},"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"enabled"}}}},"status":{"$setElementOrder/conditions":[{"type":"Progressing"},{"type":"Available"}],"conditions":[{"message":"Created new replica set \"voting-f999bd4d7\"","reason":"NewReplicaSetCreated","type":"Progressing"}]}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "voting", Namespace: "emojivoto"
for: "STDIN": Operation cannot be fulfilled on deployments.apps "voting": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"deployment.kubernetes.io/revision\":\"1\"},\"generation\":1,\"labels\":{\"app.kubernetes.io/name\":\"web\",\"app.kubernetes.io/part-of\":\"emojivoto\",\"app.kubernetes.io/version\":\"v11\"},\"name\":\"web\",\"namespace\":\"emojivoto\",\"resourceVersion\":\"1184\",\"uid\":\"0a8bc1ca-1afa-47bd-b7c6-fc8a2a50fdf8\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"app\":\"web-svc\",\"version\":\"v11\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"annotations\":{\"linkerd.io/inject\":\"enabled\"},\"labels\":{\"app\":\"web-svc\",\"version\":\"v11\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"WEB_PORT\",\"value\":\"8080\"},{\"name\":\"EMOJISVC_HOST\",\"value\":\"emoji-svc.emojivoto:8080\"},{\"name\":\"VOTINGSVC_HOST\",\"value\":\"voting-svc.emojivoto:8080\"},{\"name\":\"INDEX_BUNDLE\",\"value\":\"dist/index_bundle.js\"}],\"image\":\"docker.l5d.io/buoyantio/emojivoto-web:v11\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"web-svc\",\"ports\":[{\"containerPort\":8080,\"name\":\"http\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"100m\"}},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\"}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"serviceAccount\":\"web\",\"serviceAccountName\":\"web\",\"terminationGracePeriodSeconds\":30}}},\"status\":{\"conditions\":[{\"message\":\"Created new replica set \\\"web-79469b946f\\\"\",\"reason\":\"NewReplicaSetCreated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"message\":\"Deployment does not have minimum availability.\",\"reason\":\"MinimumReplicasUnavailable\",\"status\":\"False\",\"type\":\"Available\"}],\"observedGeneration\":1,\"unavailableReplicas\":1}}\n"},"resourceVersion":"1184"},"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"enabled"}}}},"status":{"$setElementOrder/conditions":[{"type":"Progressing"},{"type":"Available"}],"conditions":[{"message":"Created new replica set \"web-79469b946f\"","reason":"NewReplicaSetCreated","type":"Progressing"}]}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "web", Namespace: "emojivoto"
for: "STDIN": Operation cannot be fulfilled on deployments.apps "web": the object has been modified; please apply your changes to the latest version and try again

@navendu-pottekkat Do you or others know about it?

[pottekkat]

The script follows this: https://linkerd.io/2.10/getting-started/

I am not sure about the error 🤔

[gyohuangxin]

The script follows this: https://linkerd.io/2.10/getting-started/

I am not sure about the error 🤔

I tried it on my own minikube cluster and it works, it's a strange behavior. 🤔

[gyohuangxin]

@navendu-pottekkat I found the cause of this issue: We should execute kubectl get -n emojivoto deploy -o yaml | linkerd inject - | kubectl apply -f - after all pods are running well. And I tried to sleep 100 before here https://github.com/layer5io/meshery-smp-action/blob/bbfa036c76a8da8d30c7fc3be6386604b4c2f412/.github/workflows/scripts/linkerd_deploy.sh#L19, linkerd can be injected. However, the error rate was also 100% in performance result, it seems that process kubectl -n emojivoto port-forward svc/web-svc 8080:80 &> /dev/null & was killed at the step of running benchmarking. And I tried nohup &, minikube tunnel &, also didn't work, do you have any comments?

[pottekkat]

@gyohuangxin Hmm, does minikube tunnel &> /dev/null & get killed?

[gyohuangxin]

@navendu-pottekkat Yes, I tried this in this action:https://github.com/gyohuangxin/meshery-smp-action/runs/4561946823 I used minikube tunnel &> /dev/null & here , and output some debug info here, then it showed the connection to port 8080 was refused.

[pottekkat]

@Revolyssup Do you have any idea why this might happen here?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pottekkat]

@gyohuangxin Is this still an issue?

[gyohuangxin]

@navendu-pottekkat Yes, as I mentioned, I tried many ways to make Linkerd's application reachable, but it still failed. It's probably a issue of Github's runner, and we are trying to run SMP actions on CNCF cluster, and I will track if this issue will still occurs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity. However, you may choose to reopen this issue.