search

openfaas / faas-netes

Serverless Functions For Kubernetes
https://www.openfaas.com
MIT License
2.12k stars 473 forks source link

Deployment Issue for the gateway, using helm template / faas-netes master repo. #818

Closed kmkcode closed 2 years ago

kmkcode commented 3 years ago

Deployment Issue for the gateway is failing when the namespace name is changed to the core services.

Expected Behaviour

It should take any namespace name for the core services which is provided in the namespace.yml file and while deploying using the template created with new namespace for the core services.

Current Behaviour

When trying to change the namespace name other than "openfaas" e.g. "test-faas". The gateway deployment is failing. and it start providing error such as mentioned below.

Verifying the kubectl logs for the gateway deployment - For gateway container : 

2021/07/22 05:17:45 HTTP Read Timeout: 1m5s
2021/07/22 05:17:45 HTTP Write Timeout: 1m5s
2021/07/22 05:17:45 Binding to external function provider: http://127.0.0.1:8081/
2021/07/22 05:17:45 Async enabled: Using NATS Streaming.
2021/07/22 05:17:45 Opening connection to nats://nats.test-serverless.svc.cluster.local:4222
2021/07/22 05:17:45 Connect: nats://nats.test-serverless.svc.cluster.local:4222
&{<nil>}
2021/07/22 05:17:49 error with upstream request to: /healthz, Get "http://127.0.0.1:8081/healthz": dial tcp 127.0.0.1:8081: connect: connection refused
2021/07/22 05:17:49 Forwarded [GET] to /healthz - [502] - 0.000458s seconds
2021/07/22 05:17:50 Get "http://127.0.0.1:8081/system/namespaces": dial tcp 127.0.0.1:8081: connect: connection refused
2021/07/22 05:17:50 Get "http://127.0.0.1:8081/system/functions?namespace=test-serverless-fn": dial tcp 127.0.0.1:8081: connect: connection refused
2021/07/22 05:17:53 error with upstream request to: /healthz, Get "http://127.0.0.1:8081/healthz": dial tcp 127.0.0.1:8081: connect: connection refused
2021/07/22 05:17:53 Forwarded [GET] to /healthz - [502] - 0.000417s seconds

For faas-netes container , following is the error : 

2021/07/22 04:45:59 Version: 0.13.4 commit: 6f34f27a2405798b5ee2846f1654bc7754991920
W0722 04:45:59.054332       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2021/07/22 04:45:59 HTTP Read Timeout: 1m0s
2021/07/22 04:45:59 HTTP Write Timeout: 1m0s
2021/07/22 04:45:59 ImagePullPolicy: Always
2021/07/22 04:45:59 DefaultFunctionNamespace: test-serverless-fn
2021/07/22 04:45:59 Starting controller
I0722 04:45:59.064549       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:deployments
E0722 04:45:59.068982       1 reflector.go:178] github.com/openfaas/faas-netes/main.go:178: Failed to list *v1.Deployment: Get "https://10.100.0.1:443/apis/apps/v1/namespaces/test-serverless-fn/deployments?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
E0722 04:45:59.069856       1 reflector.go:178] github.com/openfaas/faas-netes/main.go:178: Failed to list *v1.Deployment: Get "https://10.100.0.1:443/apis/apps/v1/namespaces/test-serverless-fn/deployments?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
E0722 04:46:16.581664       1 reflector.go:178] github.com/openfaas/faas-netes/main.go:178: Failed to list *v1.Deployment: an error on the server ("") has prevented the request from succeeding (get deployments.apps)

Are you a GitHub Sponsor (Yes/No?)

No

Check at: https://github.com/sponsors/openfaas

List All Possible Solutions and Workarounds

To stick with Original Default name "openfaas" for core services.

Which Solution Do You Recommend?

Preferred Solution is using the default namespace name i.e. "openfaas" for the core services, else it will not work.

Steps to Reproduce (for bugs)

  1. Downloaded faas-netes master branch : https://github.com/openfaas/faas-netes
  2. cd faas-netes folder
  3. Created template using following command : helm template openfaas chart/openfaas/ --namespace test-serverless --set basic_auth=false --set functionNamespace=test- 4.kubectl apply -f namespaces.yml,openfaas.yaml

Context

Your Environment

Kuberenetes version is following :+1: Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6+vmware.1", GitCommit:"afe4eda368a924376517be0c262541f358673efa", GitTreeState:"clean", BuildDate:"2020-12-22T01:08:16Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Please let me know on this.

kmkcode commented 3 years ago

Openfaas deployment doesn't work, when instead of namespace "openfaas" if any other name is given for "core services" namespace (e.g. test-serverless) , in namespaces.yml and while creating the template with command :

Can any one please provide the reason or where else the namespace name need to changes for the core services. ?

LucasRoesler commented 3 years ago

This error

Failed to list *v1.Deployment: an error on the server ("") has prevented the request from succeeding (get deployments.apps)

Looks like a problem with your k8s cluster. If it was just the namespace that was an issue I would expect something like an RBAC auth error.

This error says that k8s had some kind of internal error.

Your code snipper mentions a "namespaces.yaml", but you don't share the content of that file. Can you share that here?

alexellis commented 3 years ago

@kmkcode could you please edit the issue and fill out all the required fields? Thanks for using openfaas!

kmkcode commented 3 years ago

Hi All, Thanks for the reply. Please find the content for the Namespaces.yml file used before using original namespace name "openfaas" for core services : 

apiVersion: v1
kind: Namespace
metadata:
  name: test-serverless
  annotations:
    linkerd.io/inject: enabled
    config.linkerd.io/skip-inbound-ports: "4222"
    config.linkerd.io/skip-outbound-ports: "4222"
  labels:
    role: openfaas-system
    access: openfaas-system
    istio-injection: enabled
---------
apiVersion: v1
kind: Namespace
metadata:
  name: test-serverless-fn
  annotations:
    linkerd.io/inject: enabled
    config.linkerd.io/skip-inbound-ports: "4222"
    config.linkerd.io/skip-outbound-ports: "4222"
  labels:
    istio-injection: enabled
    role: test-serverless-fn
kmkcode commented 3 years ago

This error

Failed to list *v1.Deployment: an error on the server ("") has prevented the request from succeeding (get deployments.apps)

Looks like a problem with your k8s cluster. If it was just the namespace that was an issue I would expect something like an RBAC auth error.

This error says that k8s had some kind of internal error.

Your code snipper mentions a "namespaces.yaml", but you don't share the content of that file. Can you share that here?

Thanks, If that is the scenario with k8 cluster, then it should be the case with namespace "openfaas" too. But with original name there no issue in deploying the gateway. Please find the content of the namespaces.yml above.

kmkcode commented 3 years ago

The Logs for using the default namespace "openfaas" is following, which successfully deployed the gateway after 2 restart. (Please Note I have tried with both Default / Operator mode both have similar issue for not accepting the userdefined namespace name for the core services ) : 

Found 2 pods, using pod/gateway-569c9f7874-cngqk
2021/07/26 17:14:54 Version: 0.13.2 commit: 6caf24ba21f7af171e2ade40daa033b05584b695
W0726 17:14:54.246296       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2021/07/26 17:14:54 HTTP Read Timeout: 1m0s
2021/07/26 17:14:54 HTTP Write Timeout: 1m0s
2021/07/26 17:14:54 ImagePullPolicy: IfNotPresent
2021/07/26 17:14:54 DefaultFunctionNamespace: test-serverless-fn
2021/07/26 17:14:54 Starting operator
I0726 17:14:54.250615       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:functions
E0726 17:14:54.251489       1 reflector.go:178] github.com/openfaas/faas-netes/main.go:169: Failed to list *v1.Function: Get "https://10.100.0.1:443/apis/openfaas.com/v1/namespaces/test-serverless-fn/functions?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
E0726 17:14:54.253151       1 reflector.go:178] github.com/openfaas/faas-netes/main.go:169: Failed to list *v1.Function: Get "https://10.100.0.1:443/apis/openfaas.com/v1/namespaces/test-serverless-fn/functions?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
E0726 17:14:57.359626       1 reflector.go:178] github.com/openfaas/faas-netes/main.go:169: Failed to list *v1.Function: Get "https://10.100.0.1:443/apis/openfaas.com/v1/namespaces/test-serverless-fn/functions?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
I0726 17:15:02.750829       1 shared_informer.go:230] Caches are synced for faas-netes:functions 
I0726 17:15:02.750974       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:deployments
I0726 17:15:02.851275       1 shared_informer.go:230] Caches are synced for faas-netes:deployments 
I0726 17:15:02.851381       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:endpoints
I0726 17:15:02.951698       1 shared_informer.go:230] Caches are synced for faas-netes:endpoints 
I0726 17:15:02.951742       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:profiles
I0726 17:15:03.051901       1 shared_informer.go:230] Caches are synced for faas-netes:profiles 
I0726 17:15:03.052188       1 controller.go:112] Setting up event handlers
I0726 17:15:03.052390       1 server.go:82] Using namespace 'test-serverless-fn'
I0726 17:15:03.052401       1 controller.go:155] Waiting for informer caches to sync
I0726 17:15:03.052407       1 controller.go:160] Starting workers
I0726 17:15:03.052411       1 controller.go:166] Started workers
I0726 17:15:03.052459       1 server.go:97] Starting HTTP server on port 8081
alexellis commented 3 years ago

@kmkcode hope that you saw my message?

You haven't filled out the issue template, and just deleted it.

We need all the fields to be filled in, thanks.

In the meantime, feel free to submit an issue with full reproducible steps - every step, no omissions. Then we can prioritise this amongst all the other users' requests.

If you need consulting and technical support on OpenFaaS, feel free to book an hour of my time with your company or manager's credit card. I'd be happy to help.

alexellis commented 3 years ago

FYI, using code blocks would help us greatly in assisting you. Please see: https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks

alexellis commented 3 years ago

/add label: technical support

kmkcode commented 3 years ago

Hi @alexellis ,

Please check above, I have tried to provide as much details as I can. Trying to understand is there any other dependency which I need to change such as role, access for Openfaas to pick the new namespace name for the core services.

alexellis commented 3 years ago

We need the exact commands you used, otherwise this falls outside the realms of what I can offer you for free.

Give me the exact bash that I can type in to see the same error for instance.

Quick question - have you looked at the link I sent you about formatting code blocks? Does it make sense?

The offer is still open for consulting or an hour's session to help you debug your environment.

kmkcode commented 3 years ago

" Steps to Reproduce " 

1. Downloaded faas-netes master branch : https://github.com/openfaas/faas-netes
2. cd faas-netes folder
3. Created template using following command :
 helm template openfaas chart/openfaas/ --namespace test-serverless --set basic_auth=false --set functionNamespace=test-serverless-fn 
4. kubectl apply -f namespaces.yml,openfaas.yaml

"namespaces.yml content : "

apiVersion: v1
kind: Namespace
metadata:
  name: test-serverless
  annotations:
    linkerd.io/inject: enabled
    config.linkerd.io/skip-inbound-ports: "4222"
    config.linkerd.io/skip-outbound-ports: "4222"
  labels:
    role: openfaas-system
    access: openfaas-system
    istio-injection: enabled
---------
apiVersion: v1
kind: Namespace
metadata:
  name: test-serverless-fn
  annotations:
    linkerd.io/inject: enabled
    config.linkerd.io/skip-inbound-ports: "4222"
    config.linkerd.io/skip-outbound-ports: "4222"
  labels:
    istio-injection: enabled
    role: test-serverless-fn
LucasRoesler commented 2 years ago

@kmkcode i know this has been stale for a bit, but i was going through the issues backlog today and decided to retry this with the data you have provided.

When I do the following, I get a functional OpenFaas

First I created namespaces.yaml with your content, then i did this 

$ kind create cluster 
$ helm template openfaas chart/openfaas/ --namespace test-serverless --set basic_auth=false --set functionNamespace=test-serverless-fn  > openfaas.yaml
$ kubectl apply -f test-namespaces.yaml,openfaas.yaml
namespace/test-serverless created
namespace/test-serverless-fn created
serviceaccount/openfaas-controller created
serviceaccount/openfaas-prometheus created
configmap/alertmanager-config created
configmap/prometheus-config created
customresourcedefinition.apiextensions.k8s.io/profiles.openfaas.com configured
role.rbac.authorization.k8s.io/openfaas-controller created
role.rbac.authorization.k8s.io/openfaas-profiles created
role.rbac.authorization.k8s.io/openfaas-prometheus created
role.rbac.authorization.k8s.io/openfaas-prometheus-fn created
rolebinding.rbac.authorization.k8s.io/openfaas-controller created
rolebinding.rbac.authorization.k8s.io/openfaas-profiles created
rolebinding.rbac.authorization.k8s.io/openfaas-prometheus created
rolebinding.rbac.authorization.k8s.io/openfaas-prometheus-fn created
service/alertmanager created
service/gateway-external created
service/gateway created
service/nats created
service/prometheus created
deployment.apps/alertmanager created
deployment.apps/gateway created
deployment.apps/nats created
deployment.apps/prometheus created
deployment.apps/queue-worker created
$ kubectl rollout status deploy gateway -n test-serverless
Waiting for deployment "gateway" rollout to finish: 0 of 1 updated replicas are available...
deployment "gateway" successfully rolled out
$  kubectl logs -n test-serverless deploy/gateway faas-netes
2021/10/23 09:29:47 Version: 0.13.8 commit: c0a8c3cba4156fac9847953a83cea03bf54e42ef
W1023 09:29:47.831039       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2021/10/23 09:29:47 HTTP Read Timeout: 1m0s
2021/10/23 09:29:47 HTTP Write Timeout: 1m0s
2021/10/23 09:29:47 ImagePullPolicy: Always
2021/10/23 09:29:47 DefaultFunctionNamespace: test-serverless-fn
2021/10/23 09:29:47 Starting controller
I1023 09:29:47.832511       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:deployments
I1023 09:29:47.934408       1 shared_informer.go:230] Caches are synced for faas-netes:deployments 
I1023 09:29:47.934570       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:endpoints
I1023 09:29:48.035466       1 shared_informer.go:230] Caches are synced for faas-netes:endpoints 
I1023 09:29:48.035541       1 shared_informer.go:223] Waiting for caches to sync for faas-netes:profiles
I1023 09:29:48.135984       1 shared_informer.go:230] Caches are synced for faas-netes:profiles

Now, clearly, based on your namespaces definitions, you are also using Linkerd or istio. Perhaps this workshop would be useful for you https://github.com/openfaas/openfaas-linkerd-workshop

kmkcode commented 2 years ago

Thanks for the update. I can see you able to rename the core and function namespace. Any blocker which you can think of in Kubernetes which may not allowing the renaming (its corporate Kubernetes server) ? I have to debug this scenario again.

I will take a look at that workshop. We use Istio. I tried with istio and without istio it was the same result.

Let me see what I can do. Thanks once again.

Best Regards, Kamakshi S.

On Sat, Oct 23, 2021 at 02:34 Lucas Roesler @.***> wrote:

@kmkcode https://github.com/kmkcode i know this has been stale for a bit, but i was going through the issues backlog today and decided to retry this with the data you have provided.

When I do the following, I get a functional OpenFaas

First I created namespaces.yaml with your content, then i did this

$ kind create cluster $ helm template openfaas chart/openfaas/ --namespace test-serverless --set basic_auth=false --set functionNamespace=test-serverless-fn > openfaas.yaml $ kubectl apply -f test-namespaces.yaml,openfaas.yaml namespace/test-serverless created namespace/test-serverless-fn created serviceaccount/openfaas-controller created serviceaccount/openfaas-prometheus created configmap/alertmanager-config created configmap/prometheus-config createdcustomresourcedefinition.apiextensions.k8s.io/profiles.openfaas.com configuredrole.rbac.authorization.k8s.io/openfaas-controller createdrole.rbac.authorization.k8s.io/openfaas-profiles createdrole.rbac.authorization.k8s.io/openfaas-prometheus createdrole.rbac.authorization.k8s.io/openfaas-prometheus-fn createdrolebinding.rbac.authorization.k8s.io/openfaas-controller createdrolebinding.rbac.authorization.k8s.io/openfaas-profiles createdrolebinding.rbac.authorization.k8s.io/openfaas-prometheus createdrolebinding.rbac.authorization.k8s.io/openfaas-prometheus-fn created service/alertmanager created service/gateway-external created service/gateway created service/nats created service/prometheus created deployment.apps/alertmanager created deployment.apps/gateway created deployment.apps/nats created deployment.apps/prometheus created deployment.apps/queue-worker created $ kubectl rollout status deploy gateway -n test-serverless Waiting for deployment "gateway" rollout to finish: 0 of 1 updated replicas are available... deployment "gateway" successfully rolled out $ kubectl logs -n test-serverless deploy/gateway faas-netes 2021/10/23 09:29:47 Version: 0.13.8 commit: c0a8c3cba4156fac9847953a83cea03bf54e42ef W1023 09:29:47.831039 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. 2021/10/23 09:29:47 HTTP Read Timeout: 1m0s 2021/10/23 09:29:47 HTTP Write Timeout: 1m0s 2021/10/23 09:29:47 ImagePullPolicy: Always 2021/10/23 09:29:47 DefaultFunctionNamespace: test-serverless-fn 2021/10/23 09:29:47 Starting controller I1023 09:29:47.832511 1 shared_informer.go:223] Waiting for caches to sync for faas-netes:deployments I1023 09:29:47.934408 1 shared_informer.go:230] Caches are synced for faas-netes:deployments I1023 09:29:47.934570 1 shared_informer.go:223] Waiting for caches to sync for faas-netes:endpoints I1023 09:29:48.035466 1 shared_informer.go:230] Caches are synced for faas-netes:endpoints I1023 09:29:48.035541 1 shared_informer.go:223] Waiting for caches to sync for faas-netes:profiles I1023 09:29:48.135984 1 shared_informer.go:230] Caches are synced for faas-netes:profiles

Now, clearly, based on your namespaces definitions, you are also using Linkerd or istio. Perhaps this workshop would be useful for you https://github.com/openfaas/openfaas-linkerd-workshop

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openfaas/faas-netes/issues/818#issuecomment-950124276, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP3HVC2W3LRMF6JTF5XAAYLUIJ6TNANCNFSM5AZI2SNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

LucasRoesler commented 2 years ago

@kmkcode we have seen some issues in the past with istio and linkerd, depending on their configuration. I am not an expert on either service mesh, so I am not sure the exact configurations that work or don't work, but I do know that people are using both linkerd and istio with openfaas, so it will work with the correct configuration.

We also have a blog post from this year specifically about istio, perhaps that will help: https://www.openfaas.com/blog/istio-functions/

alexellis commented 2 years ago

It seems like we cannot reproduce this and are falling into the realms of R&D which should really come under an enterprise support agreement with @kmkcode's employer.

@LucasRoesler thanks for confirming that this cannot be reproduced in a normal environment.

@kmkcode feel free to reach out to us via email contact@openfaas.com about some consulting or support to dedicate time to helping you debug your environment.

alexellis commented 2 years ago

/add label: support,question