Issue with CRD when namespaced

[volatilemolotov]

Describe the bug When installing Ambassador Edge Stack in namespaced mode (using new chart datawire/ambassasdor) it fails with 'panic: ratelimits.getambassador.io is forbidden: User "system:serviceaccount:a8r-default:ambassador-a8r-default" cannot list resource "ratelimits" in API group "getambassador.io" at the cluster scope'

To Reproduce Steps to reproduce the behavior:

1. Install ambassador with chart using scope: singleNamespace: true
2. Go to logs

Expected behavior Ambassador should boot normally. It should not try to list ratelimits in cluster scope. It should just list them at namespace scope

Versions (please complete the following information):

• Ambassador: 1.0.0
• Kubernetes environment GKE 1.14

Additional context Similar issue: https://github.com/datawire/ambassador/issues/1576

[etotten]

@volatilemolotov - I'm having this same issue too. Did you find a work-around?

[volatilemolotov]

Issue still persists, its not causing distruption for me, its just logs spam

[etotten]

Oh, interesting. It was actually crashing the pod for me in Ambassador 1.2.2, so I worked-around by adding a ClusterRole and Binding giving cluster-level perms. That work-around feels pretty sloppy, so was hoping this might get some attention.

[volatilemolotov]

Yeah i just realised that i also gave it a clusterrole and a clusterrolebinding as a temp fix. Sorry for misleading you.

Also hoping that this gets proper attention

[zwbear]

I get the same error as you with following version, as i installed ambassador via helm chart datawire/ambassador.

appVersion: 1.3.2 chartVersion: 6.2.3

@etotten Is there any problem by asign ClusterRole with multiple ambassador instance?

[iNoahNothing]

Sorry for missing this issue until now all.

This is an issue with the RBAC permissions that AES needs.

Setting scope.singleNamespace: true configures the chart to install the RBAC resources as Role and RoleBinding instead of ClusterRole and ClusterRoleBinding. With the AES, there is an object that it is looking for the it needs permission to look at the cluster scope for.

For this reason, the solution to get Ambassador running in single namespace scoped is to manually set the AMBASSADOR_SINGLE_NAMESPACE environment variable but keep the rbac cluster-scoped.

You can do this with manually passing in setting with helm install ambassador -n ambassador datawire/ambassador --set env.AMBASSADOR_SINGLE_NAMESPACE=true

Or simply setting

env:
  AMBASSADOR_SINGLE_NAMESPACE: true

in a values.yaml file.

This will need to be addressed since we should support installing without cluster-scoped RBAC.

[zwbear]

@nbkrause Thanks for your reply. By me, it seems like a problem on the other corner. I have using the following values.yaml.

namespace:
  name: testspace
env:
  AMBASSADOR_ID: testsapce
service:
  type: ClusterIP
crds:
  create: false
scope:
  singleNamespace: true

The role, rolebindung and serviceaccount are created only within the namespace, but the program does not starts due to the "ratelimitservice" resource scanning.

Error Message:

E0403 14:18:06.084383     110 reflector.go:270] pkg/mod/k8s.io/client-go@v11.0.1-0.20190816222228-6d55c1b1f1ca+incompatible/tools/cache/reflector.go:94: Failed to watch u003cnilu003e: Get https://10.1.0.1:443/apis/getambassador.io/v1/namespaces/testspace/ratelimitservices?resourceVersion=77508503u0026timeoutSeconds=375u0026watch=true: dial tcp 10.1.0.1:443: connect: connection refused 

[iNoahNothing]

@zwbear I think your issue is different. You are getting a connection refused trying to connect to the kubernetes api to look for ratelimitservices. The issue as originally described is cannot list resource "ratelimits" in API group "getambassador.io" at the cluster scope which is a permission issue when setting scope.singleNamespace: true.

Yours seems more related to some networking failure in your cluster.

[zwbear]

I get the namespaces ambassador to work with following ClusterRole and ClusterRoleBinding installed before the ambassador installation. I think, the version 6.3.4 still get the same problem as 6.2.3

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ambassador-namespaced-additional
rules:
  - apiGroups:
      - getambassador.io
    resources:
      - ratelimits
      - filters
      - filterpolicies
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ambassador-namespaced-additional
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ambassador-namespaced-additional
subjects:
  - kind: ServiceAccount
    name: <<MyReleaseName>>
    namespace: <<MyReleaseNamespace>>

It seems like, that those three resources ratelimit, filters and filterpolicies works not fine with single namespace.

[yxzhm]

Sorry for missing this issue until now all.

This is an issue with the RBAC permissions that AES needs.

Setting scope.singleNamespace: true configures the chart to install the RBAC resources as Role and RoleBinding instead of ClusterRole and ClusterRoleBinding. With the AES, there is an object that it is looking for the it needs permission to look at the cluster scope for.

For this reason, the solution to get Ambassador running in single namespace scoped is to manually set the AMBASSADOR_SINGLE_NAMESPACE environment variable but keep the rbac cluster-scoped.

You can do this with manually passing in setting with helm install ambassador -n ambassador datawire/ambassador --set env.AMBASSADOR_SINGLE_NAMESPACE=true

Or simply setting

env:
  AMBASSADOR_SINGLE_NAMESPACE: true

in a values.yaml file.

This will need to be addressed since we should support installing without cluster-scoped RBAC.

Does that means, the cluster-scoped RBAC is still required, even though set the "AMBASSADOR_SINGLE_NAMESPACE: true"? Do we plan to fix this issue in recent version?

[zwbear]

@yxzhm Thanks for your reply. By the installation, I already set env.AMBASSADOR_SINGLE_NAMESPACE=true in values.yaml. But it doen't works.

---
namespace:
  name: project-test
env:
  AMBASSADOR_ID: project-test
  AMBASSADOR_SINGLE_NAMESPACE: true
service:
  type: ClusterIP
scope:
  singleNamespace: true
authService:
  create: false
RateLimit:
  create: false

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nshazly]

I was able to fix this error by correcting the clusterrolebinding that is created by ambassador. The helm chart assigns the namespace in the roleref to 'ambassador'. If ambassador is installed into another namespace the serviceaccount will be able to assume the cluster role.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cindymullins-dw]

Looks like there is a fix/workaround for this issue. Please let us know if issues persist on 2.x or 3.x Ambassador versions.

[vyom-soft]

Hello,

I followed https://www.getambassador.io/docs/emissary/latest/tutorials/getting-started#emissary-ingress-quick-start

I deployed the emissary CRD v3.9.1 in a namespace called emissary And latest emissary-ingress also in the same namespace emissary. Now I see the error:

I am not sure why the error is show in in emissary-ingress logs. In my values.yaml

  # Set the AMBASSADOR_SINGLE_NAMESPACE environment variable and create namespaced RBAC if rbac.enabled: true
  singleNamespace: false