PermissionDenied on app creation using project tokens in >= v1.8.3

[jcstryker]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

WIth v1.8.3 I am getting this error when attempting to create/sync an application using project tokens. Seems similar to #1019

The same process/yaml worked in v1.7.6

FATA[0000] rpc error: code = PermissionDenied desc = permission denied: applications, create, machine-learning-platform/mlp-buildandtrain, sub: proj:machine-learning-platform:ci-cd, iat: 2021-02-02T20:46:06Z 

To Reproduce

I have created a project using this project yaml and running kubectl create -f project.yaml:

---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: machine-learning-platform
  namespace: argocd
spec:
  description: Project for MLP applications managed by the MLP team
  sourceRepos:
    - "*"
  destinations:
    - server: https://kubernetes.default.svc
      namespace: "redacted"
    - server: https://kubernetes.default.svc
      namespace: "argocd"
  clusterResourceWhitelist:
    - group: "*"
      kind: "*"
  roles:
    - description: MLP Project role for Admins
      name: admin
      policies:
        - p, proj:machine-learning-platform:admin, applications, *, machine-learning-platform/*, allow
      groups:
        - redacted
    - description: MLP Project role for CICD
      name: jenkins
      policies:
        - p, proj:machine-learning-platform:jenkins, applications, *, machine-learning-platform/*, allow

I then create a project token using this command

argocd proj role create-token machine-learning-platform ci-cd

which returns successfully

ID: b529a434-3e0f-443f-9a11-7da706581e3f
  Issued At: 2021-02-02T15:46:06-05:00
  Expires At: Never
  Token: eyJ...

I then attempt to create this application using this manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: mlp-buildandtrain
  namespace: argocd
spec:
  project: machine-learning-platform
  destination:
    server: https://kubernetes.default.svc
    namespace: redacted
  source:
    repoURL: ...
    path: /
    targetRevision: master

command:

export ARGOCD_AUTH_TOKEN=<token>

export ARGOCD_SERVER=<server>

argocd app create --upsert --file mlp-buildandtrain.yaml \

where I then recieve this error

FATA[0000] rpc error: code = PermissionDenied desc = permission denied: applications, create, machine-learning-platform/mlp-buildandtrain, sub: proj:machine-learning-platform:ci-cd, iat: 2021-02-02T20:46:06Z 

Expected behavior

I would expect the application to get created properly, instead of permission denied

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v1.8.3+0f9c684.dirty
  BuildDate: 2021-01-23T03:45:34Z
  GitCommit: 0f9c68427882bf4633d395cbfcd7c9271795fd9b
  GitTreeState: dirty
  GoVersion: go1.15.7
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.8.3+0f9c684
  BuildDate: 2021-01-21T22:20:39Z
  GitCommit: 0f9c68427882bf4633d395cbfcd7c9271795fd9b
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.8.1 2020-07-16T00:58:46Z
  Helm Version: v3.4.1+gc4e7485
  Kubectl Version: v1.17.8
  Jsonnet Version: v0.17.0

Logs

time="2021-02-02T20:37:47Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2021-02-02T20:37:47Z" grpc.time_ms=4.409 span.kind=server system=grpc
time="2021-02-02T20:42:10Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1612298161,\"iss\":\"argocd\",\"jti\":\"jenkins\",\"nbf\":1612298161,\"sub\":\"proj:machine-learning-platform:ci-cd\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Bytes of non-byte slice)" grpc.service=application.ApplicationService grpc.start_time="2021-02-02T20:42:10Z" span.kind=server system=grpc
time="2021-02-02T20:42:10Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = permission denied: applications, create, machine-learning-platform/mlp-buildandtrain, sub: proj:machine-learning-platform:ci-cd, iat: 2021-02-02T20:36:01Z" grpc.code=PermissionDenied grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-02-02T20:42:10Z" grpc.time_ms=0.552 span.kind=server system=grpc

[alexmt]

I've tried to reproduce it and noticed that machine-learning-platform project has roles admin and jenkins but not ci-cd. Tried to generate token for jenkins role and was able to successfully create app. It is possible that ci-cd was deleted ?

[jcstryker]

I think I copied the wrong yaml, here is the manifest right out of the cluster

kind: AppProject
metadata:
  creationTimestamp: "2021-02-02T22:27:37Z"
  generation: 2
  managedFields:
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        .: {}
        f:clusterResourceWhitelist: {}
        f:description: {}
        f:destinations: {}
        f:sourceRepos: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-02-02T22:27:37Z"
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:roles: {}
    manager: argocd-server
    operation: Update
    time: "2021-02-02T22:28:18Z"
  name: machine-learning-platform
  namespace: argocd
  resourceVersion: "1634800"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/argocd/appprojects/machine-learning-platform
  uid: a7acb4f7-eca6-4564-bdd4-487c37109a58
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  description: Project for MLP applications managed by the MLP team
  destinations:
  - namespace: ...
    server: https://kubernetes.default.svc
  - namespace: ...
    server: https://kubernetes.default.svc
  - namespace: argocd
    server: https://kubernetes.default.svc
  roles:
  - description: MLP Project role for Admins
    groups:
    - ...
    name: admin
    policies:
    - p, proj:machine-learning-platform:admin, applications, *, machine-learning-platform/*,
      allow
  - description: MLP Project role for CICD
    jwtTokens:
    - iat: 1612304898
      id: 2195d4c1-284f-4baa-a949-dacea3c86ec5
    name: ci-cd
    policies:
    - p, proj:machine-learning-platform:ci-cd, applications, *, machine-learning-platform/*,
      allow
  sourceRepos:
  - '*'

The ci-cd role is definitly there and lists a token, still getting the permission denied

[jcstryker]

@alexmt can you share how you reproduced it? maybe I am doing something wrong

[jcstryker]

I tried this again with v1.8.4 and got an interesting new error

FATA[0000] rpc error: code = Unauthenticated desc = invalid session: JWT token for role 'admin' issued at '1612552250' does not exist in project 'machine-learning-platform' 
FATA[0000] rpc error: code = Unauthenticated desc = invalid session: JWT token for role 'admin' issued at '1612552250' does not exist in project 'machine-learning-platform'

[alexmt]

That looks like another issue. Trying to reproduce a theory

[jcstryker]

  roles:
  - description: MLP Project role for Admins
    groups:
    - redacted
    jwtTokens:
    - iat: 1612553791
      id: test
    - iat: 1612552250
      id: d57c8f6a-60f8-4760-8ba5-e76699d373fd
    - iat: 1612552167
      id: jenkins
    name: admin
    policies:
    - p, proj:machine-learning-platform:admin, applications, *, machine-learning-platform/mlp-buildandtrain,
      allow
  sourceRepos:
  - '*'

Token is definitely there in the CR

[omgapuppy]

seeing the same in 1.8.5

[jcstryker]

still seeing this issue in v1.8.7