[Bug]: Neo4j loading not loading credentials from service role

colintkn commented 1 year ago

Contact Details

No response

What happened?

I've created a serviceAccount with annotations to a role which permissions to run S3 commands. However, neo4j is not using the serviceAccount, instead its using the node role. See this link to see the credential chain. https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.

Here are the configurations

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx:role/neo4j-eks
  creationTimestamp: "2023-08-23T02:11:45Z"
  name: neo4j
  namespace: neo4j

A check on the pod shows the correct role being used

    - name: AWS_ROLE_ARN
      value: arn:aws:iam::xxxxxx:role/neo4j-eks
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Chart Name

Cluster

Chart Version

5.5.0

Environment

Google Cloud Platform

Relevant log output

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

harshitsinghvi22 commented 1 year ago

Can you please share more information ? Which helm chart are you using ? Can you share the values.yaml ?

colintkn commented 1 year ago

helm chart version: 5.10.0 hosted on: eks Service account is manually created outside this helm chart as advised in #212

this is the values.yaml

---
# Configuration https://neo4j.com/docs/operations-manual/current/kubernetes/configuration/
neo4j:
  name: neo4j
  minimumClusterSize: ${cluster_size}
  resources:
    cpu: "0.5"
    memory: "2Gi"
  password: ${password}
  edition: "enterprise"
  acceptLicenseAgreement: "yes"

services:
  neo4j:
    enabled: false

image:
  customImage: "xxxxxxx.dkr.ecr.ap-southeast-1.amazonaws.com/docker.io/library/neo4j:5.10.0"
  imagePullPolicy: Always

volumes:
  data:
    mode: "dynamic"
    dynamic:
     # gp2 is a general-purpose SSD volume
     storageClassName: ${storage_class_name}

nodeSelector:
  workload-type : "db"
  topology.kubernetes.io/zone: ${az}

# https://neo4j.com/docs/operations-manual/current/configuration/configuration-settings/
config:
  server.metrics.prometheus.enabled: "true"
  server.metrics.prometheus.endpoint: "0.0.0.0:12004"
  dbms.logs.http.enabled: "true"
  server.logs.debug.enabled: "true"
  # Configure and installl APOC core https://neo4j.com/docs/operations-manual/current/kubernetes/plugins/
  server.directories.plugins: "/var/lib/neo4j/labs"
  dbms.security.procedures.unrestricted: "apoc.*"
  server.config.strict_validation.enabled: "false"
  dbms.security.procedures.allowlist: "apoc.*"

podSpec: 
  # Name of service account to use for the Neo4j Pod (optional)
  # this is useful if you want to use Workload Identity to grant permissions to access cloud resources e.g. cloud object storage (AWS S3 etc.)
  serviceAccountName: ${service_account_name} # service account created with role and role bindings 

apoc_config:
  apoc.trigger.enabled: "true"
  apoc.jdbc.neo4j.url: "jdbc:foo:bar"
  apoc.import.file.enabled: "true"

# Plugins Required https://neo4j.com/docs/apoc/5/import/web-apis/#_using_s3_protocol

harshitsinghvi22 commented 1 year ago

@colintkn I tried to reproduce the issue and i was not able to

i followed the instructions here for the eks service account annotations and created a service account called "my-service-account" with an s3 read and list access policy for a bucket named harshit-cft

I started my neo4j cluster using that and i can see the aws roles getting attached to the pod

    Environment:
      HELM_NEO4J_VERSION:           5.10.0
      HELM_CHART_VERSION:           5.10.0
      POD_NAME:                     standalone1-0 (v1:metadata.name)
      SERVICE_NEO4J_ADMIN:          standalone1-admin.default.svc.cluster.local
      SERVICE_NEO4J_INTERNALS:      standalone1-internals.default.svc.cluster.local
      SERVICE_NEO4J:                standalone1.default.svc.cluster.local
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::XXXX:role/harshit-role
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

 $ k describe sa my-service-account
Name:                my-service-account
Namespace:           default
Labels:              app.kubernetes.io/managed-by=eksctl
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXX:role/harshit-role
Image pull secrets:  <none>
Mountable secrets:   my-service-account-token-mld5k
Tokens:              my-service-account-token-mld5k
Events:              <none>

I was also able to login to the neo4j instance and perform an aws s3 ls command Note: I have used a custom image here which had aws s3 cli pre-installed

neo4j@standalone1-0:~$ aws s3 ls harshit-cft
2022-06-29 10:50:41       7428 02fba4e6da2ea5a1f4e537694e93da7a.template
2022-06-29 13:59:46       7011 98a0e3f6dbd2f839a1ff7671e14c3512.template
2022-06-29 09:43:03       7424 ad1611dae5c3c3a0e8cf17e98d8415f1.template
neo4j@standalone1-0:~$

Also in order for the neo4j cluster to be up and running you have to create the below roles and role-bindings so that the instances can refer to the internal services created and form a connection

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: "default"
  name: "standalone-service-reader"
  labels:
    app: "test6"
rules:
  - apiGroups: [""] # "" indicates the core API group
    resources: ["services", "endpoints"]
    verbs: ["get", "watch", "list"]
---
# Source: neo4j/templates/neo4j-service-account.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: "default"
  name: "standalone-service-binding"
  labels:
    app: "test6"
subjects:
  - kind: ServiceAccount
    name: my-service-account
roleRef:
  # "roleRef" specifies the binding to a Role / ClusterRole
  kind: Role # this must be Role or ClusterRole
  name: standalone-service-reader # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

In conclusion, I cannot see any issue with neo4j when a customized service account is used

colintkn commented 1 year ago

Hi Harshit,

Yes. An AWS s3 ls command works. But an APOC call will not work in the cypher-shell of the pod.

CALL apoc.import.csv([{fileName: "s3://s3.ap-southeast-1.amazonaws.com/xxxx/
neo4j_import/xxxx.csv", labels: ["entities"
]}], [], {ignoreDuplicateNodes: true, ignoreBlankString: true});

It yields this error:

Failed to invoke procedure `apoc.import.csv`: Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

Is there a reason why the APOC plugin does not get credentials from the serviceAccount.

harshitsinghvi22 commented 1 year ago

@colintkn it would have been really helpful if you could have mentioned the issue was with apoc.import.csv in the description earlier. It has not been mentioned.

You can read more about how to use s3 protocol with apoc here https://neo4j.com/labs/apoc/5/import/web-apis/#_using_s3_protocol

For apoc related issues please use the below repos whichever applicable https://github.com/neo4j/apoc https://github.com/neo4j-contrib/neo4j-apoc-procedures

neo4j / helm-charts