Closed colintkn closed 1 year ago
Can you please share more information ? Which helm chart are you using ? Can you share the values.yaml ?
helm chart version: 5.10.0 hosted on: eks Service account is manually created outside this helm chart as advised in #212
this is the values.yaml
---
# Configuration https://neo4j.com/docs/operations-manual/current/kubernetes/configuration/
neo4j:
name: neo4j
minimumClusterSize: ${cluster_size}
resources:
cpu: "0.5"
memory: "2Gi"
password: ${password}
edition: "enterprise"
acceptLicenseAgreement: "yes"
services:
neo4j:
enabled: false
image:
customImage: "xxxxxxx.dkr.ecr.ap-southeast-1.amazonaws.com/docker.io/library/neo4j:5.10.0"
imagePullPolicy: Always
volumes:
data:
mode: "dynamic"
dynamic:
# gp2 is a general-purpose SSD volume
storageClassName: ${storage_class_name}
nodeSelector:
workload-type : "db"
topology.kubernetes.io/zone: ${az}
# https://neo4j.com/docs/operations-manual/current/configuration/configuration-settings/
config:
server.metrics.prometheus.enabled: "true"
server.metrics.prometheus.endpoint: "0.0.0.0:12004"
dbms.logs.http.enabled: "true"
server.logs.debug.enabled: "true"
# Configure and installl APOC core https://neo4j.com/docs/operations-manual/current/kubernetes/plugins/
server.directories.plugins: "/var/lib/neo4j/labs"
dbms.security.procedures.unrestricted: "apoc.*"
server.config.strict_validation.enabled: "false"
dbms.security.procedures.allowlist: "apoc.*"
podSpec:
# Name of service account to use for the Neo4j Pod (optional)
# this is useful if you want to use Workload Identity to grant permissions to access cloud resources e.g. cloud object storage (AWS S3 etc.)
serviceAccountName: ${service_account_name} # service account created with role and role bindings
apoc_config:
apoc.trigger.enabled: "true"
apoc.jdbc.neo4j.url: "jdbc:foo:bar"
apoc.import.file.enabled: "true"
# Plugins Required https://neo4j.com/docs/apoc/5/import/web-apis/#_using_s3_protocol
@colintkn I tried to reproduce the issue and i was not able to
i followed the instructions here for the eks service account annotations and created a service account called "my-service-account" with an s3 read and list access policy for a bucket named harshit-cft
I started my neo4j cluster using that and i can see the aws roles getting attached to the pod
Environment:
HELM_NEO4J_VERSION: 5.10.0
HELM_CHART_VERSION: 5.10.0
POD_NAME: standalone1-0 (v1:metadata.name)
SERVICE_NEO4J_ADMIN: standalone1-admin.default.svc.cluster.local
SERVICE_NEO4J_INTERNALS: standalone1-internals.default.svc.cluster.local
SERVICE_NEO4J: standalone1.default.svc.cluster.local
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: eu-west-1
AWS_REGION: eu-west-1
AWS_ROLE_ARN: arn:aws:iam::XXXX:role/harshit-role
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
$ k describe sa my-service-account
Name: my-service-account
Namespace: default
Labels: app.kubernetes.io/managed-by=eksctl
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXX:role/harshit-role
Image pull secrets: <none>
Mountable secrets: my-service-account-token-mld5k
Tokens: my-service-account-token-mld5k
Events: <none>
I was also able to login to the neo4j instance and perform an aws s3 ls command Note: I have used a custom image here which had aws s3 cli pre-installed
neo4j@standalone1-0:~$ aws s3 ls harshit-cft
2022-06-29 10:50:41 7428 02fba4e6da2ea5a1f4e537694e93da7a.template
2022-06-29 13:59:46 7011 98a0e3f6dbd2f839a1ff7671e14c3512.template
2022-06-29 09:43:03 7424 ad1611dae5c3c3a0e8cf17e98d8415f1.template
neo4j@standalone1-0:~$
Also in order for the neo4j cluster to be up and running you have to create the below roles and role-bindings so that the instances can refer to the internal services created and form a connection
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: "default"
name: "standalone-service-reader"
labels:
app: "test6"
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["services", "endpoints"]
verbs: ["get", "watch", "list"]
---
# Source: neo4j/templates/neo4j-service-account.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: "default"
name: "standalone-service-binding"
labels:
app: "test6"
subjects:
- kind: ServiceAccount
name: my-service-account
roleRef:
# "roleRef" specifies the binding to a Role / ClusterRole
kind: Role # this must be Role or ClusterRole
name: standalone-service-reader # this must match the name of the Role or ClusterRole you wish to bind to
apiGroup: rbac.authorization.k8s.io
In conclusion, I cannot see any issue with neo4j when a customized service account is used
Hi Harshit,
Yes. An AWS s3 ls command works. But an APOC call will not work in the cypher-shell of the pod.
CALL apoc.import.csv([{fileName: "s3://s3.ap-southeast-1.amazonaws.com/xxxx/
neo4j_import/xxxx.csv", labels: ["entities"
]}], [], {ignoreDuplicateNodes: true, ignoreBlankString: true});
It yields this error:
Failed to invoke procedure `apoc.import.csv`: Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;
Is there a reason why the APOC plugin does not get credentials from the serviceAccount.
@colintkn it would have been really helpful if you could have mentioned the issue was with apoc.import.csv in the description earlier. It has not been mentioned.
You can read more about how to use s3 protocol with apoc here https://neo4j.com/labs/apoc/5/import/web-apis/#_using_s3_protocol
For apoc related issues please use the below repos whichever applicable https://github.com/neo4j/apoc https://github.com/neo4j-contrib/neo4j-apoc-procedures
Contact Details
No response
What happened?
I've created a serviceAccount with annotations to a role which permissions to run S3 commands. However, neo4j is not using the serviceAccount, instead its using the node role. See this link to see the credential chain. https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.
Here are the configurations
A check on the pod shows the correct role being used
Chart Name
Cluster
Chart Version
5.5.0
Environment
Google Cloud Platform
Relevant log output
No response
Code of Conduct