arangodb / kube-arangodb

ArangoDB Kubernetes Operator - Start ArangoDB on Kubernetes in 5min
https://arangodb.github.io/kube-arangodb/
Apache License 2.0
227 stars 71 forks source link

kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access #716

Closed tmd313 closed 2 years ago

tmd313 commented 3 years ago

I was trying to use the API to get in via JWT. The token created in the secret: arango-kube-arangodb-1618862786-operator-token-bwdvh I took out, converted it with | base64 -d and then used that via jwtgen using the recipe described in the docs to create a superuser jwt/token. I also created the cacert by taking the ca.crt from the secret, decoding it w/base64 -d, and putting it in a cacert file. I attempted connecting via kubernetes internal network by going into another pod (I can't get any kind of shell in the current/slim arangodb pod (likely the alpine:311 is so lean) and using curl. These are what I was using in an attempt to get to the server using the new jwt I generated using the decoded token in the secret.

Or if I can find out a way I'm supposed to generate the root user password that could help too. (BTW, I don't need to nor want to obfuscate the token below, just so that you can see what it looks like.)

the myjson has only this in it (again root user password is supposed to be empty):

{ "username": "root", "password": "" }

The script did this. I get a 400/bad request on the first one and a 404 on each subsequent API call. Seems like it's not really a 404 issue though as these are basic, are they not? Script:

TOKEN="eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2MTg5NDczODQsImV4cCI6MTYxODk1MDk4NSwiaXNzIjoiYXJhbmdvZGIiLCJzZXJ2ZXJfaWQiOiJteWNsaWVudCJ9.J1chHuzfMAkT6_xQrG_BJB9IHsCv7vbnjqLxnKVw5b4"

CACERT="/tmp/cacert"

curl -i -X POST --cacert ${CACERT} -k -H "Content-Type:application/json; charset=utf-8; Accept:application/json" -H "Authorization: Bearer ${TOKEN}" -H "Content-Length: 43" "https://arango-kube-arangodb-1618862786-operator.default.svc:8528/_open/auth -d@/tmp/myjson" curl -i -X GET --cacert ${CACERT} -k -H "Content-Type:application/json, Accept:application/json" -H "Authorization: Bearer ${TOKEN}" "https://arango-kube-arangodb-1618862786-operator.default.svc:8528/_admin/server/jwt"

curl -i -X GET --cacert ${CACERT} -k -H "Content-Type:application/json, Accept:application/json" -H "Authorization: Bearer ${TOKEN}" "https://arango-kube-arangodb-1618862786-operator.default.svc:8528/api/version"

curl -i -X GET --cacert ${CACERT} -k -H "Content-Type:application/json, Accept:application/json" -H "Authorization: Bearer ${TOKEN}" "https://arango-kube-arangodb-1618862786-operator.default.svc:8528/_api/version"

curl -i -X GET --cacert ${CACERT} -k -H "Content-Type:application/json, Accept:application/json" -H "Authorization: bearer ${TOKEN}" "https://arango-kube-arangodb-1618862786-operator.default.svc:8528/_api/query/current"

When I go to the https external endpoint:8528 (I changed it to a LoadBalancer (from the ClusterIP it originally was) and gave it an externalIP address) and all I get is a login screen with login: and password: fields but the green box on the lower right stays "grayed out" until I enter something and neither '' nor "" work as the "empty" password.

Here's the output from a run of the script above, which also contains an /api/version url w/out the _ underscore, just to try that too.

this is from within another pod in the weave network:

./queryRootUser.sh

HTTP/1.1 400 Bad Request Content-Type: text/plain; charset=utf-8 Connection: close

400 Bad RequestHTTP/1.1 404 Not Found Content-Type: text/plain Date: Wed, 21 Apr 2021 20:03:34 GMT Content-Length: 18

404 page not foundHTTP/1.1 404 Not Found Content-Type: text/plain Date: Wed, 21 Apr 2021 20:03:34 GMT Content-Length: 18

404 page not foundHTTP/1.1 404 Not Found Content-Type: text/plain Date: Wed, 21 Apr 2021 20:03:34 GMT Content-Length: 18

404 page not foundHTTP/1.1 404 Not Found Content-Type: text/plain Date: Wed, 21 Apr 2021 20:03:34 GMT Content-Length: 18

404 page not found

Please help and/or point me to reference docs that I may have missed, I've been scouring around for days on this and read the documentation over and over again.

Thanks, Tim

ajanikow commented 3 years ago

Hello!

Do you try to access Operator or ArangoDB endpoint? From API calls I see you to to reach ArangoDB, but endpoint and JWT token is related to Operator itself.

Once you install chart you get only Operator, after that you need to define ArangoDeployment: https://github.com/arangodb/kube-arangodb/blob/master/examples/production-cluster.yaml

It will create for you service production-cluster-ea with LoadBalancer type and production-cluster-jwt with JWT secret.

Best Regards, Adam.

tmd313 commented 3 years ago

Thank you sir. I will do so, I wasn’t sure about that and what I had so far. I’ll do that and inspect it. I’m assuming then I will get root user with empty password at that time, but more likely at either 8529 or 8530, from examples I’ve seen so far. And will that make a service for coordinators, or is that implicit now. Sorry, I’m asking more questions. I’ll do what you advise and take it from there.

Thanks so much,

        Tim

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Adam Janikowski @.> Sent: Wednesday, April 21, 2021 5:23 PM To: arangodb/kube-arangodb @.> Cc: Dillon, Tim @.>; Author @.> Subject: Re: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

Hello! Do you try to access Operator or ArangoDB endpoint? From API calls I see you to to reach ArangoDB, but endpoint and JWT token is related to Operator itself. Once you install chart you get only External @.**@.>) Report This Emailhttps://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL3RpbS5kaWxsb25AY29tbXNjb3BlLmNvbS9lYjY1NWVjNzk4MzFmYTljYTQwMWRiMTJiMThhNjYyOC8xNjE5MDQwMTg1Ljg1#key=d33e94acc88a588be295dff2623d3f77 FAQhttps://www.inky.com/banner-faq/ Protection by INKYhttps://www.inky.com

Hello!

Do you try to access Operator or ArangoDB endpoint? From API calls I see you to to reach ArangoDB, but endpoint and JWT token is related to Operator itself.

Once you install chart you get only Operator, after that you need to define ArangoDeployment: https://github.com/arangodb/kube-arangodb/blob/master/examples/production-cluster.yamlhttps://secure-web.cisco.com/1ygwDpvm846vdYldhImSp91Tn1OKrWliFrFTPL6qh2s2h1E5xSoAKoLB3ZBY1PJzNkTMgAwNY5_Cy0uTSrh1KUywhyXPw70tlA8HG8XzW5SDampgvEvFf4gKKC5QheSrMaWqz1MfkAEQ7ZyH1_l-mJBh-gCz_fVqRu-qQ-Vy5vQ15UfJETS-VThqheUdmmmsb5FTLwMcTpW-xAYuAABk2jcI-ZtYuYv3LlRGgIS7LZ7a-CyXtYwa3-YuMaKmlCsKS8EcV1f9wwPmovOpZ-Oe3MVAlewpEQWfoFMOLfYlv2nRmon_UTIKjwkTv9TL2GiHJ/https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb%2Fblob%2Fmaster%2Fexamples%2Fproduction-cluster.yaml

It will create for you service production-cluster-ea with LoadBalancer type and production-cluster-jwt with JWT secret.

Best Regards, Adam.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1xtiS8h1_YMVAoMleg63seRd8V9KcJDdB4oBjC44UEDoZ34a3LRaLBhLmcdtMKn6MwOehXXMV07wcnx19hvJu7g3FXvHv5L1gyXefy269CGjxqdKE-FzCY_P-qLPRkgy8RLFh9xtMnQWOb38XcLuSEjk749HHgSrFPB8GF35lv6cgQmNz6VI1niYf0pzpDcxmD463BGnHiilgbELFR0T9nSznvzmwXhO_Md8iUTQFKIbdrtHWU-2amysePi3gCPgJFI96hX76A4eE-DCa_8JQRx24leIobDtkLF8PTT61mKvo4ESqD7vJDwCPJRQ56vGT/https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb%2Fissues%2F716%23issuecomment-824366433, or unsubscribehttps://secure-web.cisco.com/1XrxUWRLRbQZXBSdXMiKijOX7OKqUzLO60CKsLEYoi6v-RBKrb2MCdtnAB3WjEMzPSNGjk8iu9s-Mzcjbco5cGvnWcCfsT0p-uSfSo68s5GWrm7Udg-2S2R-aZJ-xd7zdNowJrN-rFHwH5fTluxMX3HEttJFPnPmqzIAvlZICf63-L_kFZSACK34RrIAaz9HNT2JDsaVDHQxymgHRVApodenr557wyIEo5O5rRywRz5xIvZZarOHKidIWEOK7IzxlqzmHdb9rC-YOWpFHfjsYj9X8pSClk1-tHfBOmdyj6_MwetBCSvmknY8QtAivKCKU/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHZ2IP6VNDIIUC24CP2UGV3TJ463NANCNFSM43LBOSDQ.

tmd313 commented 3 years ago

BTW, I was using 1.1.5 and came across a comment on the 1.1.5.1 that that was a bit more aligned with kubernetes version 1.19, we're using 1.19.7 currently. Should I move to the 1.1.5.1? Thanks, Tim

ajanikow commented 3 years ago

Hello!

Not needed, 1.17 is fully compatible with 1.19. We changed it only due to fact that one of our API users required this.

Best, Adam.

tmd313 commented 3 years ago

I ran this, using the following, the other day I downloaded the latest of this image from arangodb/arangodb:latest and moved it into our repo since my local site has a problem with docker hub such that we cannot get images from it.

cat production-cluster.yaml apiVersion: "database.arangodb.com/v1" kind: "ArangoDeployment" metadata: name: "production-cluster" spec: mode: Cluster image: ':5000/arangodb/arangodb:lastest' environment: Production

I get these artifacts in bold from kubectl apply -f production-cluster.yaml All of the others were already there. I added the service production-cluster-ea2 in attempt to get a LoadBalancer type of service with an external-IP as our kubernetes cluster does not give them out automatically:

kubemaster1 arangodbCharts ]$ kubectl get pods,svc,secrets,deployments --all-namespaces -o wide|egrep arango default pod/arango-kube-arangodb-1618862786-operator-6fd78fb5ff-97c8w 1/1 Running 4 6d20h 172.16.224.3 p12-tdillon-compute1.broadbus.com default pod/arango-kube-arangodb-1618862786-operator-6fd78fb5ff-phfhx 1/1 Running 0 6d20h 172.16.16.4 p12-tdillon-compute2.broadbus.com default service/arango-kube-arangodb-1618862786-operator LoadBalancer 10.101.150.62 10.184.138.170 8528:31269/TCP 6d20h app.kubernetes.io/instance=kube-arangodb-1618862786,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kube-arangodb,release=kube-arangodb-1618862786,role=leader default service/production-cluster ClusterIP 10.101.54.185 8529/TCP 51m app=arangodb,arango_deployment=production-cluster,role=coordinator default service/production-cluster-ea NodePort 10.108.212.231 8529:30630/TCP 32m app=arangodb,arango_deployment=production-cluster,role=coordinator default service/production-cluster-ea2 LoadBalancer 10.96.6.32 10.184.138.169 8529:31403/TCP 18m app=arangodb,arango_deployment=production-cluster,role=coordinator default service/production-cluster-int ClusterIP None 8529/TCP 51m app=arangodb,arango_deployment=production-cluster default secret/arango-kube-arangodb-1618862786-operator-token-bwdvh kubernetes.io/service-account-token 3 6d20h default secret/sh.helm.release.v1.kube-arangodb-1618862786.v1 helm.sh/release.v1 1 6d20h default secret/sh.helm.release.v1.kube-arangodb-crd-1618840162.v1 helm.sh/release.v1 1 7d3h default deployment.apps/arango-kube-arangodb-1618862786-operator 2/2 2 2 6d20h operator 10.184.128.10:5000/arangodb/kube-arangodb:1.1.5 app.kubernetes.io/instance=kube-arangodb-1618862786,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kube-arangodb,release=kube-arangodb-1618862786

BUT, I don’t see any new pods? Should there be some?

I still cannot connect to arangodb at :8529 port on any of the nodePorts or LB at external-IP, :8529. I do “see” the nodePort via “lsof -i :30630 on our compute node(s).

Is there a way to pass in an LB External-IP to the cluster?

Thanks

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Adam Janikowski @.> Sent: Wednesday, April 21, 2021 5:23 PM To: arangodb/kube-arangodb @.> Cc: Dillon, Tim @.>; Author @.> Subject: Re: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

Hello! Do you try to access Operator or ArangoDB endpoint? From API calls I see you to to reach ArangoDB, but endpoint and JWT token is related to Operator itself. Once you install chart you get only External @.**@.>) Report This Emailhttps://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL3RpbS5kaWxsb25AY29tbXNjb3BlLmNvbS9lYjY1NWVjNzk4MzFmYTljYTQwMWRiMTJiMThhNjYyOC8xNjE5MDQwMTg1Ljg1#key=d33e94acc88a588be295dff2623d3f77 FAQhttps://www.inky.com/banner-faq/ Protection by INKYhttps://www.inky.com

Hello!

Do you try to access Operator or ArangoDB endpoint? From API calls I see you to to reach ArangoDB, but endpoint and JWT token is related to Operator itself.

Once you install chart you get only Operator, after that you need to define ArangoDeployment: https://github.com/arangodb/kube-arangodb/blob/master/examples/production-cluster.yamlhttps://secure-web.cisco.com/1ygwDpvm846vdYldhImSp91Tn1OKrWliFrFTPL6qh2s2h1E5xSoAKoLB3ZBY1PJzNkTMgAwNY5_Cy0uTSrh1KUywhyXPw70tlA8HG8XzW5SDampgvEvFf4gKKC5QheSrMaWqz1MfkAEQ7ZyH1_l-mJBh-gCz_fVqRu-qQ-Vy5vQ15UfJETS-VThqheUdmmmsb5FTLwMcTpW-xAYuAABk2jcI-ZtYuYv3LlRGgIS7LZ7a-CyXtYwa3-YuMaKmlCsKS8EcV1f9wwPmovOpZ-Oe3MVAlewpEQWfoFMOLfYlv2nRmon_UTIKjwkTv9TL2GiHJ/https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb%2Fblob%2Fmaster%2Fexamples%2Fproduction-cluster.yaml

It will create for you service production-cluster-ea with LoadBalancer type and production-cluster-jwt with JWT secret.

Best Regards, Adam.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1xtiS8h1_YMVAoMleg63seRd8V9KcJDdB4oBjC44UEDoZ34a3LRaLBhLmcdtMKn6MwOehXXMV07wcnx19hvJu7g3FXvHv5L1gyXefy269CGjxqdKE-FzCY_P-qLPRkgy8RLFh9xtMnQWOb38XcLuSEjk749HHgSrFPB8GF35lv6cgQmNz6VI1niYf0pzpDcxmD463BGnHiilgbELFR0T9nSznvzmwXhO_Md8iUTQFKIbdrtHWU-2amysePi3gCPgJFI96hX76A4eE-DCa_8JQRx24leIobDtkLF8PTT61mKvo4ESqD7vJDwCPJRQ56vGT/https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb%2Fissues%2F716%23issuecomment-824366433, or unsubscribehttps://secure-web.cisco.com/1XrxUWRLRbQZXBSdXMiKijOX7OKqUzLO60CKsLEYoi6v-RBKrb2MCdtnAB3WjEMzPSNGjk8iu9s-Mzcjbco5cGvnWcCfsT0p-uSfSo68s5GWrm7Udg-2S2R-aZJ-xd7zdNowJrN-rFHwH5fTluxMX3HEttJFPnPmqzIAvlZICf63-L_kFZSACK34RrIAaz9HNT2JDsaVDHQxymgHRVApodenr557wyIEo5O5rRywRz5xIvZZarOHKidIWEOK7IzxlqzmHdb9rC-YOWpFHfjsYj9X8pSClk1-tHfBOmdyj6_MwetBCSvmknY8QtAivKCKU/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHZ2IP6VNDIIUC24CP2UGV3TJ463NANCNFSM43LBOSDQ.

ajanikow commented 3 years ago

Hello!

Can you check logs of Operator? It should say why pods cant be created. For example image cannot be pulled.

About question: https://www.arangodb.com/docs/stable/deployment-kubernetes-deployment-resource.html#specexternalaccessloadbalancerip-string

Best Regards, Adam.

tmd313 commented 3 years ago

The attached log snippet from the leading operator shows the issues. The first one that cropped up was:

2021-04-26T16:13:36Z DBG Failed to fetch deployment shard sync state error="JWT Secret is missing" component=deployment deployment=production-cluster operator-id=97c8w 2021-04-26T16:13:36Z DBG Failed to fetch deployment health error="JWT Secret is missing" component=deployment deployment=production-cluster operator-id=97c8w

It then shows what appears to be attempting to bring up AGNTs (3), PRMRs (3) and CRDNs (3) but no live cluster.

Let me know if there’s anything else I can collect to help with the diagnosis.

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Adam Janikowski @.> Sent: Tuesday, April 27, 2021 4:33 AM To: arangodb/kube-arangodb @.> Cc: Dillon, Tim @.>; Author @.> Subject: Re: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

Hello! Can you check logs of Operator? It should say why pods cant be created. For example image cannot be pulled. About question: https://www.arangodb.com/docs/stable/deployment-kubernetes-deployment External @.**@.>) Report This Emailhttps://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL3RpbS5kaWxsb25AY29tbXNjb3BlLmNvbS9kYTQ0NGM3MzkzZjA4NDNlMDA5MTBlYjZlNjMyZDRjMy8xNjE5NTEyMzkzLjA5#key=4ce51a4f8c0f1e1bc6d11f93317198e8 FAQhttps://www.inky.com/banner-faq/ Protection by INKYhttps://www.inky.com

Hello!

Can you check logs of Operator? It should say why pods cant be created. For example image cannot be pulled.

About question: https://www.arangodb.com/docs/stable/deployment-kubernetes-deployment-resource.html#specexternalaccessloadbalancerip-stringhttps://secure-web.cisco.com/1QyzGAbDFQIocZIYPG_SeOK_C2gUrQV9RQC3QOKs9CervwFqBsOjjxawhoeDayNQSq75ClJF9NaB4BKjP9NKELzbHzRFk9xUjIhMfWGvbeVCYr1egQIXIeguj_RpjIvYUreXd8IgswpdnsBtHdonz1B7hmV3l3b89NRhwa277fhn-OCxcf1Wqe3NDs_y3j0OitPm6095h-eYGa6gpuzU_Z5EEKyzHufPR2cBqdr9OHzJmCfoCTKrKDw0FmmdTLDbEXUZPynbmFTrnO_S-ymqQhyLCCMO0BRViKcNUJ4R8e6JhUcUumj5wsjukb2rP43Za/https%3A%2F%2Fwww.arangodb.com%2Fdocs%2Fstable%2Fdeployment-kubernetes-deployment-resource.html%23specexternalaccessloadbalancerip-string

Best Regards, Adam.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1Guvybh1KSYYoxCb6bG_ldBNlh1g-e00M4bZdVkHSm5UzTTxGYoUBTHC7WrILNT_0VRipaP0tnPmQm3_b_JiI9th2M1UvEK5yRMVhYSbjBWIzeRfWwrUlXTYYP7IKzx42UTsd4tcZDN1MA64YH61QGwJ85OB46LPLn-lxXof7x57PKiSiwAEas1yPfztxieQEVXFosP0NOlYXWsR0Qv65SLeg3ZUQoe02NZgUNBlK7N6ltdFT7mzUx_dsRoP53LuNqh_Jwf_do4DirVU7T9w-DpwcDVa-TpURdLfX99NmRfIU5rzXwMmgByW_nwzRSiOA/https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb%2Fissues%2F716%23issuecomment-827424199, or unsubscribehttps://secure-web.cisco.com/1fTdmEsMWItUbcgFIw92LiaAZSDwqsqG6uwvwfFl7Xo3_f_kyhdNKsdBp4tukf9ABSTzTx7tRYKgquijmUZ31QnaDQEEEVSY8FZNeZNidoqt56g7AKb6p5bJz5lEsFHepZt2NIgvVd5N8vWedtKr13VDtXofyz_Mpzf8YaqStTs4g7vD80YAOzf_hWRHF2VxtrRV_fNP68_vBB5wYTzVaCise_ZqyA-RN-n_ov2ePhTkFffnSKoR8bnXoBM1Wz8Z1_qfb2mNjmZUMgDZ25GkYEYhedsbU0W5DsVQy1VB0AoEkHuHi8jxdf2EaC00YqsMg/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHZ2IP7V5NIZLG36N6EN2DDTKZZEJANCNFSM43LBOSDQ.

tmd313 commented 3 years ago

So Adam, looks like I need to create a jwt secret (auth.jwtSecretName) and a cacert secret (tls.caSecretName) for the deployment. I could use some direction on what exactly to put in those. I can, of course, create the secrets but not sure on generating the content for them.

As for this:

spec.bootstrap.passwordSecretNames.root: string This setting specifies a secret name for the credentials of the root user. When a deployment is created the operator will setup the root user account according to the credentials given by the secret. If the secret doesn’t exist the operator creates a secret with a random password. There are two magic values for the secret name:

What are the credential field keys it will be looking for? Perhaps “username” and “password”? or maybe something less obvious, or is it implied that it is the root user and I only need a proper tag, aka key, for the base64 encrypted password?

Can you point me to any examples in the documentation, please?

I’m getting there now. The link you gave me yesterday has more info in it than the pages similar to it that I have come across. The expanded "example-arangodb-cluster" Yaml/”ArangoDeployment” spec is helpful.

Thanks, Tim

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Dillon, Tim Sent: Tuesday, April 27, 2021 9:47 AM To: arangodb/kube-arangodb @.>; arangodb/kube-arangodb @.> Cc: Author @.***> Subject: RE: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

The attached log snippet from the leading operator shows the issues. The first one that cropped up was:

2021-04-26T16:13:36Z DBG Failed to fetch deployment shard sync state error="JWT Secret is missing" component=deployment deployment=production-cluster operator-id=97c8w 2021-04-26T16:13:36Z DBG Failed to fetch deployment health error="JWT Secret is missing" component=deployment deployment=production-cluster operator-id=97c8w

It then shows what appears to be attempting to bring up AGNTs (3), PRMRs (3) and CRDNs (3) but no live cluster.

Let me know if there’s anything else I can collect to help with the diagnosis.

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Adam Janikowski @.**@.>> Sent: Tuesday, April 27, 2021 4:33 AM To: arangodb/kube-arangodb @.**@.>> Cc: Dillon, Tim @.**@.>>; Author @.**@.>> Subject: Re: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

Hello! Can you check logs of Operator? It should say why pods cant be created. For example image cannot be pulled. About question: https://www.arangodb.com/docs/stable/deployment-kubernetes-deployment External @.**@.>) Report This Emailhttps://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL3RpbS5kaWxsb25AY29tbXNjb3BlLmNvbS9kYTQ0NGM3MzkzZjA4NDNlMDA5MTBlYjZlNjMyZDRjMy8xNjE5NTEyMzkzLjA5#key=4ce51a4f8c0f1e1bc6d11f93317198e8 FAQhttps://www.inky.com/banner-faq/ Protection by INKYhttps://www.inky.com

Hello!

Can you check logs of Operator? It should say why pods cant be created. For example image cannot be pulled.

About question: https://www.arangodb.com/docs/stable/deployment-kubernetes-deployment-resource.html#specexternalaccessloadbalancerip-stringhttps://secure-web.cisco.com/1QyzGAbDFQIocZIYPG_SeOK_C2gUrQV9RQC3QOKs9CervwFqBsOjjxawhoeDayNQSq75ClJF9NaB4BKjP9NKELzbHzRFk9xUjIhMfWGvbeVCYr1egQIXIeguj_RpjIvYUreXd8IgswpdnsBtHdonz1B7hmV3l3b89NRhwa277fhn-OCxcf1Wqe3NDs_y3j0OitPm6095h-eYGa6gpuzU_Z5EEKyzHufPR2cBqdr9OHzJmCfoCTKrKDw0FmmdTLDbEXUZPynbmFTrnO_S-ymqQhyLCCMO0BRViKcNUJ4R8e6JhUcUumj5wsjukb2rP43Za/https%3A%2F%2Fwww.arangodb.com%2Fdocs%2Fstable%2Fdeployment-kubernetes-deployment-resource.html%23specexternalaccessloadbalancerip-string

Best Regards, Adam.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1Guvybh1KSYYoxCb6bG_ldBNlh1g-e00M4bZdVkHSm5UzTTxGYoUBTHC7WrILNT_0VRipaP0tnPmQm3_b_JiI9th2M1UvEK5yRMVhYSbjBWIzeRfWwrUlXTYYP7IKzx42UTsd4tcZDN1MA64YH61QGwJ85OB46LPLn-lxXof7x57PKiSiwAEas1yPfztxieQEVXFosP0NOlYXWsR0Qv65SLeg3ZUQoe02NZgUNBlK7N6ltdFT7mzUx_dsRoP53LuNqh_Jwf_do4DirVU7T9w-DpwcDVa-TpURdLfX99NmRfIU5rzXwMmgByW_nwzRSiOA/https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb%2Fissues%2F716%23issuecomment-827424199, or unsubscribehttps://secure-web.cisco.com/1fTdmEsMWItUbcgFIw92LiaAZSDwqsqG6uwvwfFl7Xo3_f_kyhdNKsdBp4tukf9ABSTzTx7tRYKgquijmUZ31QnaDQEEEVSY8FZNeZNidoqt56g7AKb6p5bJz5lEsFHepZt2NIgvVd5N8vWedtKr13VDtXofyz_Mpzf8YaqStTs4g7vD80YAOzf_hWRHF2VxtrRV_fNP68_vBB5wYTzVaCise_ZqyA-RN-n_ov2ePhTkFffnSKoR8bnXoBM1Wz8Z1_qfb2mNjmZUMgDZ25GkYEYhedsbU0W5DsVQy1VB0AoEkHuHi8jxdf2EaC00YqsMg/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHZ2IP7V5NIZLG36N6EN2DDTKZZEJANCNFSM43LBOSDQ.

tmd313 commented 3 years ago

I went further and defined this yaml to launch the production-cluster, but it has issues setting up the ExternalIP address for the cluster. This is the yaml I launch with:

apiVersion: "database.arangodb.com/v1" kind: "ArangoDeployment" metadata: name: "production-cluster" spec: mode: Cluster environment: Production image: "10.184.128.10:5000/arangodb/arangodb:lastest" externalAccess: type: LoadBalancer loadBalancerIP: “10.184.138.169” auth: jwtSecretName: None <-- until I can figure out the requirements for the content of the secret of this name agents: count: 3 args:

Here’s what we “see” via kubectl:

arangodbCharts ]$ kubectl get pods,svc,secrets,deployments --all-namespaces -o wide|egrep arango default pod/arango-kube-arangodb-1618862786-operator-6fd78fb5ff-97c8w 1/1 Running 4 8d 172.16.224.3 p12-tdillon-compute1.broadbus.com default pod/arango-kube-arangodb-1618862786-operator-6fd78fb5ff-phfhx 1/1 Running 0 8d 172.16.16.4 p12-tdillon-compute2.broadbus.com default service/arango-kube-arangodb-1618862786-operator LoadBalancer 10.101.150.62 10.184.138.170 8528:31269/TCP 8d app.kubernetes.io/instance=kube-arangodb-1618862786,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kube-arangodb,release=kube-arangodb-1618862786,role=leader default service/production-cluster ClusterIP 10.97.55.184 8529/TCP 13m app=arangodb,arango_deployment=production-cluster,role=coordinator default service/production-cluster-ea LoadBalancer 10.103.198.123 8529:32671/TCP 13m app=arangodb,arango_deployment=production-cluster,role=coordinator default service/production-cluster-int ClusterIP None 8529/TCP 13m app=arangodb,arango_deployment=production-cluster default secret/arango-kube-arangodb-1618862786-operator-token-bwdvh kubernetes.io/service-account-token 3 8d default secret/sh.helm.release.v1.kube-arangodb-1618862786.v1 helm.sh/release.v1 1 8d default secret/sh.helm.release.v1.kube-arangodb-crd-1618840162.v1 helm.sh/release.v1 1 8d default deployment.apps/arango-kube-arangodb-1618862786-operator 2/2 2 2 8d operator 10.184.128.10:5000/arangodb/kube-arangodb:1.1.5 app.kubernetes.io/instance=kube-arangodb-1618862786,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kube-arangodb,release=kube-arangodb-1618862786

And attached is the log of this latest attempt. It seems it has issues w/DNS (as seen by the :53 address that it fails on) and name resolution. The name that it complains about, does resolve. This is the error:

2021-04-27T20:17:20Z DBG Failed to fetch deployment health error="Get \"https://production-cluster.default.svc:8529/_admin/server/role\": dial tcp: lookup production-cluster.default.svc on 10.96.0.10:53: no such host" component=deployment deployment=production-cluster operator-id=97c8w

Note the resolution, from another pod in the k8s cluster:

@.***:/# ping production-cluster.default.svc PING production-cluster.default.svc.cluster.local (10.97.55.184) 56(84) bytes of data.

Also note, the presence of that IP in the services shown above by the “kubectl get” command:

-kubemaster1 arangodbCharts ]$ kubectl get pods,svc,secrets,deployments --all-namespaces -o wide|grep 10.97.55.184 default service/production-cluster ClusterIP 10.97.55.184 8529/TCP 16m app=arangodb,arango_deployment=production-cluster,role=coordinator

One particular question also on the “Pending” shown above for the externalIP I defined in the yaml being: 10.184.138.169 That is pingable and active in the local environment in k8s:

@.*** arangodbCharts ]$ ping 10.184.138.169 PING 10.184.138.169 (10.184.138.169) 56(84) bytes of data. 64 bytes from 10.184.138.169: icmp_seq=1 ttl=64 time=0.288 ms ^C --- 10.184.138.169 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms

Any pointers you can supply are greatly appreciated.

Thanks, Tim

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Dillon, Tim Sent: Tuesday, April 27, 2021 3:36 PM To: arangodb/kube-arangodb @.>; arangodb/kube-arangodb @.> Cc: Author @.***> Subject: RE: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

So Adam, looks like I need to create a jwt secret (auth.jwtSecretName) and a cacert secret (tls.caSecretName) for the deployment. I could use some direction on what exactly to put in those. I can, of course, create the secrets but not sure on generating the content for them.

As for this:

spec.bootstrap.passwordSecretNames.root: string This setting specifies a secret name for the credentials of the root user. When a deployment is created the operator will setup the root user account according to the credentials given by the secret. If the secret doesn’t exist the operator creates a secret with a random password. There are two magic values for the secret name:

What are the credential field keys it will be looking for? Perhaps “username” and “password”? or maybe something less obvious, or is it implied that it is the root user and I only need a proper tag, aka key, for the base64 encrypted password?

Can you point me to any examples in the documentation, please?

I’m getting there now. The link you gave me yesterday has more info in it than the pages similar to it that I have come across. The expanded "example-arangodb-cluster" Yaml/”ArangoDeployment” spec is helpful.

Thanks, Tim

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Dillon, Tim Sent: Tuesday, April 27, 2021 9:47 AM To: arangodb/kube-arangodb @.**@.>>; arangodb/kube-arangodb @.**@.>> Cc: Author @.**@.>> Subject: RE: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

The attached log snippet from the leading operator shows the issues. The first one that cropped up was:

2021-04-26T16:13:36Z DBG Failed to fetch deployment shard sync state error="JWT Secret is missing" component=deployment deployment=production-cluster operator-id=97c8w 2021-04-26T16:13:36Z DBG Failed to fetch deployment health error="JWT Secret is missing" component=deployment deployment=production-cluster operator-id=97c8w

It then shows what appears to be attempting to bring up AGNTs (3), PRMRs (3) and CRDNs (3) but no live cluster.

Let me know if there’s anything else I can collect to help with the diagnosis.

Tim Dillon Distinguished Software Engineer

ARRIS AND RUCKUS HAVE JOINED COMMSCOPE 900 Chelmsford St, Lowell, MA 01851 USA Office: +01-978-614-3145

From: Adam Janikowski @.**@.>> Sent: Tuesday, April 27, 2021 4:33 AM To: arangodb/kube-arangodb @.**@.>> Cc: Dillon, Tim @.**@.>>; Author @.**@.>> Subject: Re: [arangodb/kube-arangodb] kube-arangodb v. 1.1.5 does not create root user with empty password and fails using jwt access (#716)

Hello! Can you check logs of Operator? It should say why pods cant be created. For example image cannot be pulled. About question: https://www.arangodb.com/docs/stable/deployment-kubernetes-deployment External @.**@.>) Report This Emailhttps://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL3RpbS5kaWxsb25AY29tbXNjb3BlLmNvbS9kYTQ0NGM3MzkzZjA4NDNlMDA5MTBlYjZlNjMyZDRjMy8xNjE5NTEyMzkzLjA5#key=4ce51a4f8c0f1e1bc6d11f93317198e8 FAQhttps://www.inky.com/banner-faq/ Protection by INKYhttps://www.inky.com

Hello!

Can you check logs of Operator? It should say why pods cant be created. For example image cannot be pulled.

About question: https://www.arangodb.com/docs/stable/deployment-kubernetes-deployment-resource.html#specexternalaccessloadbalancerip-stringhttps://secure-web.cisco.com/1QyzGAbDFQIocZIYPG_SeOK_C2gUrQV9RQC3QOKs9CervwFqBsOjjxawhoeDayNQSq75ClJF9NaB4BKjP9NKELzbHzRFk9xUjIhMfWGvbeVCYr1egQIXIeguj_RpjIvYUreXd8IgswpdnsBtHdonz1B7hmV3l3b89NRhwa277fhn-OCxcf1Wqe3NDs_y3j0OitPm6095h-eYGa6gpuzU_Z5EEKyzHufPR2cBqdr9OHzJmCfoCTKrKDw0FmmdTLDbEXUZPynbmFTrnO_S-ymqQhyLCCMO0BRViKcNUJ4R8e6JhUcUumj5wsjukb2rP43Za/https%3A%2F%2Fwww.arangodb.com%2Fdocs%2Fstable%2Fdeployment-kubernetes-deployment-resource.html%23specexternalaccessloadbalancerip-string

Best Regards, Adam.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1Guvybh1KSYYoxCb6bG_ldBNlh1g-e00M4bZdVkHSm5UzTTxGYoUBTHC7WrILNT_0VRipaP0tnPmQm3_b_JiI9th2M1UvEK5yRMVhYSbjBWIzeRfWwrUlXTYYP7IKzx42UTsd4tcZDN1MA64YH61QGwJ85OB46LPLn-lxXof7x57PKiSiwAEas1yPfztxieQEVXFosP0NOlYXWsR0Qv65SLeg3ZUQoe02NZgUNBlK7N6ltdFT7mzUx_dsRoP53LuNqh_Jwf_do4DirVU7T9w-DpwcDVa-TpURdLfX99NmRfIU5rzXwMmgByW_nwzRSiOA/https%3A%2F%2Fgithub.com%2Farangodb%2Fkube-arangodb%2Fissues%2F716%23issuecomment-827424199, or unsubscribehttps://secure-web.cisco.com/1fTdmEsMWItUbcgFIw92LiaAZSDwqsqG6uwvwfFl7Xo3_f_kyhdNKsdBp4tukf9ABSTzTx7tRYKgquijmUZ31QnaDQEEEVSY8FZNeZNidoqt56g7AKb6p5bJz5lEsFHepZt2NIgvVd5N8vWedtKr13VDtXofyz_Mpzf8YaqStTs4g7vD80YAOzf_hWRHF2VxtrRV_fNP68_vBB5wYTzVaCise_ZqyA-RN-n_ov2ePhTkFffnSKoR8bnXoBM1Wz8Z1_qfb2mNjmZUMgDZ25GkYEYhedsbU0W5DsVQy1VB0AoEkHuHi8jxdf2EaC00YqsMg/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHZ2IP7V5NIZLG36N6EN2DDTKZZEJANCNFSM43LBOSDQ.

ajanikow commented 3 years ago

Hello!

To get content of file you can try to omit this field - just comment auth part and apply deployment. Operator will create all required secrets for you. In principal jwt secret is just secret with 'token' in data. Token can be any random string with size up to 32.

Best Regards, Adam