dhruvdave-iv
commented
1 year ago I am facing the exact same problem.
Open luiscvega opened 1 year ago
dhruvdave-iv
commented
1 year ago I am facing the exact same problem.
dgoldssfo
commented
1 year ago @luiscvega @dhruvdave-iv I have this occasionally as well, and the standard advice from our developers is to just run the script a second time, and usually it works. If it doesn't try running it again.
I have a suspicion it might be due to slow download speeds getting the Docker images down to the Docker repository on Minikube when my Internet speed is slow. I've been working from home since Covid started; never used to have this problem when working from an office. How is your connection speed - might that be a source of the problem?
One thing you could try is to docker pull the images to the Minikube Docker repo and see if that helps.
dgoldssfo
commented
1 year ago @luiscvega @dhruvdave-iv I just ran a test where I pulled the Docker images after I ran cdk-minikube, and before I ran forgeops install. It ran in 8 minutes with no timeouts.
From the release/7.2.0 branch, my steps were:
cd /path/to/forgeops/cluster/minikube
./cdk-minikube start
eval $(minikube docker-env) (needed so that the next steps prime the Docker repo in Minikube)
docker pull gcr.io/forgerock-io/am-base:7.2.0
docker pull gcr.io/forgerock-io/amster:7.2.0
docker pull gcr.io/forgerock-io/ds-empty:7.2.0
docker pull gcr.io/forgerock-io/idm-cdk:7.2.0
kubectl create namespace dgoldssfo
kubens dgoldssfo
cd ../../bin
date;./forgeops install --cdk --fqdn cdk.example.com --namespace dgoldssfo;dateCould you please try to replicate this, and let me know if it you avoid the timeouts (and how long it took to run forgeops install)?
If it does, I'll either update the cdk-minikube script to automatically prime the Minikube docker repo, or I'll add steps to the documentation to tell users to do it manually.
dhruvdave-iv
commented
1 year ago @dgoldssfo , I tried the steps you suggested, still the same error.
I did a bit more troubleshooting and found that ds-operator is failing to update the password. Bellow are the logs:
{"level":"info","ts":1675070210.6966136,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1675070210.6980314,"logger":"setup","msg":"starting manager"} I0130 09:16:50.699730 1 leaderelection.go:243] attempting to acquire leader lease fr-system/5bca1d4b.forgerock.io... {"level":"info","ts":1675070210.6997473,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} I0130 09:16:51.632629 1 leaderelection.go:253] successfully acquired lease fr-system/5bca1d4b.forgerock.io {"level":"info","ts":1675070211.6335588,"logger":"controller-runtime.manager.controller.directoryrestore","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryRestore","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.6336682,"logger":"controller-runtime.manager.controller.directoryrestore","msg":"Starting Controller","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryRestore"} {"level":"info","ts":1675070211.6351874,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.6352594,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.6352875,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.6353066,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Starting Controller","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService"} {"level":"info","ts":1675070211.6353812,"logger":"controller-runtime.manager.controller.directorybackup","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryBackup","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.635433,"logger":"controller-runtime.manager.controller.directorybackup","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryBackup","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.635444,"logger":"controller-runtime.manager.controller.directorybackup","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryBackup","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.6354647,"logger":"controller-runtime.manager.controller.directorybackup","msg":"Starting EventSource","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryBackup","source":"kind source: /, Kind="} {"level":"info","ts":1675070211.6354742,"logger":"controller-runtime.manager.controller.directorybackup","msg":"Starting Controller","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryBackup"} {"level":"info","ts":1675070212.4984589,"logger":"controller-runtime.manager.controller.directoryrestore","msg":"Starting workers","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryRestore","worker count":1} {"level":"info","ts":1675070212.6014392,"logger":"controller-runtime.manager.controller.directorybackup","msg":"Starting workers","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryBackup","worker count":1} {"level":"info","ts":1675070212.7932448,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Starting workers","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","worker count":1} {"level":"info","ts":1675070218.7873743,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"info","ts":1675070219.4201753,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Can't connect to ldap server, will try again later","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","url":"ldaps://ds-idrepo-0.ds-idrepo.cdk.svc:1636","err":"Cant open ldap connection to ldaps://ds-idrepo-0.ds-idrepo.cdk.svc:1636 using dn uid=admin : LDAP Result Code 200 \"Network Error\": dial tcp: lookup ds-idrepo-0.ds-idrepo.cdk.svc on 10.96.0.10:53: no such host"} {"level":"info","ts":1675070219.4228942,"logger":"controller-runtime.manager.controller.directoryservice","msg":"cant get ldap connection, will retry later","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"info","ts":1675070219.4246392,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"info","ts":1675070219.5979226,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Can't connect to ldap server, will try again later","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","url":"ldaps://ds-idrepo-0.ds-idrepo.cdk.svc:1636","err":"Cant open ldap connection to ldaps://ds-idrepo-0.ds-idrepo.cdk.svc:1636 using dn uid=admin : LDAP Result Code 200 \"Network Error\": dial tcp: lookup ds-idrepo-0.ds-idrepo.cdk.svc on 10.96.0.10:53: no such host"} {"level":"info","ts":1675070219.5980656,"logger":"controller-runtime.manager.controller.directoryservice","msg":"cant get ldap connection, will retry later","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"info","ts":1675070309.4242961,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"info","ts":1675070309.515982,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Can't connect to ldap server, will try again later","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","url":"ldaps://ds-idrepo-0.ds-idrepo.cdk.svc:1636","err":"Cant open ldap connection to ldaps://ds-idrepo-0.ds-idrepo.cdk.svc:1636 using dn uid=admin : LDAP Result Code 200 \"Network Error\": dial tcp: lookup ds-idrepo-0.ds-idrepo.cdk.svc on 10.96.0.10:53: no such host"} {"level":"info","ts":1675070309.516088,"logger":"controller-runtime.manager.controller.directoryservice","msg":"cant get ldap connection, will retry later","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"info","ts":1675070379.2989883,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"error","ts":1675070379.5741618,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Failed to update the password","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","dn":"uid=am-config,ou=admins,ou=am-config","error":"LDAP Result Code 2 \"Protocol Error\": The password modify extended request cannot be processed because it was not possible to identify the user entry to update based on the authorization DN of \"uid=am-config,ou=admins,ou=am-config\" The password modify extended request cannot be processed because it contained an invalid userIdentity field. The provided userIdentity string was \"uid=am-config,ou=admins,ou=am-config\"","stacktrace":"github.com/ForgeRock/ds-operator/controllers.(DirectoryServiceReconciler).Reconcile\n\t/workspace/controllers/directoryservice_controller.go:158\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:214"} {"level":"info","ts":1675070399.517484,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"error","ts":1675070399.8249173,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Failed to update the password","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","dn":"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens","error":"LDAP Result Code 2 \"Protocol Error\": The password modify extended request cannot be processed because it was not possible to identify the user entry to update based on the authorization DN of \"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens\" The password modify extended request cannot be processed because it contained an invalid userIdentity field. The provided userIdentity string was \"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens\"","stacktrace":"github.com/ForgeRock/ds-operator/controllers.(DirectoryServiceReconciler).Reconcile\n\t/workspace/controllers/directoryservice_controller.go:158\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:214"} {"level":"info","ts":1675070459.8271756,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"error","ts":1675070460.0026762,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Failed to update the password","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","dn":"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens","error":"LDAP Result Code 2 \"Protocol Error\": The password modify extended request cannot be processed because it was not possible to identify the user entry to update based on the authorization DN of \"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens\" The password modify extended request cannot be processed because it contained an invalid userIdentity field. The provided userIdentity string was \"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens\"","stacktrace":"github.com/ForgeRock/ds-operator/controllers.(DirectoryServiceReconciler).Reconcile\n\t/workspace/controllers/directoryservice_controller.go:158\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:214"} {"level":"info","ts":1675070520.002985,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"error","ts":1675070520.4222047,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Failed to update the password","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","dn":"uid=am-config,ou=admins,ou=am-config","error":"LDAP Result Code 2 \"Protocol Error\": The password modify extended request cannot be processed because it was not possible to identify the user entry to update based on the authorization DN of \"uid=am-config,ou=admins,ou=am-config\" The password modify extended request cannot be processed because it contained an invalid userIdentity field. The provided userIdentity string was \"uid=am-config,ou=admins,ou=am-config\"","stacktrace":"github.com/ForgeRock/ds-operator/controllers.(DirectoryServiceReconciler).Reconcile\n\t/workspace/controllers/directoryservice_controller.go:158\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:214"} {"level":"info","ts":1675070580.4230812,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Reconcile","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk"} {"level":"error","ts":1675070580.5175536,"logger":"controller-runtime.manager.controller.directoryservice","msg":"Failed to update the password","reconciler group":"directory.forgerock.io","reconciler kind":"DirectoryService","name":"ds-idrepo","namespace":"cdk","dn":"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens","error":"LDAP Result Code 2 \"Protocol Error\": The password modify extended request cannot be processed because it was not possible to identify the user entry to update based on the authorization DN of \"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens\" The password modify extended request cannot be processed because it contained an invalid userIdentity field. The provided userIdentity string was \"uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens\"","stacktrace":"github.com/ForgeRock/ds-operator/controllers.(DirectoryServiceReconciler).Reconcile\n\t/workspace/controllers/directoryservice_controller.go:158\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.2/pkg/internal/controller/controller.go:214"}
dgoldssfo
commented
1 year ago @luiscvega @dhruvdave-iv Are both of you attempting this on Linux? Could you say a little more about your environment or run the bin/debug-logs script (in the forgeops repo) and post the results here?
I haven't been able to duplicate this on macOS so far, so maybe it's something OS-specific. I'll see if I can grab a Linux laptop tomorrow and have at it.
luiscvega
commented
1 year ago It works! I pulled the images first before running it all over again. Although, I'm not sure if this is actually what fixed it. @dgoldssfo, can you confirm that this flow is not using any other external services? Probably not related but I also added the --namespace foo to the command this time.
In any case, I am very grateful for the assistance, @dgoldssfo. I am using Ubuntu 22.04.1 LTS. I'll keep this open for now just in case @dhruvdave-iv still needs assistance.
@dhruvdave-iv, what steps are you running? You can check my commands below and see if it will work.
/tmp/forgeops$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/k8s-minikube/kicbase v0.0.36 866c1fe4e3f2 3 months ago 1.11GB
gcr.io/forgerock-io/idm-cdk 7.2.0 33e04917e12b 7 months ago 457MB
gcr.io/forgerock-io/am-base 7.2.0 4544d9a8eb3c 7 months ago 831MB
gcr.io/forgerock-io/amster 7.2.0 facdc507968f 7 months ago 600MB
gcr.io/forgerock-io/ds-empty 7.2.0 8c486d924d5c 7 months ago 208MB
/tmp/forgeops$ ./cluster/minikube/cdk-minikube start
Running: "minikube start --cpus=3 --memory=9g --disk-size=40g --cni=true --kubernetes-version=1.23.3 --addons=ingress,volumesnapshots "
😄 minikube v1.28.0 on Ubuntu 22.04
🎉 minikube 1.29.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.29.0
💡 To disable this notice, run: 'minikube config set WantUpdateNotification false'
✨ Automatically selected the docker driver
📌 Using Docker driver with root privileges
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=3, Memory=9216MB) ...
🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.20 ...
▪ kubelet.cni-conf-dir=/etc/cni/net.mk
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
▪ Using image k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎 Verifying ingress addon...
🌟 Enabled addons: storage-provisioner, default-storageclass, volumesnapshots, ingress
❗ /usr/local/bin/kubectl is version 1.26.1, which may have incompatibilities with Kubernetes 1.23.3.
▪ Want kubectl v1.23.3? Try 'minikube kubectl -- get pods -A'
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
/tmp/forgeops$ date
Wednesday, 01 February, 2023 01:31:32 PM PST
/tmp/forgeops$ kubectl create namespace foo
namespace/foo created
/tmp/forgeops$ kubens foo
Context "minikube" modified.
Active namespace is "foo".
/tmp/forgeops$ date; ./bin/forgeops install --cdk --fqdn cdk.example.com --namespace foo; date
Wednesday, 01 February, 2023 01:32:19 PM PST
Checking cert-manager and related CRDs: cert-manager CRD not found. Installing cert-manager.
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
Waiting for cert-manager CRD registration...
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io condition met
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io condition met
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io unchanged
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io unchanged
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
Waiting for cert-manager pods...
deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met
pod/cert-manager-5b65cb968c-fshbm condition met
pod/cert-manager-cainjector-56b88bcdf7-25ft6 condition met
pod/cert-manager-webhook-c784c79c7-72wdd condition met
Installing cert-manager's self-signed issuer: .Done.
clusterissuer.cert-manager.io/default-issuer created
Checking secret-agent operator and related CRDs: secret-agent CRD not found. Installing secret-agent.
namespace/secret-agent-system created
customresourcedefinition.apiextensions.k8s.io/secretagentconfigurations.secret-agent.secrets.forgerock.io created
serviceaccount/secret-agent-controller-manager created
role.rbac.authorization.k8s.io/secret-agent-leader-election-role created
clusterrole.rbac.authorization.k8s.io/secret-agent-manager-role created
clusterrole.rbac.authorization.k8s.io/secret-agent-metrics-reader created
clusterrole.rbac.authorization.k8s.io/secret-agent-proxy-role created
rolebinding.rbac.authorization.k8s.io/secret-agent-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/secret-agent-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/secret-agent-proxy-rolebinding created
configmap/secret-agent-manager-config created
service/secret-agent-controller-manager-metrics-service created
service/secret-agent-webhook-service created
deployment.apps/secret-agent-controller-manager created
mutatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-validating-webhook-configuration created
Waiting for secret agent operator...
customresourcedefinition.apiextensions.k8s.io/secretagentconfigurations.secret-agent.secrets.forgerock.io condition met
deployment.apps/secret-agent-controller-manager condition met
pod/secret-agent-controller-manager-75c755487b-qvdr8 condition met
Checking ds-operator and related CRDs: ds-operator CRD not found. Installing ds-operator.
namespace/fr-system created
customresourcedefinition.apiextensions.k8s.io/directorybackups.directory.forgerock.io created
customresourcedefinition.apiextensions.k8s.io/directoryrestores.directory.forgerock.io created
customresourcedefinition.apiextensions.k8s.io/directoryservices.directory.forgerock.io created
role.rbac.authorization.k8s.io/ds-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/ds-operator-directorybackup-editor-role created
clusterrole.rbac.authorization.k8s.io/ds-operator-directoryrestore-editor-role created
clusterrole.rbac.authorization.k8s.io/ds-operator-directoryservice-editor-role created
clusterrole.rbac.authorization.k8s.io/ds-operator-manager-role created
rolebinding.rbac.authorization.k8s.io/ds-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/ds-operator-directorybackup-editor-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/ds-operator-directoryrestore-editor-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/ds-operator-directoryservice-editor-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/ds-operator-manager-rolebinding created
deployment.apps/ds-operator-ds-operator created
Waiting for ds-operator...
customresourcedefinition.apiextensions.k8s.io/directoryservices.directory.forgerock.io condition met
deployment.apps/ds-operator-ds-operator condition met
Installing component(s): ['all'] platform: "cdk" in namespace: "foo".
Deploying base.yaml. This is a one time activity.
configmap/dev-utils created
configmap/platform-config created
ingress.networking.k8s.io/forgerock created
ingress.networking.k8s.io/ig created
certificate.cert-manager.io/ds-master-cert created
certificate.cert-manager.io/ds-ssl-cert created
issuer.cert-manager.io/selfsigned-issuer created
secretagentconfiguration.secret-agent.secrets.forgerock.io/forgerock-sac created
Deploying ds.yaml. This includes all directory resources.
directoryservice.directory.forgerock.io/ds-idrepo created
Waiting for DS deployment. This can take a few minutes. First installation takes longer.
Waiting for statefulset "ds-idrepo" to exist in the cluster: done
Waiting for 1 pods to be ready...
partitioned roll out complete: 1 new pods have been updated...
Waiting for Service Account Password Update: ...........................................................................done
Cleaning up amster components.
Deploying apps.
configmap/amster-files created
configmap/idm created
configmap/idm-logging-properties created
service/am created
service/idm created
deployment.apps/am created
deployment.apps/idm created
job.batch/amster created
Waiting for AM deployment. This can take a few minutes. First installation takes longer.
Waiting for deployment "am" to exist in the cluster: done
deployment.apps/am condition met
configmap/amster-retain created
Waiting for amster job to complete. This can take several minutes.
Waiting for job "amster" to exist in the cluster: done
job.batch/amster condition met
Waiting for IDM deployment. This can take a few minutes. First installation takes longer.
Waiting for deployment "idm" to exist in the cluster: done
pod/idm-5d5d4595f8-fkn2c condition met
Deploying UI.
service/admin-ui created
service/end-user-ui created
service/login-ui created
deployment.apps/admin-ui created
deployment.apps/end-user-ui created
deployment.apps/login-ui created
Waiting for K8s secrets.
Waiting for secret "am-env-secrets" to exist in the cluster: done
Waiting for secret "idm-env-secrets" to exist in the cluster: done
Waiting for secret "ds-passwords" to exist in the cluster: done
Waiting for secret "ds-env-secrets" to exist in the cluster: done
Relevant passwords:
p0OHct9MfKQZaHQ387G1YMIl (amadmin user)
fuWf05niiX2vnEXMOB9iqAcOiEvk7pBI (uid=admin user)
hHLKV3GdEF3aVJJzZagozimF4o0s6Ys9 (App str svc acct (uid=am-config,ou=admins,ou=am-config))
zoD7cqKIPQqnYQtkp78fjRT3LixesvIH (CTS svc acct (uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens))
hoslrFMYTQX5OKaRQTKRbWnJ3ZuHPcWR (ID repo svc acct (uid=am-identity-bind-account,ou=admins,ou=identities))
Relevant URLs:
https://cdk.example.com/platform
https://cdk.example.com/admin
https://cdk.example.com/am
https://cdk.example.com/enduser
Enjoy your deployment!
Wednesday, 01 February, 2023 01:39:55 PM PST@luiscvega Yeah, it's self-contained. The forgeops install command checks to se if additional services the deployment relies on (like cert-manager and a couple of operators) are already there, and. if they are not, it installs them.
I'm reasonably certain that pre-loading the images fixed your problem, and I'm thinking of adding the docker pulls to the cdk-minikube command. If you have an even somewhat slow internet connection, long docker pulls can cause timeouts in forgeops install.
dgoldssfo
commented
1 year ago @dhruvdave-iv I'll have another look if you can send me more details about your deployment environment, the stuff I suggested earlier. It's much easier to troubleshoot these kinds of problems if you can send me the debug-logs output instead of just some logs (or just stdout).
Here's what we ask ForgeRock customers to send when they encounter a problem: https://backstage.forgerock.com/docs/forgeops/7.2/start/support.html#problem_reports_and_information_requests
Speaking of which, if you're a customer, you could file a support ticket and might get an answer more quickly.
dhruvdave-iv
commented
1 year ago @dgoldssfo ,
I did a clean install again and getting the same error.
Attached is the output of the forgeops/bin/debug-logs. cdk_install.zip
dhruv@ubuntu:~/forgeops/bin$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/jetstack/cert-manager-controller v1.11.0 ea6670b27349 3 weeks ago 61.9MB
quay.io/jetstack/cert-manager-webhook v1.11.0 a26ccff6c31b 3 weeks ago 47MB
quay.io/jetstack/cert-manager-cainjector v1.11.0 9604d46e1dda 3 weeks ago 39.6MB
kindest/kindnetd v20221004-44d545d1 d6e3e26021b6 4 months ago 61.8MB
forgerock/secret-agent v1.1.7 a10a8fd859e3 4 months ago 291MB
us-docker.pkg.dev/forgeops-public/images/ds 7.2.0 5e48dfe73203 7 months ago 311MB
gcr.io/forgerock-io/idm-cdk 7.2.0 33e04917e12b 7 months ago 457MB
gcr.io/forgerock-io/am-base 7.2.0 4544d9a8eb3c 7 months ago 831MB
gcr.io/forgerock-io/amster 7.2.0 facdc507968f 7 months ago 600MB
us-docker.pkg.dev/forgeops-public/images/ds-operator v0.2.5 17240e948d34 7 months ago 48.4MB
gcr.io/forgerock-io/ds-empty 7.2.0 8c486d924d5c 7 months ago 208MB
k8s.gcr.io/ingress-nginx/controller
dgoldssfo
commented
1 year ago Thanks for sending the diagnostics, @dhruvdave-iv. Our engineers are going to look at it - they've seen this problem before sporadically and are going to see if they can get to the bottom of it.
dhruvdave-iv
commented
1 year ago Thanks @dgoldssfo.
I noticed that the ds deployment is using us-docker.pkg.dev/forgeops-public/images/ds:7.2.0 image instead of gcr.io/forgerock-io/ds-empty:7.2.0 or gcr.io/forgerock-io/ds:7.2.0. And these images have different sha and size. Not sure if this has any bearing on this issue.
franklimstrand
commented
1 year ago Regarding the initial error, "Waiting for Service Account Password Update: ..........<function _waitfords at 0x7f0650f196c0> timed out after 600 secs", I struggled with this when I was testing the CDK on Linux (Ubuntu 22.04.1 LTS). The solution for me was to use the --driver kvm2 option to the minikube start command. Might be worth trying to see if that works for you.
andylow
commented
1 year ago Does anyone managed to install cdk locally via minikube or docker desktop kubernetes on Windows?
chicodeme
commented
1 year ago @andylow I have installed it on Windows 11 desktop with Docker Desktop. But you have to put a Linux VM down. Check this out at 3:15. https://community.forgerock.com/t/forgeops-how-to-spin-up-a-minikube-on-your-local-device/2875
The docs call this out here: https://backstage.forgerock.com/docs/forgeops/7.3/cdk/setup-cdk.html#windows-cdk
andylow
commented
1 year ago @chicodeme Thanks a lot, unfortunately, due to organization policy, I'm not allow to use any VM including Linux VM. But good news, I'm able to make a workaround without using official python script to deploy it in Windows. I have documented here : How to run ForgeOps in Windows using Docker Desktop Kubernetes
chicodeme
commented
1 year ago @andylow awesome!
I'm currently trying to follow the instructions in Cloud Developer's Kit (CDK) locally using
minikube. I am using the branch atreleases/7.2.0. I was able to start the minikube (./cdn-minikube start), able to create a namespace and added theminikube ipto/etc/hosts. However, I'm running into a few issues:Attempt 1: Install "all"
When I try to run
./foregeops install --cdk --fqdn cdk.example.com, it times out after 600 seconds. Kindly see the commands below:As you can see, the pod is running but it still times out.
Attempt 2: Staged install
So I attempted to do an install component by component. Following the instruction, I first installed
base, and thends. However, when I installam, I'm getting several of the following error from the logs:I have also tried to increase the CPUs to
4and RAM to12gbut I'm still running into the these issues.Is there a step I'm missing to run the CDK locally?