search

openshift-metal3 / dev-scripts

Scripts to automate development/test setup for openshift integration with https://github.com/metal3-io/
Apache License 2.0
93 stars 185 forks source link

baremetal-operator deployed by dev-scripts can't reach Ironic #318

Closed russellb closed 5 years ago

russellb commented 5 years ago

When we deploy the baremetal-operator using dev-scripts, it needs to be configured to talk to Ironic on the provisioning host until we have the baremetal-operator updated to run Ironic itself inside the cluster.

Right now it assumes it can reach Ironic on localhost, which fails when running the BMO as a pod.

{"level":"error","ts":1554831354.3558333,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"metalkube-baremetalhost-controller","request":"openshift-machine-api/openshift-worker-0","error":"phase validate access failed: failed to validate BMC access: failed to find existing host: failed to find node by name openshift-worker-0: Get http://localhost:6385/v1/nodes/openshift-worker-0: dial tcp [::1]:6385: connect: connection refused","errorVerbose":"Get http://localhost:6385/v1/nodes/openshift-worker-0: dial tcp [::1]:6385: connect: connection refused\nfailed to find node by name openshift-worker-0\ngithub.com/metalkube/baremetal-operator/pkg/provisioner/ironic.(*ironicProvisioner).findExistingHost\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/provisioner/ironic/ironic.go:145\ngithub.com/metalkube/baremetal-operator/pkg/provisioner/ironic.(*ironicProvisioner).ValidateManagementAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/provisioner/ironic/ironic.go:161\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).phaseValidateAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:403\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).(github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.phaseValidateAccess)-fm\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:268\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:283\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361\nfailed to find existing host\ngithub.com/metalkube/baremetal-operator/pkg/provisioner/ironic.(*ironicProvisioner).ValidateManagementAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/provisioner/ironic/ironic.go:163\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).phaseValidateAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:403\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).(github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.phaseValidateAccess)-fm\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:268\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:283\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361\nfailed to validate BMC access\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).phaseValidateAccess\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:405\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).(github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.phaseValidateAccess)-fm\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:268\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:283\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361\nphase validate access failed\ngithub.com/metalkube/baremetal-operator/pkg/controller/baremetalhost.(*ReconcileBareMetalHost).Reconcile\n\t/go/src/github.com/metalkube/baremetal-operator/pkg/controller/baremetalhost/baremetalhost_controller.go:285\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361","stacktrace":"github.com/metalkube/baremetal-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/metalkube/baremetal-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
dhellmann commented 5 years ago

There isn't currently a configuration option for this, because we didn't think we were going to need one.

russellb commented 5 years ago

Yeah, I was thinking that we need a workaround until ironic is running in the baremetal-operator pod

russellb commented 5 years ago

workaround here: https://github.com/metalkube/baremetal-operator/pull/127