bareMetalHost CR is stuck in inspecting state.

metal3-io / baremetal-operator

Bare metal host provisioning integration for Kubernetes

Apache License 2.0

591 stars 254 forks source link

bareMetalHost CR is stuck in inspecting state. #1284

Closed iamvikaskumar closed 1 year ago

iamvikaskumar commented 1 year ago

What steps did you take and what happened: I installed BareMetalOperator using the script : https://github.com/metal3-io/baremetal-operator/blob/main/tools/deploy.sh. Everything comes up fine. But when I create a BMH instance, it is always tuck in inspecting state.

What did you expect to happen: Expected BMH instance to be in ready/available state.

Anything else you would like to add: After debugging I can see that ironic inspector is not able to find the node which ironic created.

below is the error snippet from baremetal-operator-ironic-65d86bf654-lx6hn pod ... Returning error to client: Could not find node d45ce2e2-d4e3-47d2-9468-aa3d86e5162a in cache error_response /usr/lib/python3.9/site-packages/ironic_inspector/main.py:139 ...

Environment:

Baremetal Operator version:
Environment (metal3-dev-env or other):

/kind bug

Rozzii commented 1 year ago

I would like to ask you to provide the logs (all if possible) from the Ironic pod, the manifest of the Ironic pod, the list of environment variables that you have used with the deploy.sh. /triage needs-information

iamvikaskumar commented 1 year ago

I switched from ubuntu 20.04 to ubuntu 18.04 and it worked.

sekharvajjula commented 1 year ago

I am facing similar issue with the latest IPA initramfs image (quay.io/metal3-io/ironic-ipa-downloader). Introspection is stuck and results in failure. Below are the issues observed:

IPA service logs not shown on console.
ERROR ironic_python_agent.utils [-] failed to run hardware-detect utility: [Errno 2] No such file or directory: 'hardware-detect': FileNotFoundError: [Errno 2] No such file or directory: 'hardware-detect'

Using sshkey, I managed to fetch ironic-python-agent service logs. Please find attached. ironic-python-agent_new.txt

dtantsur commented 1 year ago

@sekharvajjula you need to update your ironic-image to remove the dependency on hardware-detect.

sekharvajjula commented 1 year ago

I was using upstream images (IPA Kernel and initramfs). These were downloaded by /usr/local/bin/get-resource.sh in quay.io/metal3-io/ironic-ipa-downloader. When I reverted back to quay.io/metal3-io/ironic-ipa-downloader:capm3-v0.4.1, the issue is resolved.

@dtantsur when you suggested to update ironic-image, did you mean IPA-initramfs?

Rozzii commented 1 year ago

@sekharvajjula , @dtantsur meant this: https://github.com/metal3-io/ironic-image and images built from that repo are here https://quay.io/repository/metal3-io/ironic?tab=tags

dtantsur commented 1 year ago

Correct.

The initial issue got resolved by switching to a different Ubuntu version (we cannot triage it further without more information), the second issue can be resolved by updating the ironic image. Closing for now. Please open new bugs if you encounter similar issues.

sekharvajjula commented 11 months ago

I used IPA image from https://images.rdoproject.org/centos9/master/rdo_trunk/current-tripleo/ironic-python-agent.tar with the help of variable "IPA_BASEURI" in ironic configmap. That solved the issue.