openvinotoolkit / operator

OpenVINO operator for OpenShift and Kubernetes
Apache License 2.0
13 stars 9 forks source link

Openvino notebook build intermittently fails #95

Open andrei-kochin opened 10 months ago

andrei-kochin commented 10 months ago

Describe the bug

Description of problem:

While testing ISVs on RHODS v1.29 RC 2 I got stuck while building an openvino image in the cluster. I keep getting this error in my cluster:

You can remove cached packages by executing 'yum clean packages'. Error: Transaction test error: package nodejs-2:14.18.1-1nodesource.x86_64 does not verify: no digest

error: build error: building at STEP "RUN curl -sL https:/...ec-severity=Moderate": while running runtime: exit status 1

Full logs: openvino-notebooks-v2022.3-1-undefined.log

I tried in other clusters without being able to reproduce it, however as I faced it, it could be faced by our customers too.

This issue may be out-of-scope for RHODS, but I think we should investigate and report to Intel if something must be changed

Prerequisites (if any, like setup, operators/versions): Steps to Reproduce Install OpenVINO operator (either marketplace or certified) deploy OpenVINO notebook with default settings from OCP Console > Operators > Installed Operators > Openvino toolkit > Notebook (ns: redhat-ods-applications) go to OCP Console > Builds > Builds check the build status Actual results:

build fails

Expected results:

build succeeds

Reproducibility (Always/Intermittent/Only Once):

Intermittent

Build Details:

The bug is still relevant on RHODS1.34 with OCP 4.12 FIPS enabled

https://github.com/openvinotoolkit/openvino_notebooks/files/13417038/openvino-notebooks-v2022.3-1-undefined.log

Transfer of https://github.com/openvinotoolkit/openvino_notebooks/issues/1398

dtrawins commented 4 months ago

The build operation was actually successful but the error was coming from the push to the local registry. Most likely this is related to the cluster issues like insufficient storage or some other problems from registry container.