search

uyuni-project / uyuni

Source code for Uyuni
https://www.uyuni-project.org/
GNU General Public License v2.0
434 stars 181 forks source link

Random Internal Server Error when using Provisioning > Power Management #8012

Open Bischoff opened 11 months ago

Bischoff commented 11 months ago

Problem description

(seen on Uyuni, but was reported in the past against SUSE Manager 4.3)

How to reproduce

Triggered by test suite in the feature "Redfish Power Management" scenario "Trigger Redfish functions".

I was able to reproduce manually, but not all the times.

What happens

After pressing one of the Power Management buttons, a screen "Internal Server Error" is displayed.

The exact error in the /etc/rhn/rhn_web_ui log file changes from time to time:

Caused by: org.cobbler.XmlRpcException: XmlRpcException calling cobbler.
(...)
Caused by: redstone.xmlrpc.XmlRpcFault: <class 'FileNotFoundError'>:[Errno 2] No such file or directory:
 '/srv/tftpboot/grub/i386-pc/efiemu.mod'

or:

Caused by: org.cobbler.XmlRpcException: XmlRpcException calling cobbler.
(...)
Caused by: redstone.xmlrpc.XmlRpcFault: <class 'FileNotFoundError'>:[Errno 2] No such file or directory:
 '/srv/tftpboot/grub/i386-pc/reboot.mod'

or:

Caused by: org.cobbler.XmlRpcException: XmlRpcException calling cobbler.
(...)
Caused by: redstone.xmlrpc.XmlRpcFault: <class 'FileNotFoundError'>:[Errno 2] No such file or directory:
 '/srv/tftpboot/grub/i386-efi/ufs1.mod'

or:

Caused by: org.cobbler.XmlRpcException: XmlRpcException calling cobbler.
(...)
Caused by: redstone.xmlrpc.XmlRpcFault: <class 'FileNotFoundError'>:[Errno 2] No such file or directory:
 '/srv/tftpboot/grub/x86_64-efi/usbserial_pl2303.mod'

When I inspect those "missing" files, they do exist.

When I look at /var/log/cobbler.log, I can see those files keep being transferred:

uploading /srv/tftpboot/grub/x86_64-efi/usbserial_pl2303.mod to all known proxies. Timeout: 15

Steps to reproduce

https://github.com/uyuni-project/uyuni/blob/master/testsuite/features/secondary/srv_power_management_redfish.feature#L34

Uyuni version

Nom                       : Uyuni-Server-release
Version                   : 2023.10-230900.209.2.uyuni3

Uyuni proxy version (if used)

Nom                       : Uyuni-Proxy-release
Version                   : 2023.10-230900.209.2.uyuni3

Useful logs

example failure:

javax.servlet.ServletException: org.cobbler.XmlRpcException: XmlRpcException calling cobbler.
        at org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:535) ~[struts.jar:1.2.9]
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:433) ~[struts.jar:1.2.9]
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:237) ~[struts.jar:1.2.9]
        at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:104) ~[rhn.jar:?]
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) ~[struts.jar:1.2.9]
        at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432) ~[struts.jar:1.2.9]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) ~[tomcat-servlet-4.0-api.jar:4.0.FR]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-servlet-4.0-api.jar:4.0.FR]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-websocket.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at com.redhat.rhn.frontend.servlets.SystemDetailsMessageFilter.doFilter(SystemDetailsMessageFilter.java:57) ~[rhn.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:99) ~[rhn.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142) ~[sitemesh.jar:?]
        at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58) ~[sitemesh.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:71) ~[rhn.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:99) ~[rhn.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:59) ~[rhn.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:98) ~[rhn.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[catalina.jar:9.0.82]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) ~[catalina.jar:9.0.82]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[catalina.jar:9.0.82]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670) ~[catalina.jar:9.0.82]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[catalina.jar:9.0.82]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) ~[catalina.jar:9.0.82]
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:424) ~[tomcat-coyote.jar:9.0.82]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-coyote.jar:9.0.82]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) ~[tomcat-coyote.jar:9.0.82]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794) ~[tomcat-coyote.jar:9.0.82]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-coyote.jar:9.0.82]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-util.jar:9.0.82]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-util.jar:9.0.82]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-util.jar:9.0.82]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.cobbler.XmlRpcException: XmlRpcException calling cobbler.
        at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:160) ~[rhn.jar:?]
        at org.cobbler.CobblerConnection.invokeTokenMethod(CobblerConnection.java:190) ~[rhn.jar:?]
        at org.cobbler.SystemRecord.invokeSave(SystemRecord.java:326) ~[rhn.jar:?]
        at org.cobbler.CobblerObject.save(CobblerObject.java:385) ~[rhn.jar:?]
        at com.redhat.rhn.manager.kickstart.cobbler.CobblerPowerSettingsUpdateCommand.store(CobblerPowerSettingsUpdateCommand.java:157) ~[rhn.jar:?]
        at com.redhat.rhn.frontend.action.kickstart.PowerManagementAction.execute(PowerManagementAction.java:106) ~[rhn.jar:?]
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) ~[struts.jar:1.2.9]
        ... 50 more
Caused by: redstone.xmlrpc.XmlRpcFault: <class 'FileNotFoundError'>:[Errno 2] No such file or directory: '/srv/tftpboot/grub/i386-pc/efiemu.mod'
        at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:444) ~[redstone-xmlrpc-client.jar:?]
        at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) ~[redstone-xmlrpc-client.jar:?]
        at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) ~[redstone-xmlrpc-client.jar:?]
        at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:157) ~[rhn.jar:?]
        at org.cobbler.CobblerConnection.invokeTokenMethod(CobblerConnection.java:190) ~[rhn.jar:?]
        at org.cobbler.SystemRecord.invokeSave(SystemRecord.java:326) ~[rhn.jar:?]
        at org.cobbler.CobblerObject.save(CobblerObject.java:385) ~[rhn.jar:?]
        at com.redhat.rhn.manager.kickstart.cobbler.CobblerPowerSettingsUpdateCommand.store(CobblerPowerSettingsUpdateCommand.java:157) ~[rhn.jar:?]
        at com.redhat.rhn.frontend.action.kickstart.PowerManagementAction.execute(PowerManagementAction.java:106) ~[rhn.jar:?]
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) ~[struts.jar:1.2.9]
        ... 50 more

Additional information

Likely cause

This looks like a race condition where Cobbler needs one of its files while it's transferring it at same time.

Possible fix

Is it possible to calm down Cobbler so it stops doing all those transfers all the time?

srbarrios commented 11 months ago

I doubt it's related... but just in case, we have this other bug reported https://bugzilla.suse.com/show_bug.cgi?id=1217237

srbarrios commented 11 months ago

@Bischoff have we a bugzilla report for this card? If not, I would assume we want one right?

Bischoff commented 11 months ago

I doubt it's related... but just in case, we have this other bug reported https://bugzilla.suse.com/show_bug.cgi?id=1217237

i think it's two different problems ; one is a packaging problem, and the other one is a race condition (from what i can see)

@Bischoff have we a bugzilla report for this card? If not, I would assume we want one right?

As it affected the Uyuni branch, I have been told to create a Uyuni issue instead of a bugzilla report.