search

uyuni-project / uyuni

Source code for Uyuni
https://www.uyuni-project.org/
GNU General Public License v2.0
428 stars 179 forks source link

Channel repo sync schedule change fails during automatic synchronization and deletes existing sync schedule #5615

Open santeri3700 opened 2 years ago

santeri3700 commented 2 years ago

Problem description

Channel repo sync schedule can't be changed while Taskomatic is (automatically) running a "repo-sync" task for the specific channel at the moment. \ Attempting to change the sync schedule will result in error and the previously configured schedule to be deleted/lost.

I would not expect the schedule change to fail and especially result in the previously set schedule to be deleted/lost.

Version of Uyuni Server and Proxy (if used)

Reproduced on two separate Uyuni Servers on versions 2022.05 and 2022.06 (updated from 2022.05). I do not know if this affects older versions.

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : Uyuni Server Stable
Name           : Uyuni-Server-release
Version        : 2022.06-183.9.uyuni2
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes (automatically)
Status         : up-to-date
Source package : Uyuni-Server-release-2022.06-183.9.uyuni2.src
Summary        : Uyuni Server
Description    : 
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Details about the issue

I discovered this issue about a week before opening this issue. I was trying to mass-update channel sync schedules via spacecmd and some channels repeatedly resulted in error. It later came apparent that the channels in question were being automatically synchronized at the moment by Taskomatic due to the daily schedule setting. \ These channels are both added via spacewalk-common-channels or manually via Web UI or spacecmd and function fine otherwise.

2022-06-29: I have so far only observed this issue on DEB repositories (Debian & Ubuntu), but this could be because the RPM repositories in my environment(s) are automatically synchronized earlier in the morning when I haven't done any tests. This affects RPM and DEB based channels.

How to reproduce

Logs and error messages

#Automatically scheduled sync is currently being run for the channel by Taskomatic. It is 11:09~ at this time.
$ spacecmd softwarechannel_listsyncschedule | grep 'ubuntu-1804-amd64-multiverse-security-uyuni'
INFO: Connected to http://localhost/rpc/api as admin
  123  ubuntu-1804-amd64-multiverse-security-uyuni 0 0 11 ? * *

#Attempting to change the sync schedule from 11:00 to 20:00.
$ spacecmd 'softwarechannel_setsyncschedule ubuntu-1804-amd64-multiverse-security-uyuni 0 0 20 ? * *'
INFO: Connected to http://localhost/rpc/api as admin
ERROR: redstone.xmlrpc.XmlRpcFault: Taskomatic is not available.

#Previous schedule change failed and the existing schedule (`0 0 11 ? * *`) was deleted without the new schedule being set.
$ spacecmd softwarechannel_listsyncschedule | grep 'ubuntu-1804-amd64-multiverse-security-uyuni'
INFO: Connected to http://localhost/rpc/api as admin
  123  ubuntu-1804-amd64-multiverse-security-uyuni

/var/log/rhn/rhn_taskomatic_daemon.log:

2022-06-29 11:09:06,311 [Thread-18] WARN  com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job repo-sync-1-123 failed to schedule.

/var/log/messages:

2022-06-29T11:09:06.328960+03:00 uyuni-server server[6462]: [Fatal Error] :-1:-1: Premature end of file.
2022-06-29T11:09:06.329935+03:00 uyuni-server server[6462]: 29-Jun-2022 11:01:52.629 WARNING [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-2] redstone.xmlrpc.XmlRpcDispatcher.writeError redstone.xmlrpc.XmlRpcFault: Taskomatic is not available.
Task Engine Runtime Status at the time: Task Id Task Name Start Time End Time Elapsed Time Status Data
85789 repo-sync 11:00 549 seconds running Ubuntu 18.04 LTS AMD64 Multiverse Security for Uyuni

The task ended successfully later and there were no new packages or patches/errata to be imported.

mbussolotto commented 2 years ago

Thanks for reporting @santeri3700 , if you have the chance it would be interesting to understand if it happens just for RPM (I guess so).

santeri3700 commented 2 years ago

@mbussolotto I just tested the issue with RPM based channels and got same results. So it doesn't seem to be specific to RPM or DEB based channels. \ Also, I think it is not actually necessary to have the "repo-sync" task start automatically, I just tested this with the "Sync now" button.

krax13 commented 1 year ago

Hi all,

I have been facing the same error message with the latest version of SUSE Manager (4.3.5). Manually repo sync not working on the Web UI, and the automatic mgr-sync-refresh schedule also run into errors.

The full error message in /var/log/rhn/rhn_taskomatic.log:

2023-06-09 14:20:53,031 [DefaultQuartzScheduler_Worker-1] DEBUG com.redhat.rhn.taskomatic.task.MgrSyncRefresh - Refreshing mgr-sync data
…
2023-06-09 14:21:43,943 [DefaultQuartzScheduler_Worker-1] DEBUG com.redhat.rhn.taskomatic.task.MgrSyncRefresh - Scheduling synchronization of all vendor channels
2023-06-09 14:21:43,986 [Thread-8] WARN  com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job single-repo-sync-bunch-1 failed to schedule.
2023-06-09 14:21:44,015 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.MgrSyncRefresh - Executing a task threw an exception: org.quartz.JobExecutionException
org.quartz.JobExecutionException: com.redhat.rhn.taskomatic.TaskomaticApiException: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:127) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:56) [rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:155) [rhn.jar:?]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.3.0.jar:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.0.jar:?]
Caused by: com.redhat.rhn.taskomatic.TaskomaticApiException: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
        at com.redhat.rhn.taskomatic.TaskomaticApi.invoke(TaskomaticApi.java:92) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.scheduleSingleRepoSync(TaskomaticApi.java:172) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:118) ~[rhn.jar:?]
        ... 4 more
Caused by: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
        at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:435) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:209) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.invoke(TaskomaticApi.java:89) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.scheduleSingleRepoSync(TaskomaticApi.java:172) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:118) ~[rhn.jar:?]
        ... 4 more
Caused by: org.xml.sax.SAXParseException: Premature end of file.
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) ~[xerces-j2.jar:?]
        at redstone.xmlrpc.XmlRpcParser.parse(XmlRpcParser.java:158) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:420) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:209) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.invoke(TaskomaticApi.java:89) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.scheduleSingleRepoSync(TaskomaticApi.java:172) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:118) ~[rhn.jar:?]
        ... 4 more

The custom and vendor channels sync are working via spacewalk-repo-sync but not manually from Web UI. Vendor channels sync are not working automatically.

Do you have any suggestion of this issue?

avshiliaev commented 1 year ago

@wweellddeerr can this one also be related to https://github.com/SUSE/spacewalk/issues/22195 ?