CentOS 8 Stream Cobbler Reposync Failed

[manoj0772]

I am trying to set up Cobbler on CentOS 8 Stream to sync packages and create local mirrors but i am getting following below error :-

[manp1@psappl165 ~]$ sudo cobbler reposync --only CentOS-8-Stream-BaseOS-x86_64
task started: 2022-01-19_151043_reposync
task started (id=Reposync, time=Wed Jan 19 15:10:43 2022)
running python triggers from /var/lib/cobbler/triggers/task/reposync/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/reposync/pre/*
shell triggers finished successfully
hello, reposync
run, reposync, run!
creating: /var/www/cobbler/repo_mirror/CentOS-8-Stream-BaseOS-x86_64/config.repo
creating: /var/www/cobbler/repo_mirror/CentOS-8-Stream-BaseOS-x86_64/.origin/CentOS-8-Stream-BaseOS-x86_64.repo
running: /usr/bin/dnf reposync -l -m -d --config=/var/www/cobbler/repo_mirror/CentOS-8-Stream-BaseOS-x86_64/.origin/CentOS-8-Stream-BaseOS-x86_64.repo --repoid=CentOS-8-Stream-BaseOS-x86_64 -p /var/www/cobbler/repo_mirror -a x86_64
received on stdout: usage: dnf [-c [config file]] [-q] [-v] [--version] [--installroot [path]]
           [--nodocs] [--noplugins] [--enableplugin [plugin]]
           [--disableplugin [plugin]] [--releasever RELEASEVER]
           [--setopt SETOPTS] [--skip-broken] [-h] [--allowerasing]
--nobest] [-C] [-R [minutes]] [-d [debug level]]
           [--debugsolver] [--showduplicates] [-e ERRORLEVEL] [--obsoletes]
           [--rpmverbosity [debug level name]] [-y] [--assumeno]
--repo [repo]]
--disable] [-x [package]] [--disableexcludes [repo]]
           [--repofrompath [repo,path]] [--noautoremove] [--nogpgcheck]
           [--color COLOR] [--refresh] [-4] [-6] [--destdir DESTDIR]
           [--downloadonly] [--comment COMMENT] [--bugfix] [--enhancement]
           [--newpackage] [--security] [--advisory ADVISORY] [--bz BUGZILLA]
           [--cve CVES] [--sec-severity {Critical,Important,Moderate,Low}]
           [--forcearch ARCH]

received on stderr: Command line error: argument -d/--debuglevel: expected one argument

Exception occurred: <class 'cobbler.cexceptions.CX'>
Exception value: 'cobbler reposync failed'
Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/utils.py", line 95, in die
    raise CX(msg)

Exception occurred: <class 'cobbler.cexceptions.CX'>
Exception value: 'cobbler reposync failed'
Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/actions/reposync.py", line 161, in run
    self.sync(repo)

  File "/usr/lib/python3.6/site-packages/cobbler/actions/reposync.py", line 204, in sync
    self.yum_sync(repo)

  File "/usr/lib/python3.6/site-packages/cobbler/actions/reposync.py", line 563, in yum_sync
    utils.die(self.logger, "cobbler reposync failed")

  File "/usr/lib/python3.6/site-packages/cobbler/utils.py", line 103, in die
    raise CX(msg)

reposync failed, tries left: 0
reposync failed, retry limit reached, skipping
running: chown -R root:apache /var/www/cobbler/repo_mirror/CentOS-8-Stream-BaseOS-x86_64
received on stdout:
received on stderr:
running: chmod -R 755 /var/www/cobbler/repo_mirror/CentOS-8-Stream-BaseOS-x86_64
received on stdout:
received on stderr:
Exception occurred: <class 'cobbler.cexceptions.CX'>
Exception value: 'overall reposync failed, at least one repo failed to synchronize'
Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/utils.py", line 95, in die
    raise CX(msg)

Exception occurred: <class 'cobbler.cexceptions.CX'>
Exception value: 'overall reposync failed, at least one repo failed to synchronize'
Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 98, in run
    rc = self._run(self)

  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 310, in runner
    name=name, nofail=nofail, logger=self.logger)

  File "/usr/lib/python3.6/site-packages/cobbler/api.py", line 1360, in reposync
    action_reposync.run(name)

  File "/usr/lib/python3.6/site-packages/cobbler/actions/reposync.py", line 189, in run
    utils.die(self.logger, "overall reposync failed, at least one repo failed to synchronize")

  File "/usr/lib/python3.6/site-packages/cobbler/utils.py", line 103, in die
    raise CX(msg)

!!! TASK FAILED !!!
[manp1@psappl165 ~]$

I believe i have all the cobbler packages it need installed.

I am running below version.

Installed Packages
Name         : cobbler
Version      : 3.2.2
Release      : 6.module_el8+13609+df89ec4a
Architecture : noarch
Size         : 2.4 M
Source       : cobbler-3.2.2-6.module_el8+13609+df89ec4a.src.rpm
Repository   : @System
From repo    : epel-modular
Summary      : Boot server configurator
URL          : https://cobbler.github.io/
License      : GPLv2+
Description  : Cobbler is a network install server.  Cobbler supports PXE, ISO
             : virtualized installs, and re-installing existing Linux machines.
             : The last two modes use a helper tool, 'koan', that integrates with
             : cobbler.  There is also a web interface 'cobbler-web'.  Cobbler's
             : advanced features include importing distributions from DVDs and rsync
             : mirrors, kickstart templating, integrated yum mirroring, and built-in
             : DHCP/DNS Management.  Cobbler has a XML-RPC API for integration with
             : other applications.

[manp1@psappl165 ~]$ uname -a
Linux psappl165.jewelry.acn 4.18.0-358.el8.x86_64 #1 SMP Mon Jan 10 13:11:20 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[manp1@psappl165 ~]$ cat /etc/*release
CentOS Stream release 8
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 8
CentOS Stream release 8

This is from cobbler logs :-

2022-01-19T14:18:37 - ERROR | Settings file was not returned due to missing keys.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cobbler/settings.py", line 536, in read_settings_file
    validate_settings(filecontent)
  File "/usr/lib/python3.6/site-packages/cobbler/settings.py", line 438, in validate_settings
    return schema.validate(settings_content)
  File "/usr/lib/python3.6/site-packages/schema.py", line 409, in validate
    raise SchemaMissingKeyError(message, e)
schema.SchemaMissingKeyError: Missing keys: 'always_write_dhcp_entries', 'bind_chroot_path', 'bind_master', 'cheetah_import_whitelist', 'client_use_https', 'client_use_localhost', 'createrepo_flags', 'default_autoinstall', 'default_name_servers', 'default_name_servers_search', 'default_ownership', 'default_password_crypted', 'default_template_type', 'default_virt_bridge', 'default_virt_file_size', 'default_virt_ram', 'default_virt_type', 'enable_gpxe', 'enable_menu', 'http_port', 'include', 'kernel_options', 'ldap_anonymous_bind', 'ldap_base_dn', 'ldap_port', 'ldap_search_bind_dn', 'ldap_search_passwd', 'ldap_search_prefix', 'ldap_server', 'ldap_tls', 'ldap_tls_cacertfile', 'ldap_tls_certfile', 'ldap_tls_keyfile', 'manage_dhcp', 'manage_dns', 'manage_forward_zones', 'manage_reverse_zones', 'manage_rsync', 'manage_tftpd', 'mgmt_classes', 'mgmt_parameters', 'next_server', 'nopxe_with_triggers', 'power_management_default_type', 'proxy_url_ext', 'proxy_url_int', 'puppet_auto_setup', 'puppetca_path', 'pxe_just_once', 'redhat_management_key', 'redhat_management_permissive', 'redhat_management_server', 'register_new_installs', 'remove_old_puppet_certs_automatically', 'replicate_repo_rsync_options', 'replicate_rsync_options', 'reposync_flags', 'restart_dhcp', 'restart_dns', 'run_install_triggers', 'scm_push_script', 'scm_track_author', 'scm_track_enabled', 'scm_track_mode', 'serializer_pretty_json', 'server', 'sign_puppet_certs_automatically', 'tftpboot_location', 'virt_auto_boot', 'webdir', 'webdir_whitelist', 'xmlrpc_port', 'yum_distro_priority', 'yum_post_install_mirror', 'yumdownloader_flags'
2022-01-19T14:18:37 - DEBUG | The settings to read were: "{'allow_duplicate_hostnames': False, 'allow_duplicate_ips': False, 'allow_duplicate_macs': False, 'allow_dynamic_settings': False, 'anamon_enabled': False, 'authn_pam_service': 'login', 'auth_token_expiration': 3600, 'autoinstall_snippets_dir': '/var/lib/cobbler/snippets', 'autoinstall_templates_dir': '/var/lib/cobbler/templates', 'boot_loader_conf_template_dir': '/etc/cobbler/boot_loader_conf', 'build_reporting_enabled': False, 'build_reporting_sender': '', 'build_reporting_email': ['root@localhost'], 'build_reporting_smtp_server': 'localhost', 'build_reporting_subject': '', 'build_reporting_ignorelist': [], 'cache_enabled': True, 'reposync_rsync_flags': '-rltDv --copy-unsafe-links'}"

Any idea please ?

[SchoolGuy]

So I see two problems here:

1. The obvious error that the settings file is invalid at the moment. This can easily fixed by taking the recommended actions from the SchemaMissingKeyError and adding the missing keys. If this doesn't help we have found a separate bu.
2. The problem teased in the issue title. This is due to wrong usage of the flags from reposync. Some of those flags can be altered through the settings. Thus I would like to continue the investigation of this problem once the first point is fixed in your environment.

[manoj0772]

Hello,

Thank you for your response.

Do you mean making changes to /etc/cobbler/settings.yaml file ?

This is what i see for reposync flags in settings.yaml file.

[root@psappl165 manp1]# grep reposync /etc/cobbler/settings.yaml
reposync_rsync_flags: "-rltDv --copy-unsafe-links"
[root@psappl165 manpt1]#

From the output its trying to run this command :-

running: /usr/bin/dnf reposync -l -m -d --config=/var/www/cobbler/repo_mirror/CentOS-8-Stream-BaseOS-x86_64/.origin/CentOS-8-Stream-BaseOS-x86_64.repo --repoid=CentOS-8-Stream-BaseOS-x86_64 -p /var/www/cobbler/repo_mirror -a x86_64

I tested it manually running on the console and i was able to run above command after removing flags -l and -d. Do you know of correct flags needs to be updated in settings file ? Python errors are bit confusing for me. Thank you.

[SchoolGuy]

So in regard to the first error:

You need to add the following keys to your settings.yaml file in order for Cobbler to start correctly:

'always_write_dhcp_entries', 'bind_chroot_path', 'bind_master', 'cheetah_import_whitelist', 'client_use_https', 'client_use_localhost', 'createrepo_flags', 'default_autoinstall', 'default_name_servers', 'default_name_servers_search', 'default_ownership', 'default_password_crypted', 'default_template_type', 'default_virt_bridge', 'default_virt_file_size', 'default_virt_ram', 'default_virt_type', 'enable_gpxe', 'enable_menu', 'http_port', 'include', 'kernel_options', 'ldap_anonymous_bind', 'ldap_base_dn', 'ldap_port', 'ldap_search_bind_dn', 'ldap_search_passwd', 'ldap_search_prefix', 'ldap_server', 'ldap_tls', 'ldap_tls_cacertfile', 'ldap_tls_certfile', 'ldap_tls_keyfile', 'manage_dhcp', 'manage_dns', 'manage_forward_zones', 'manage_reverse_zones', 'manage_rsync', 'manage_tftpd', 'mgmt_classes', 'mgmt_parameters', 'next_server', 'nopxe_with_triggers', 'power_management_default_type', 'proxy_url_ext', 'proxy_url_int', 'puppet_auto_setup', 'puppetca_path', 'pxe_just_once', 'redhat_management_key', 'redhat_management_permissive', 'redhat_management_server', 'register_new_installs', 'remove_old_puppet_certs_automatically', 'replicate_repo_rsync_options', 'replicate_rsync_options', 'reposync_flags', 'restart_dhcp', 'restart_dns', 'run_install_triggers', 'scm_push_script', 'scm_track_author', 'scm_track_enabled', 'scm_track_mode', 'serializer_pretty_json', 'server', 'sign_puppet_certs_automatically', 'tftpboot_location', 'virt_auto_boot', 'webdir', 'webdir_whitelist', 'xmlrpc_port', 'yum_distro_priority', 'yum_post_install_mirror', 'yumdownloader_flags'

The flags you listed are for the rsync part of Cobbler. That is not the flags we need to edit. Maybe I was wrong and the new version of dnf changed the behavior of reposync which would mean we need to adjust our code...

[manoj0772]

Thanks. When i am looking at C7 cobbler server which is working there is a flag for "reposync_flags:" but same does not exist in newer version. So i added a new line on new version "reposync_flags: "-n" . By default on old version it is reposync_flags: "-l -n -d"

Anyways here is the content of settings.yaml file on new version . I added the last line.

[root@psappl165 ~]# grep -v '^ *#' /etc/cobbler/settings.yaml

allow_duplicate_hostnames: false

allow_duplicate_ips: false

allow_duplicate_macs: false

allow_dynamic_settings: false

anamon_enabled: false

authn_pam_service: "login"

auth_token_expiration: 3600

autoinstall_snippets_dir: /var/lib/cobbler/snippets
autoinstall_templates_dir: /var/lib/cobbler/templates

boot_loader_conf_template_dir: "/etc/cobbler/boot_loader_conf"

build_reporting_enabled: false
build_reporting_sender: ""
build_reporting_email: [ 'root@localhost' ]
build_reporting_smtp_server: "localhost"
build_reporting_subject: ""
build_reporting_ignorelist: []

cache_enabled: true
reposync_rsync_flags: "-rltDv --copy-unsafe-links"
reposync_flags: "-n"
[root@psappl165 ~]#

[SchoolGuy]

@manoj0772 So did you add the keys Cobbler says that are missing? And if yes do you still see the error in the Cobbler log or not reappearing?

[manoj0772]

I tried to add some keys and tested but it giving errors for certain keys for example :-

schema.SchemaError: Key 'default_name_servers' error: '' should be instance of 'list'

schema.SchemaError: Key 'cheetah_import_whitelist' error: '' should be instance of 'list' It did not error out for 1st 3 keys.

Is there a correct format value i should be using when adding these keys in settings.yaml. Should i try with updated version after 3.2 to see if these things are fixed ? Thank you

always_write_dhcp_entries: ""
bind_chroot_path: ""
bind_master: ""
#cheetah_import_whitelist: ""
client_use_https: ""
client_use_localhost: ""
createrepo_flags: ""
default_autoinstall: ""
default_name_servers: ""
default_name_servers_search: ""
default_ownership: ""
default_password_crypted: ""
default_template_type: ""
yumdownloader_flags: ""

[SchoolGuy]

@manoj0772 Here is an example of a valid Cobbler configuration file from master: https://github.com/cobbler/cobbler/blob/master/config/cobbler/settings.yaml

Since you are on 3.2.2 I would have a look here: https://github.com/cobbler/cobbler/blob/108287a3060a52cbf9c9f11a4959b8c4563d2bca/config/cobbler/settings.yaml

Please change all values to valid and reasonable values for your environment.

[SchoolGuy]

In regard to reposync it could be that the following issue helps: https://github.com/cobbler/cobbler/issues/2519

[manoj0772]

I think i am good now. I am not getting any missing key errors.

What i did is replace /etc/cobbler/settings.yaml file from here https://github.com/cobbler/cobbler/blob/108287a3060a52cbf9c9f11a4959b8c4563d2bca/config/cobbler/settings.yaml and then updated reposync flag with this reposync_flags: "-n --delete" .

Thank you for your help.

[SchoolGuy]

Okay so the action item for this is that we update the flags also per default... So we can't close this one for now.

[markgyver]

Thank you so much. I also had problems running reposync on my cobbler installation, running on RockyLinux 8, and I got it solved following your recommendations.