cobbler / cobbler

Cobbler is a versatile Linux deployment server
https://cobbler.github.io
GNU General Public License v2.0
2.63k stars 651 forks source link

Cobbler Import fails #2805

Closed duanyz1201 closed 3 years ago

duanyz1201 commented 3 years ago

Describe the bug

Attempting to import locally mounted media fails with complaints about signature imports:

root@cobbler3.2.2 ~#cobbler import --name=centos7.9 --arch=x86_64 --path=/mnt
task started: 2021-09-23_213148_import
task started (id=Media import, time=Thu Sep 23 21:31:48 2021)
running python triggers from /var/lib/cobbler/triggers/task/import/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/import/pre/*
shell triggers finished successfully
import_tree; ['/mnt', 'centos7.9', None, None, None]
importing from a network location, running rsync to fetch the files first
mkdir: /var/www/cobbler/distro_mirror/centos7.9-x86_64
running: rsync -a  '/mnt/' /var/www/cobbler/distro_mirror/centos7.9-x86_64 --progress
received on stdout: sending incremental file list
received on stderr: 
Found a candidate signature: breed=redhat, version=rhel6
Found a candidate signature: breed=redhat, version=rhel7
Found a candidate signature: breed=redhat, version=ovz7
Found a candidate signature: breed=redhat, version=fedora16
Found a candidate signature: breed=redhat, version=fedora17
Found a candidate signature: breed=redhat, version=fedora18
Found a candidate signature: breed=redhat, version=fedora19
Found a candidate signature: breed=redhat, version=fedora20
Found a candidate signature: breed=redhat, version=fedora21
Found a candidate signature: breed=redhat, version=fedora22
Found a candidate signature: breed=redhat, version=fedora23
Found a candidate signature: breed=redhat, version=fedora24
Found a candidate signature: breed=redhat, version=fedora25
Found a candidate signature: breed=redhat, version=fedora26
Found a candidate signature: breed=redhat, version=fedora27
Found a candidate signature: breed=redhat, version=fedora28
Found a candidate signature: breed=redhat, version=fedora29
Found a candidate signature: breed=redhat, version=fedora30
Found a candidate signature: breed=redhat, version=fedora31
Found a candidate signature: breed=redhat, version=fedora32
Found a candidate signature: breed=redhat, version=fedora33
Found a candidate signature: breed=redhat, version=fedora34
Found a candidate signature: breed=redhat, version=cloudlinux6
Found a candidate signature: breed=suse, version=opensuse15.0
Found a candidate signature: breed=suse, version=opensuse15.1
Found a candidate signature: breed=suse, version=opensuse15.2
Found a candidate signature: breed=suse, version=opensuse15.3
Found a candidate signature: breed=suse, version=sles15generic
Exception occurred: <class 'KeyError'>
Exception value: 'signatures'
Exception Info:
!!! TASK FAILED !!!

Expected behavior

TASK COMPLETE

Cobbler version

root@cobbler3.2.2 ~#cobbler version 
Cobbler 3.3.0
  source: ?, ?
  build time: Thu Sep 23 12:28:42 2021

Operating system

root@cobbler3.2.2 ~#cat /etc/centos-release CentOS Linux release 8.4.2105

Cobbler log

[2021-09-23_215415_import] 2021-09-23T21:54:17 - INFO | Exception occurred: <class 'KeyError'>
[2021-09-23_215415_import] 2021-09-23T21:54:17 - INFO | Exception value: 'signatures'
[2021-09-23_215415_import] 2021-09-23T21:54:17 - INFO | Exception Info:
  File "/usr/local/lib/python3.6/site-packages/cobbler/remote.py", line 99, in run
    rc = self._run(self)

  File "/usr/local/lib/python3.6/site-packages/cobbler/remote.py", line 313, in runner
    self.options.get("os_version", None),

  File "/usr/local/lib/python3.6/site-packages/cobbler/api.py", line 1598, in import_tree
    import_module.run(path, mirror_name, network_root, autoinstall_file, arch, breed, os_version)

  File "/usr/local/lib/python3.6/site-packages/cobbler/modules/managers/import_signatures.py", line 189, in run
    self.signature = self.scan_signatures()

  File "/usr/local/lib/python3.6/site-packages/cobbler/modules/managers/import_signatures.py", line 286, in scan_signatures
    for sig in sigdata["breeds"][breed][version]["signatures"]:

[2021-09-23_215415_import] 2021-09-23T21:54:17 - ERROR | ### TASK FAILED ###

Screenshots

Additional information

SchoolGuy commented 3 years ago

I am confused. Is it Cobbler 3.2.2 or is it Cobbler 3.3.0?

duanyz1201 commented 3 years ago

Cobbler 3.3.0

duanyz1201 commented 3 years ago

signature reload error

root@cobbler-3.3.0 ~#cobbler signature update
task started: 2021-09-24_184952_sigupdate
task started (id=Updating Signatures, time=Fri Sep 24 18:49:52 2021)
running python triggers from /var/lib/cobbler/triggers/task/sigupdate/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/sigupdate/pre/*
shell triggers finished successfully
Successfully got file from https://cobbler.github.io/signatures/3.0.x/latest.json
*** TASK COMPLETE ***
root@cobbler-3.3.0 ~#cobbler signature reload
There was an error loading the signature data in .
Please check the JSON file or run 'cobbler signature update'.
hveini commented 3 years ago

I got the same error importing Centos 7.9. Current signature for rhel7 is:

[root@localhost cobbler]# cobbler --version
Cobbler 3.3.0
  source: ?, ?
  build time: Sun Oct 10 15:26:12 2021
[root@localhost cobbler]# grep rhel7 /var/lib/cobbler/distro_signatures.json -A6
      "rhel7": {
        "signatures": [
          "Packages"
        ],
        "version_file": "(redhat|sl|slf|centos|oraclelinux|vzlinux)-release-(?!notes)([\\w]*-)*7(Server)*[\\.-]+(.*)\\.rpm",
        "version_file_regex": null,
        "kernel_arch": "kernel-(.*).rpm",
[root@localhost cobbler]# cat /var/lib/cobbler/distro_signatures.json|grep -E '"version_file_regex": (""|null)'|wc -l
61

--> it does not have the "version_file_regex". Same, with 61 signatures.

Double checking from the code "cobbler/modules/managers/import_signatures.py / “scan_signatures(self) -function:

                                if f_re.match(fname):
                                    # if the version file regex exists, we use it to scan the contents of the target
                                    # version file to ensure it's the right version
                                    if sigdata["breeds"][breed][version]["version_file_regex"]:
                                        vf_re = re.compile(sigdata["breeds"][breed][version]["version_file_regex"])
                                        vf_lines = self.get_file_lines(os.path.join(root, fname))
                                        for line in vf_lines:
                                            if vf_re.match(line):
                                                self.logger.debug("Found a matching signature: breed=%s, version=%s",
                                                                  breed, version)
                                                if not self.breed:
                                                    self.breed = breed
                                                if not self.os_version:
                                                    self.os_version = version
                                                if not self.autoinstall_file:
                                                    self.autoinstall_file = sigdata["breeds"][breed][version]["default_autoinstall"]
                                                self.pkgdir = pkgdir
                                                return sigdata["breeds"][breed][version]
                        break
        return None

--> for "return sigdata", it always requires there need to be “version_file_regex”, and also regex-match need to be done.

I replaced the function from version 3.2.0, and was able to import Centos 7.9. Same part of the code:

                                if f_re.match(fname):
                                    # if the version file regex exists, we use it
                                    # to scan the contents of the target version file
                                    # to ensure it's the right version
                                    if sigdata["breeds"][breed][version]["version_file_regex"]:
                                        vf_re = re.compile(sigdata["breeds"][breed][version]["version_file_regex"])
                                        vf_lines = self.get_file_lines(os.path.join(root, fname))
                                        for line in vf_lines:
                                            if vf_re.match(line):
                                                break
                                        else:
                                            continue
                                    self.logger.debug(
                                        "Found a matching signature: breed=%s, version=%s" % (breed, version))
                                    if not self.breed:
                                        self.breed = breed
                                    if not self.os_version:
                                        self.os_version = version
                                    if not self.autoinstall_file:
                                        self.autoinstall_file = sigdata["breeds"][breed][version]["default_autoinstall"]
                                    self.pkgdir = pkgdir
                                    return sigdata["breeds"][breed][version]
        return None

--> only checks the version file regex-match, if "version_file_regex" exists.

I think the function was changed by #2767

tpw56j commented 3 years ago

@hveini Sorry, 2 bugs and both were introduced by me.