Closed fsidoli closed 2 years ago
Further info, which may hopefully help.
cobblerd will not start. Log file showed the following error. Sure enough, 'cache_enabled' and 'reposync_rsync_flags' were not in the settings file.
2021-11-21T20:24:41 - ERROR | Settings file was not returned due to missing keys.
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/cobbler/settings.py", line 536, in read_settings_file
validate_settings(filecontent)
File "/usr/lib/python3.6/site-packages/cobbler/settings.py", line 438, in validate_settings
return schema.validate(settings_content)
File "/usr/lib/python3.6/site-packages/schema.py", line 409, in validate
raise SchemaMissingKeyError(message, e)
schema.SchemaMissingKeyError: Missing keys: 'cache_enabled', 'reposync_rsync_flags'
On investigation:
# yum history info cobbler
Updating Subscription Management repositories.
Transaction ID : 72
Begin time : Sun 17 Oct 2021 08:52:06 BST
Begin rpmdb : 1123:f2f45cc5d51fac4f9c010705bfd39a4c42436919 **
End time : Sun 17 Oct 2021 08:52:29 BST (23 seconds)
End rpmdb : 1125:80191553869f71145e77b230308077440fafac53
User : root <root>
Return-Code : Success
Releasever : 8
Command Line : upgrade
Comment :
Packages Altered:
<snip>
Upgrade cobbler-3.2.2-3.module_el8+13166+f291d70c.noarch @epel-modular
Upgraded cobbler-3.2.0-2.module_el8+10474+71f42bad.noarch @@System
Upgrade cobbler-web-3.2.2-3.module_el8+13166+f291d70c.noarch @epel-modular
Upgraded cobbler-web-3.2.0-2.module_el8+10474+71f42bad.noarch @@System
<snip>
Scriptlet output:
**1 warning: /var/lib/cobbler/distro_signatures.json created as /var/lib/cobbler/distro_signatures.json.rpmnew
2 warning: /etc/cobbler/settings saved as /etc/cobbler/settings.rpmsave
3 warning: migrating old settings to settings.yaml**
Warning 2 implies the cobbler settings were saved as settings.rpmsave, but this appears to be settings.yaml.rpmsave
# ls -lhtr /etc/cobbler/settings.*
-rw-r--r-- 1 root root 20K Oct 25 2020 /etc/cobbler/settings.rpmnew
-rw-r--r-- 1 root root 20K Oct 8 05:44 /etc/cobbler/settings.yaml.rpmnew
-rw-r--r-- 1 root root 19K Oct 17 08:52 /etc/cobbler/settings.yaml.foobar
-rw-r--r-- 1 root root 21K Nov 21 21:01 /etc/cobbler/settings.yaml
[after:
#cp -ap /etc/cobbler/settings.yaml /etc/cobbler/settings.yaml.foobar
#cp -ap /etc/cobbler/settings.yaml.rpmnew /etc/cobbler/settings.yaml
]
Interestingly, the "new" settings.yaml file (now called settings.yaml.foobar) appears to be using an older version of YAML compared to the "old" settings file I was using before the upgrade, so I'm not sure what was migrated across under warning 3 from the yum history dump. The two keys mentioned above were missing from the "new" settings.yaml file. I restored my original file and have removed the error above.
# head -n 5 /etc/cobbler/settings.yaml.foobar
---
# cobbler settings file
# restart cobblerd and run "cobbler sync" after making changes
# This config file is in YAML 1.0 format
# see http://yaml.org
# head -n 5 /etc/cobbler/settings.yaml
# Cobbler settings file
# Restart cobblerd and run "cobbler sync" after making changes.
# This config file is in YAML 1.2 format; see "http://yaml.org".
I can also confirm, having installed cobbler on another RHEL 8 box, that the settings file installed by default is like the /etc/cobbler/settings.yaml.rpmnew file that was apparently saved during the upgrade.
If I try to restart cobblerd and then check the status, I see the following message.
# systemctl -l status cobblerd
● cobblerd.service - Cobbler Helper Daemon
Loaded: loaded (/usr/lib/systemd/system/cobblerd.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Sun 2021-11-21 21:31:31 GMT; 20s ago
Process: 201183 ExecStartPost=/usr/bin/touch /usr/share/cobbler/web/cobbler.wsgi (code=exited, status=0/SUCCESS)
Process: 201182 ExecStart=/usr/bin/cobblerd -F (code=exited, status=0/SUCCESS)
Main PID: 201182 (code=exited, status=0/SUCCESS)
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: File "/usr/bin/cobblerd", line 75, in main
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: api = cobbler_api.CobblerAPI(is_cobblerd=True)
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: File "/usr/lib/python3.6/site-packages/cobbler/api.py", line 102, in __init__
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: self.deserialize()
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: File "/usr/lib/python3.6/site-packages/cobbler/api.py", line 1528, in deserialize
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: return self._collection_mgr.deserialize()
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: File "/usr/lib/python3.6/site-packages/cobbler/cobbler_collections/manager.py", line 202, in deserialize
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: % (collection.collection_type(), e)) from e
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk cobblerd[201182]: cobbler.cexceptions.CX: 'serializer: error loading collection distro: "Error with distro RHEL-8.2-x86_64 - kernel \'/var//www/cobbler/distro_mirror/RHEL-8.2-x86_64/images/pxeboot/vmlinuz\' not found". Check /etc/cobbler/modules.conf'
Nov 21 21:31:31 apollo.lcn.ucl.ac.uk systemd[1]: cobblerd.service: Succeeded.
I'm not sure what I'm looking for in /etc/cobbler/modules.conf
.
I think I have figured this out.
The mangled settings file that was dumped in place during the upgrade resulted in my distro_mirror directory being delete.
I have manually recreated these file so I could get cobbler back up and running. I'm now believe I need to reimport all my distros to undo the damage caused by the upgrade.
Can you attach a copy of your settings.yaml from before and after the update? Thanks.
Apologies for the weird formatting above. I'm not sure what went wrong.
Attached are the two files (with .txt extensions so I could upload).
The file called settings.yaml.foobar is what was left in place after the update and looks to be an old style config file.
Time stamps of the files on the server itself are shown below (settings.yaml having been edited last night by me).
~> ls -ltrh /etc/cobbler/settings.yaml* -rw-r--r-- 1 root root 20K Oct 8 05:44 /etc/cobbler/settings.yaml.rpmnew -rw-r--r-- 1 root root 19K Oct 17 08:52 /etc/cobbler/settings.yaml.foobar -rw-r--r-- 1 root root 21K Nov 21 21:01 /etc/cobbler/settings.yaml
The settings.yaml.rpmnew looks to the default settings file that gets installed on a new build. What I was left with was settings.yaml.foobar (renamed of course) which looks to be the format of an old cobbler config file - although I don't know how old as I've been running it for years.
My expectation might have been that the old settings file be moved to settings.yaml.rpmsave and that settings.yaml.rpmnew be moved to settings.yaml.
This didn't happen, so when I ran a 'cobbler sync' it was not able to interpret the config file correctly, and importantly the whitelist_dir, so nuked all my image files in /var/www/cobbler/distro_mirror. I had to recreate thi s manually last night for all my distros just to get cobbler restarted.
This morning I need to spend time re-importing all my distros to get the correct initrd and vmlinuz files in place.
Also, this is now the contents of /var/www/cobbler/ on my server
~> ls -l /var/www/cobbler/
total 0
drwxr-xr-x 7 root root 135 Nov 22 09:20 distro_mirror
drwxr-xr-x 8 root root 177 Nov 21 22:07 images
drwxr-xr-x 2 root root 6 Nov 17 17:02 rendered
On a fresh build
~> ls -l /var/www/cobbler/
total 36
drwxr-xr-x 3 root root 4096 Nov 21 20:48 distro_mirror
drwxr-xr-x 2 root root 4096 Oct 8 05:44 images
drwxr-xr-x 2 root root 4096 Oct 8 05:44 links
drwxr-xr-x 2 root root 4096 Oct 8 05:44 localmirror
drwxr-xr-x 2 root root 4096 Nov 21 20:48 misc
drwxr-xr-x 2 root root 4096 Oct 8 05:44 pub
drwxr-xr-x 2 root root 4096 Oct 8 05:44 rendered
drwxr-xr-x 2 root root 4096 Oct 8 05:44 repo_mirror
drwxr-xr-x 2 root root 4096 Nov 21 20:48 svc
What impact is this going to have?
@fsidoli I took the liberty of formatting your comments in a way I believe makes them more readable.
Regarding your issue: So far no idea but it seems @opoplawski is already helping, so I step back and fix other issues we have! If you need something feel free to ping me.
@SchoolGuy thank you. I realised last night that it's the '#' from my command line copy and paste that was doing this but never got around to sorting this out.
@fsidoli I would use back-ticks for code formatting because it is explicit and you don't have to mess with white space. Wrapping it in three back-ticks for a code-block and a single one for inline code does the trick for months now for me.
@fsidoli Any chance you have a copy of /etc/cobbler/settings from before the update?
Well, so much for my attempt at migrating the settings file to the new format :(
I find it surprising the when confronted with a settings.yaml file that is missing required keys that instead of exiting with a failure cobblerd simply continues running but without apparently having loaded any settings.
@opoplawski With the new format all of this should be automated. That is what we have the migrations for, so what were you trying to do?
@opoplawski, please see the attached file which I found in a backup dir in /var/lib/cobbler/backup/upgrade-20211017-085210 settings.txt
Hi guys,
I'm having an issue with my kickstart environment. I get a message during PXE boot to say that the kickstart can't be found, despite it being able to load the image file. I'm not sure where cblr/svc/op/ks/system lives on my cobbler server.
Thanks, Fab
@fsidoli It doesn't the file is generated on the fly. The thing you can do is try to get URL via a CURL.
@fsidoli Do you have any update on this issue? Or can we close this one?
Hi Enno,
I was hoping someone would have sorted out this bug and updated me. I have left cobbler untouched since this issue.
Essentially, a new format of the settings file was put in place by the update I did which broke my cobbler set up when I ran the sync.
It took me a few days to figure this was what had happened and was lucky enough to dig myself out of a hole because of the backups I had. I this I have updated the post to mention this, although it’s been a while since I last checked in.
Dr Fabrizio Sidoli | IT Systems Administrator London Centre for Nanotechnology (LCN) | london-nano.comhttps://london-nano.com/ University College London | 17-19 Gordon Street | London | WC1H 0AH Tel: +44 (0)20 7679 2869 | Ext: 32869 | Email: @.> / LCN IS @.>
From: Enno G. @.> Date: Wednesday, 23 March 2022 at 08:49 To: cobbler/cobbler @.> Cc: Sidoli, Fabrizio @.>, Mention @.> Subject: Re: [cobbler/cobbler] cobbler sync ignoring webdir_whitelist since upgrade to 3.2.2-3 (Issue #2839)
⚠ Caution: External sender
@fsidolihttps://github.com/fsidoli Do you have any update on this issue? Or can we close this one?
— Reply to this email directly, view it on GitHubhttps://github.com/cobbler/cobbler/issues/2839#issuecomment-1076101681, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AWRW6L62IPI2HXQTJBVLFJ3VBLLJRANCNFSM5II4A5EA. You are receiving this because you were mentioned.Message ID: @.***>
@fsidoli So as we explained earlier: This was sadly done by a downstream adjustment we weren't aware of upstream. So if the settings file is now fixed by you manually, all should work now. If not I need information on how to reproduce the bug. So far I don't see any upstream problems.
Hi Enno,
OK. If that’s the case then feel free to close this off.
I will try to update cobbler at some point this week and will reopen if I hit problems again.
Fab
Dr Fabrizio Sidoli | IT Systems Administrator London Centre for Nanotechnology (LCN) | london-nano.comhttps://london-nano.com/ University College London | 17-19 Gordon Street | London | WC1H 0AH Tel: +44 (0)20 7679 2869 | Ext: 32869 | Email: @.> / LCN IS @.>
From: Enno G. @.> Date: Monday, 28 March 2022 at 07:34 To: cobbler/cobbler @.> Cc: Sidoli, Fabrizio @.>, Mention @.> Subject: Re: [cobbler/cobbler] cobbler sync ignoring webdir_whitelist since upgrade to 3.2.2-3 (Issue #2839)
⚠ Caution: External sender
@fsidolihttps://github.com/fsidoli So as we explained earlier: This was sadly done by a downstream adjustment we weren't aware of upstream. So if the settings file is now fixed by you manually, all should work now. If not I need information on how to reproduce the bug. So far I don't see any upstream problems.
— Reply to this email directly, view it on GitHubhttps://github.com/cobbler/cobbler/issues/2839#issuecomment-1080255668, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AWRW6L5452WEDRPUT7O4AJTVCFHHXANCNFSM5II4A5EA. You are receiving this because you were mentioned.Message ID: @.***>
This morning I came to add a new machine to my cobbler config to find the following "File not found" errors below.
The last time I ran a cobbler sync was on the 11 & 13 Oct. Back then, cobbler ran without issues. It seems that cobbler was patched on Oct 17, as per the below. Since then, nothing seems to work.
After running the cobbler sync command above, the contents of
/var/lib/tftpboot/images/${distro}/
. and/var/www/cobbler/images/${distro}
are empty.I imported RHEL 8.5 today and can confirm that these directories get populated on import, but subsequently delete on running
cobbler sync
.I also cannot edit any profiles. My cobbler instance is completely broken as it stands.