Differences between rhel8/centos8

[Klaas-]

Hi, I am seeing a few big differences in the setup of centos8 and rhel8. I am wondering if those could be re-aligned for a consistent usage experience for the two products.

a) RHEL8.2(payg, images created by microsoft) on Azure defaults the ephemeral disk to /mnt instead of /mnt/resource. I don't care either way but I would hope to have it the same for EL8 and clones. (see bug https://github.com/Azure/WALinuxAgent/issues/1971).

b) the firewall is disabled by default, RHEL payg defaults to an enabled firewall. I would very much have this aligned as well. Personal opinion: I would rather use firewalld than nsg on the azure level.

Greetings Klaas

[N3WWN]

Hi @Klaas- !

a) Thanks for referencing that WALA ticket! I just provided my input in that ticket to hopefully get the RHEL images to align with the CentOS images.

b) Can you tell me exactly which RHEL payg image has an enabled firewall? If you have the exact URN, that'd be awesome! I'll take a look at their firewall configuration.

Thanks!

-Rich

[Klaas-]

Hi Rich, I think it should be all of them, I am using this for my current tests, I've included the active rules, but it looks like a fairly default setup for RHEL with ssh/cockpit and some rules that have been added by WALinuxAgent.

$ curl -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance?api-version=2020-09-01"| python3 -m json.tool
[...]
            "imageReference": {
                "id": "",
                "offer": "RHEL",
                "publisher": "RedHat",
                "sku": "8-lvm-gen2",
                "version": "latest"
            },
[...]
$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
[...]

$ nft list ruleset
table ip filter {
        chain INPUT {
                type filter hook input priority filter; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority filter; policy accept;
        }
}
table ip6 filter {
        chain INPUT {
                type filter hook input priority filter; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority filter; policy accept;
        }
}
table bridge filter {
        chain INPUT {
                type filter hook input priority filter; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority filter; policy accept;
        }
}
table ip security {
        chain INPUT {
                type filter hook input priority 150; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority 150; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority 150; policy accept;
                meta l4proto tcp ip daddr 168.63.129.16 skuid 0 counter packets 654 bytes 182444 accept
                meta l4proto tcp ip daddr 168.63.129.16 ct state invalid,new counter packets 0 bytes 0 drop
        }
}
table ip raw {
        chain PREROUTING {
                type filter hook prerouting priority raw; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority raw; policy accept;
        }
}
table ip mangle {
        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
        }

        chain INPUT {
                type filter hook input priority mangle; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority mangle; policy accept;
        }

        chain OUTPUT {
                type route hook output priority mangle; policy accept;
        }

        chain POSTROUTING {
                type filter hook postrouting priority mangle; policy accept;
        }
}
table ip nat {
        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
        }

        chain INPUT {
                type nat hook input priority 100; policy accept;
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
        }
}
table ip6 security {
        chain INPUT {
                type filter hook input priority 150; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority 150; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority 150; policy accept;
        }
}
table ip6 raw {
        chain PREROUTING {
                type filter hook prerouting priority raw; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority raw; policy accept;
        }
}
table ip6 mangle {
        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
        }

        chain INPUT {
                type filter hook input priority mangle; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority mangle; policy accept;
        }

        chain OUTPUT {
                type route hook output priority mangle; policy accept;
        }

        chain POSTROUTING {
                type filter hook postrouting priority mangle; policy accept;
        }
}
table ip6 nat {
        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
        }

        chain INPUT {
                type nat hook input priority 100; policy accept;
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
        }
}
table bridge nat {
        chain PREROUTING {
                type filter hook prerouting priority dstnat; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority out; policy accept;
        }

        chain POSTROUTING {
                type filter hook postrouting priority srcnat; policy accept;
        }
}
table inet firewalld {
        chain raw_PREROUTING {
                type filter hook prerouting priority raw + 10; policy accept;
                icmpv6 type { nd-router-advert, nd-neighbor-solicit } accept
                meta nfproto ipv6 fib saddr . iif oif missing drop
                jump raw_PREROUTING_ZONES_SOURCE
                jump raw_PREROUTING_ZONES
        }

        chain raw_PREROUTING_ZONES_SOURCE {
        }

        chain raw_PREROUTING_ZONES {
                iifname "eth0" goto raw_PRE_public
                goto raw_PRE_public
        }

        chain mangle_PREROUTING {
                type filter hook prerouting priority mangle + 10; policy accept;
                jump mangle_PREROUTING_ZONES_SOURCE
                jump mangle_PREROUTING_ZONES
        }

        chain mangle_PREROUTING_ZONES_SOURCE {
        }

        chain mangle_PREROUTING_ZONES {
                iifname "eth0" goto mangle_PRE_public
                goto mangle_PRE_public
        }

        chain filter_INPUT {
                type filter hook input priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                jump filter_INPUT_ZONES_SOURCE
                jump filter_INPUT_ZONES
                ct state { invalid } drop
                reject with icmpx type admin-prohibited
        }

        chain filter_FORWARD {
                type filter hook forward priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 type addr-unreachable
                jump filter_FORWARD_IN_ZONES_SOURCE
                jump filter_FORWARD_IN_ZONES
                jump filter_FORWARD_OUT_ZONES_SOURCE
                jump filter_FORWARD_OUT_ZONES
                ct state { invalid } drop
                reject with icmpx type admin-prohibited
        }

        chain filter_OUTPUT {
                type filter hook output priority filter + 10; policy accept;
                oifname "lo" accept
                ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 type addr-unreachable
        }

        chain filter_INPUT_ZONES_SOURCE {
        }

        chain filter_INPUT_ZONES {
                iifname "eth0" goto filter_IN_public
                goto filter_IN_public
        }

        chain filter_FORWARD_IN_ZONES_SOURCE {
        }

        chain filter_FORWARD_IN_ZONES {
                iifname "eth0" goto filter_FWDI_public
                goto filter_FWDI_public
        }

        chain filter_FORWARD_OUT_ZONES_SOURCE {
        }

        chain filter_FORWARD_OUT_ZONES {
                oifname "eth0" goto filter_FWDO_public
                goto filter_FWDO_public
        }

        chain raw_PRE_public {
                jump raw_PRE_public_pre
                jump raw_PRE_public_log
                jump raw_PRE_public_deny
                jump raw_PRE_public_allow
                jump raw_PRE_public_post
        }

        chain raw_PRE_public_pre {
        }

        chain raw_PRE_public_log {
        }

        chain raw_PRE_public_deny {
        }

        chain raw_PRE_public_allow {
        }

        chain raw_PRE_public_post {
        }

        chain filter_IN_public {
                jump filter_IN_public_pre
                jump filter_IN_public_log
                jump filter_IN_public_deny
                jump filter_IN_public_allow
                jump filter_IN_public_post
                meta l4proto { icmp, ipv6-icmp } accept
        }

        chain filter_IN_public_pre {
        }

        chain filter_IN_public_log {
        }

        chain filter_IN_public_deny {
        }

        chain filter_IN_public_allow {
                tcp dport 22 ct state { new, untracked } accept
                ip6 daddr fe80::/64 udp dport 546 ct state { new, untracked } accept
                tcp dport 9090 ct state { new, untracked } accept
        }

        chain filter_IN_public_post {
        }

        chain filter_FWDI_public {
                jump filter_FWDI_public_pre
                jump filter_FWDI_public_log
                jump filter_FWDI_public_deny
                jump filter_FWDI_public_allow
                jump filter_FWDI_public_post
                meta l4proto { icmp, ipv6-icmp } accept
        }

        chain filter_FWDI_public_pre {
        }

        chain filter_FWDI_public_log {
        }

        chain filter_FWDI_public_deny {
        }

        chain filter_FWDI_public_allow {
        }

        chain filter_FWDI_public_post {
        }

        chain mangle_PRE_public {
                jump mangle_PRE_public_pre
                jump mangle_PRE_public_log
                jump mangle_PRE_public_deny
                jump mangle_PRE_public_allow
                jump mangle_PRE_public_post
        }

        chain mangle_PRE_public_pre {
        }

        chain mangle_PRE_public_log {
        }

        chain mangle_PRE_public_deny {
        }

        chain mangle_PRE_public_allow {
        }

        chain mangle_PRE_public_post {
        }

        chain filter_FWDO_public {
                jump filter_FWDO_public_pre
                jump filter_FWDO_public_log
                jump filter_FWDO_public_deny
                jump filter_FWDO_public_allow
                jump filter_FWDO_public_post
        }

        chain filter_FWDO_public_pre {
        }

        chain filter_FWDO_public_log {
        }

        chain filter_FWDO_public_deny {
        }

        chain filter_FWDO_public_allow {
        }

        chain filter_FWDO_public_post {
        }
}
table ip firewalld {
        chain nat_PREROUTING {
                type nat hook prerouting priority dstnat + 10; policy accept;
                jump nat_PREROUTING_ZONES_SOURCE
                jump nat_PREROUTING_ZONES
        }

        chain nat_PREROUTING_ZONES_SOURCE {
        }

        chain nat_PREROUTING_ZONES {
                iifname "eth0" goto nat_PRE_public
                goto nat_PRE_public
        }

        chain nat_POSTROUTING {
                type nat hook postrouting priority srcnat + 10; policy accept;
                jump nat_POSTROUTING_ZONES_SOURCE
                jump nat_POSTROUTING_ZONES
        }

        chain nat_POSTROUTING_ZONES_SOURCE {
        }

        chain nat_POSTROUTING_ZONES {
                oifname "eth0" goto nat_POST_public
                goto nat_POST_public
        }

        chain nat_PRE_public {
                jump nat_PRE_public_pre
                jump nat_PRE_public_log
                jump nat_PRE_public_deny
                jump nat_PRE_public_allow
                jump nat_PRE_public_post
        }

        chain nat_PRE_public_pre {
        }

        chain nat_PRE_public_log {
        }

        chain nat_PRE_public_deny {
        }

        chain nat_PRE_public_allow {
        }

        chain nat_PRE_public_post {
        }

        chain nat_POST_public {
                jump nat_POST_public_pre
                jump nat_POST_public_log
                jump nat_POST_public_deny
                jump nat_POST_public_allow
                jump nat_POST_public_post
        }

        chain nat_POST_public_pre {
        }

        chain nat_POST_public_log {
        }

        chain nat_POST_public_deny {
        }

        chain nat_POST_public_allow {
        }

        chain nat_POST_public_post {
        }
}
table ip6 firewalld {
        chain nat_PREROUTING {
                type nat hook prerouting priority dstnat + 10; policy accept;
                jump nat_PREROUTING_ZONES_SOURCE
                jump nat_PREROUTING_ZONES
        }

        chain nat_PREROUTING_ZONES_SOURCE {
        }

        chain nat_PREROUTING_ZONES {
                iifname "eth0" goto nat_PRE_public
                goto nat_PRE_public
        }

        chain nat_POSTROUTING {
                type nat hook postrouting priority srcnat + 10; policy accept;
                jump nat_POSTROUTING_ZONES_SOURCE
                jump nat_POSTROUTING_ZONES
        }

        chain nat_POSTROUTING_ZONES_SOURCE {
        }

        chain nat_POSTROUTING_ZONES {
                oifname "eth0" goto nat_POST_public
                goto nat_POST_public
        }

        chain nat_PRE_public {
                jump nat_PRE_public_pre
                jump nat_PRE_public_log
                jump nat_PRE_public_deny
                jump nat_PRE_public_allow
                jump nat_PRE_public_post
        }

        chain nat_PRE_public_pre {
        }

        chain nat_PRE_public_log {
        }

        chain nat_PRE_public_deny {
        }

        chain nat_PRE_public_allow {
        }

        chain nat_PRE_public_post {
        }

        chain nat_POST_public {
                jump nat_POST_public_pre
                jump nat_POST_public_log
                jump nat_POST_public_deny
                jump nat_POST_public_allow
                jump nat_POST_public_post
        }

        chain nat_POST_public_pre {
        }

        chain nat_POST_public_log {
        }

        chain nat_POST_public_deny {
        }

        chain nat_POST_public_allow {
        }

        chain nat_POST_public_post {
        }
}

Greetings Klaas

[Klaas-]

There is another difference I noticed while looking at the cloud-init config -- rhel-payg:

$ cat /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg
# This configuration file is provided by the WALinuxAgent package.
datasource_list: [ Azure ]
datasource:
   Azure:
      apply_network_config: False

Greetings Klaas

[tbugfinder]

Hi @Klaas- out of curiosity, do you have a specific testsuite or how do you "compare" OS installations?

[Klaas-]

hehe no, just things I notice while applying my configuration management. But it should be fairly easy to compare everything, problem is just that the kickstarts used for creating the rhel-payg images are not being published by microsoft.

[tbugfinder]

I'm thinking about creating a public inspec (cinc-auditor) profile which would act as an test-suite for the HPC image builds.

[Klaas-]

@N3WWN did you decide if it's a good idea to automatically enable the firewall? :)

[N3WWN]

Hi @Klaas- !

No decision has been made yet, but we have another call to discuss upcoming changes next week.

[Klaas-]

I got two more follow ups: 1) xfs/ext4 https://github.com/openlogic/AzureBuildCentOS/blob/master/ks/azure/centos83-LVM.ks#L60-L65 ext4 https://github.com/openlogic/AzureBuildCentOS/blob/master/ks/azure/centos83.ks#L54-L55 xfs

The rhel image I looked at uses xfs which is the Red Hat default. I would also suggest to use xfs for /mnt/resource

2) Swap: I think you should disable the swapfile and use a swap partition via cloud-init as default. Reasoning behind dropping the swapfile: https://lkml.org/lkml/2021/3/3/1332 -- swapfile really does not seem suggested by kernel developers. The cloud-init swap partition still has some issues, I have asked red hat and microsoft to come up with a decent solution. (Not sure how integrated you are with Microsoft, but the Case is 2103040050000703) If you implement it like is currently suggested here https://docs.microsoft.com/en-us/azure/virtual-machines/linux/cloudinit-configure-swapfile then systemd will wait for a timeout (if /mnt/resource is lost) because of the /etc/fstab entries before cloud-init is creating the swap. I am guessing they should recommend to use the same options as for the resource disk "nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig" but the case has not yet progressed to them taking my suggestion :)

Greetings Klaas

[Klaas-]

for the firewalld enablement: currently I am just starting/enabling firewalld via ansible postinstall, this will allow sshd and cockpit automatiacally incoming. I have not noticed any problems with waagent or any other microsoft extension I am using

[N3WWN]

Thanks for following up on this. I have this issue noted for our discussions with Microsoft regarding the changes to the images. At this point, I don't think we'll see these changes incorporated in the next image refresh, but hopefully the next one.

[Klaas-]

I noticed another difference: RHEL8-LVM seems to no longer separate /opt on it's own logical volume. I am not sure if this is on purpose or not. Microsoft should really start publishing the kickstarts for their images :)

            "imageReference": {
                "id": "",
                "offer": "RHEL",
                "publisher": "RedHat",
                "sku": "8-lvm-gen2",
                "version": "latest"

# lsblk
NAME              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                 8:0    0   64G  0 disk
├─sda1              8:1    0  500M  0 part /boot
├─sda2              8:2    0   63G  0 part
│ ├─rootvg-tmplv  253:0    0    2G  0 lvm  /tmp
│ ├─rootvg-usrlv  253:1    0   10G  0 lvm  /usr
│ ├─rootvg-homelv 253:2    0    1G  0 lvm  /home
│ ├─rootvg-varlv  253:3    0    8G  0 lvm  /var
│ └─rootvg-rootlv 253:4    0    2G  0 lvm  /
├─sda14             8:14   0    4M  0 part
└─sda15             8:15   0  495M  0 part /boot/efi

[N3WWN]

@Klaas- With the impending demise of CentOS 8, I doubt these issues will get resolved. I brought up this ticket each time we have had meetings to discuss image updates, but we were unable to make progress.

I apologize for any issues that these deltas have caused and hopefully Microsoft will be sharing the kickstarts with the other EL8 cloud image teams.