search

RedHat-EMEA-SSA-Team / hetzner-ocp4

Installing OCP 4 on single bare metal server.
Apache License 2.0
181 stars 114 forks source link

IPv6 only / air-gapped test installation #319

Open rbo opened 1 month ago

rbo commented 1 month ago

Looks like etc/host is wrong:

[root@pluto mirror]# virsh net-dumpxml demo
<network ipv6='yes'>
  <name>demo</name>
  <uuid>83811258-b28c-47ca-8886-782cd4fd6f5e</uuid>
  <forward mode='open'/>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:36:1b:26'/>
  <domain name='compute.local'/>
  <dns>
    <host ip='2a01:4f9:4a:3355:32::1'>
      <hostname>host.compute.local</hostname>
      <hostname>api-int.demo.openshift.pub</hostname>
      <hostname>api.demo.openshift.pub</hostname>
      <hostname>oauth-openshift.apps.demo.openshift.pub</hostname>
      <hostname>console-openshift-console.apps.demo.openshift.pub</hostname>
    </host>
  </dns>
  <ip family='ipv6' address='2a01:4f9:4a:3355:32::1' prefix='80'>
    <dhcp>
      <range start='2a01:4f9:4a:3355:32::1000' end='2a01:4f9:4a:3355:32::2000'/>
      <host id='00:03:00:01:52:54:00:a8:32:02' name='bootstrap.compute.local' ip='2a01:4f9:4a:3355:32::2'/>
      <host id='00:03:00:01:52:54:00:a8:32:0a' name='master-0.compute.local' ip='2a01:4f9:4a:3355:32::1000'/>
      <host id='00:03:00:01:52:54:00:a8:32:0b' name='master-1.compute.local' ip='2a01:4f9:4a:3355:32::1001'/>
      <host id='00:03:00:01:52:54:00:a8:32:0c' name='master-2.compute.local' ip='2a01:4f9:4a:3355:32::1002'/>
    </dhcp>
  </ip>
</network>

[root@pluto mirror]# cat /etc/hosts 
### Hetzner Online GmbH installimage
127.0.0.1 localhost.localdomain localhost
95.217.117.251 95.217.117.251 95
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
2a01:4f9:4a:3355::2 95.217.117.251 95
# BEGIN ANSIBLE MANAGED BLOCK demo.openshift.pub
95.217.117.251 api.demo.openshift.pub
# END ANSIBLE MANAGED BLOCK demo.openshift.pub
192.168.50.1 host.compute.local
[root@pluto mirror]#
rbo commented 1 month ago

By default podman run's only with IPv4. To change this:

Enable IPv6 at podman network

podman network create --ipv6 --gateway fd00::1:8:1 --subnet fd00::1:8:0/112 --gateway 10.90.0.1 --subnet 10.90.0.0/16 podman1

sudo cp /usr/share/containers/containers.conf /etc/containers/
# change default network
vim /etc/containers/containers.conf
diff -Nuar /usr/share/containers/containers.conf /etc/containers/containers.conf 
--- /usr/share/containers/containers.conf       2024-04-18 10:30:44.000000000 +0200
+++ /etc/containers/containers.conf     2024-06-30 16:10:23.516423665 +0200
@@ -342,7 +342,7 @@

 # The network name of the default network to attach pods to.
 #
-#default_network = "podman"
+default_network = "podman1"

 # The default subnet for the default network given in default_network.
 # If a network with that name does not exist, a new network using that name and

Change quay mirror registry listen config

mkdir executionvars
mv execution-environment.tar executionvars/
cd executionvars
tar xvf execution-environment.tar
# Add FEATURE_LISTEN_IP_VERSION: dual-stack  
# or
# FEATURE_LISTEN_IP_VERSION: IPv6
vim runner/project/roles/mirror_appliance/templates/config.yaml.j2

tar cvjf execution-environment.tar ./*
mv execution-environment.tar ../

Resources:

rbo commented 1 month ago

I don't get DHCPv6 running, let's try agent base installer with static ipv6

agent-config.yaml ```yaml apiVersion: v1alpha1 kind: AgentConfig metadata: name: demo rendezvousIP: 2a01:4f9:4a:3355:32::1000 hosts: - hostname: master-0 interfaces: - name: eno1 macAddress: 52:54:00:a8:32:0a networkConfig: interfaces: - name: eno1 type: ethernet state: up mac-address: 52:54:00:a8:32:0a ipv6: enabled: true address: - ip: 2a01:4f9:4a:3355:32::1000 prefix-length: 80 dhcp: false ipv4: enabled: false dns-resolver: config: server: - 2a01:4f9:4a:3355:32::1 routes: config: - destination: ::/0 next-hop-address: 2a01:4f9:4a:3355:32::1 next-hop-interface: eno1 table-id: 254 - hostname: master-1 interfaces: - name: eno1 macAddress: 52:54:00:a8:32:0b networkConfig: interfaces: - name: eno1 type: ethernet state: up mac-address: 52:54:00:a8:32:0b ipv6: enabled: true address: - ip: 2a01:4f9:4a:3355:32::1001 prefix-length: 80 dhcp: false ipv4: enabled: false dns-resolver: config: server: - 2a01:4f9:4a:3355:32::1 routes: config: - destination: ::/0 next-hop-address: 2a01:4f9:4a:3355:32::1 next-hop-interface: eno1 table-id: 254 - hostname: master-0 interfaces: - name: eno1 macAddress: 52:54:00:a8:32:0c networkConfig: interfaces: - name: eno1 type: ethernet state: up mac-address: 52:54:00:a8:32:0c ipv6: enabled: true address: - ip: 2a01:4f9:4a:3355:32::1002 prefix-length: 80 dhcp: false ipv4: enabled: false dns-resolver: config: server: - 2a01:4f9:4a:3355:32::1 routes: config: - destination: ::/0 next-hop-address: 2a01:4f9:4a:3355:32::1 next-hop-interface: eno1 table-id: 254 ```

firewalld rules are not applied to IPv6 🤷🏻‍♀️

systemctl stop firewalld.service
rbo commented 1 month ago 
[root@pluto foo]# /root/hetzner-ocp4/openshift-install agent wait-for install-complete                                        
INFO Cluster is not ready for install. Check validations 
WARNING Cluster validation: The cluster has hosts that are not ready to install. 
WARNING Host master-0 validation: Hostname master-0 is not unique in cluster                                                  
INFO Host master-1: updated status from insufficient to known (Host is ready to be installed)

fixed agent-config.yaml, duplicated hostname

rbo commented 1 month ago 
[root@master-0 kubernetes]# crictl logs 138dc7f8d13af 2>&1| tail -5
I0701 19:02:05.955568       1 flags.go:64] FLAG: --vmodule=""
I0701 19:02:05.955571       1 flags.go:64] FLAG: --watch-cache="true"
I0701 19:02:05.955573       1 flags.go:64] FLAG: --watch-cache-sizes="[]"
I0701 19:02:05.955611       1 options.go:222] external host was not specified, using 10.88.0.1
E0701 19:02:05.956410       1 run.go:74] "command failed" err="service IP family \"fd00:172:16::/112\" must match public address family \"10.88.0.1\""
[root@master-0 kubernetes]#

Installion stuck in bootstrap...

rbo commented 1 month ago 
[root@master-0 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:a8:32:0a brd ff:ff:ff:ff:ff:ff
    inet6 2a01:4f9:4a:3355:32::1000/80 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fea8:320a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: cni-podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 82:73:68:3f:ae:10 brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.1/16 brd 10.88.255.255 scope global cni-podman0
       valid_lft forever preferred_lft forever
    inet6 fe80::8073:68ff:fe3f:ae10/64 scope link 
       valid_lft forever preferred_lft forever
4: vethf71d1b3e@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni-podman0 state UP group default 
    link/ether 4e:63:83:ae:b5:1f brd ff:ff:ff:ff:ff:ff link-netns netns-8497190a-9936-7298-b9ac-05e06a7f3c97
    inet6 fe80::4c63:83ff:feae:b51f/64 scope link 
       valid_lft forever preferred_lft forever
rbo commented 1 month ago

Looks like it pick the cni-podman0 interface ip...