Error while appending rules to iptable chains

[sridhargaddam]

In an OCP deployment with Submariner version 0.12, it was seen that after we create a chain, its taking few milliseconds for the chain to be actually programmed on the node. From the Globalnet pod logs...

+ trap 'exit 1' SIGTERM SIGINT
+ SUBMARINER_VERBOSITY=2
+ '[' '' == true ']'
+ DEBUG=-v=2
+ exec submariner-globalnet -v=2 -alsologtostderr
W0322 08:45:15.847390       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0322 08:45:15.852666       1 main.go:76] Starting submariner-globalnet{acm-nmanos-devcluster-a2-aws submariner-operator [] false}
I0322 08:45:23.452615       1 request.go:645] Throttling request took 1.148508792s, request: GET:https://172.30.0.1:443/apis/whereabouts.cni.cncf.io/v1alpha1?timeout=32s
I0322 08:45:26.457138       1 gateway_monitor.go:117] Starting GatewayMonitor to monitor the active Gateway node in the cluster.
I0322 08:45:26.558063       1 gateway_monitor.go:144] In processNextEndpoint, endpoint info: {"metadata":{"name":"acm-api-default-cl1-devcluster-openshift-com-submariner-cable-acm-api-default-cl1-devcluster-openshift-com-10-167-3-176","namespace":"submariner-operator","uid":"0461a6fe-972f-4637-82b9-f7ea36264187","resourceVersion":"110500","generation":2,"creationTimestamp":"2022-03-22T08:28:10Z","labels":{"submariner-io/clusterID":"acm-api-default-cl1-devcluster-openshift-com"},"managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-03-22T08:28:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:submariner-io/clusterID":{}}},"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"acm-api-default-cl1-devcluster-openshift-com","cable_name":"submariner-cable-acm-api-default-cl1-devcluster-openshift-com-10-167-3-176","healthCheckIP":"242.1.255.254","hostname":"default-cl1-7cdfw-worker-0-gfck4","subnets":["242.1.0.0/16"],"private_ip":"10.167.3.176","public_ip":"66.187.232.127","nat_enabled":true,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4502"}}}
I0322 08:45:26.558206       1 gateway_monitor.go:147] Endpoint "acm-api-default-cl1-devcluster-openshift-com", host: "default-cl1-7cdfw-worker-0-gfck4" belongs to a remote cluster
I0322 08:45:26.558502       1 gateway_monitor.go:448] Marking traffic destined to remote cluster: -d 242.1.0.0/16 -j MARK --set-mark 0xC0000/0xC0000
I0322 08:45:26.558504       1 gateway_monitor.go:332] Install/ensure SUBMARINER-GN-MARK chain exists
E0322 08:45:26.569034       1 gateway_monitor.go:451] error appending iptables rule "-d 242.1.0.0/16 -j MARK --set-mark 0xC0000/0xC0000": running [/usr/sbin/iptables -t nat -A SUBMARINER-GN-MARK -d 242.1.0.0/16 -j MARK --set-mark 0xC0000/0xC0000 --wait 5]: exit status 1: iptables: No chain/target/match by that name.

I0322 08:45:26.569192       1 gateway_monitor.go:144] In processNextEndpoint, endpoint info: {"metadata":{"name":"acm-nmanos-devcluster-a2-aws-submariner-cable-acm-nmanos-devcluster-a2-aws-10-0-94-155","namespace":"submariner-operator","uid":"87f4f202-d388-40af-84ad-03d1244eccf8","resourceVersion":"104077","generation":2,"creationTimestamp":"2022-03-22T08:25:05Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-03-22T08:25:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"acm-nmanos-devcluster-a2-aws","cable_name":"submariner-cable-acm-nmanos-devcluster-a2-aws-10-0-94-155","healthCheckIP":"242.0.255.254","hostname":"ip-10-0-94-155","subnets":["242.0.0.0/16"],"private_ip":"10.0.94.155","public_ip":"54.176.41.197","nat_enabled":true,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4502"}}}
I0322 08:45:26.569230       1 gateway_monitor.go:448] Marking traffic destined to remote cluster: -d 242.1.0.0/16 -j MARK --set-mark 0xC0000/0xC0000
E0322 08:45:26.576641       1 gateway_monitor.go:451] error appending iptables rule "-d 242.1.0.0/16 -j MARK --set-mark 0xC0000/0xC0000": running [/usr/sbin/iptables -t nat -A SUBMARINER-GN-MARK -d 242.1.0.0/16 -j MARK --set-mark 0xC0000/0xC0000 --wait 5]: exit status 1: iptables: No chain/target/match by that name.

subctl version: | v0.12.0-rc1 Submariner version: | v0.12.0 Kubernetes Server version: | v1.23.3+e419edf Globalnet enabled.

[yboaron]

Do you think it might be related to iptables modes (nft, legacy) ? like the MSS clamping bug?

[sridhargaddam]

Do you think it might be related to iptables modes (nft, legacy) ? like the MSS clamping bug?

In case of MSS clamping rules the rules were not properly translated in the automatic translation layer. Whereas these are regular iptable rules which are properly supported, just that it seems to take some time to get sync'ed. So I believe it may not be related.

[sridhargaddam]

This issue is fixed and backported, closing it.