search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
736 stars 1.42k forks source link

NTP service bind v6 loopback failed and unable to create socket on lo for ::1#123 when mgmt VRF is enabled #5752

Open irene-pan1202 opened 4 years ago

irene-pan1202 commented 4 years ago

Description Why does the loopback network interface of mgmt VRF not need to be configured ::1/128? This should cause NTP service bind v6 loopback failed and unable to create socket on lo for ::1#123 when mgmt VRF is enabled.

Steps to reproduce the issue:

  1. Enable mgmt vrf

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

ERR ntpd[30772]: bind(18) AF_INET6 ::1#123 flags 0x5 failed: Cannot assign requested address\n',
ERR ntpd[30772]: unable to create socket on lo (2) for ::1#123\n',
anshuv-mfst commented 3 years ago

Please provide more info:

  1. Which release is the issue seen?
  2. Is the issue seen manual test/automation test?
irene-pan1202 commented 3 years ago

@anshuv-mfst Enable mgmt VRF will happen, no matter manual or automatic test

AndriiLozovyi commented 3 years ago

@anshuv-mfst I saw this issue when tried to run test_mgmtvrf, same error messages appears in syslog's. Also same errors in logs when tried it manually using command sudo config vrf add mgmt.

Logs: 

INFO ntp[5783]: Starting NTP server: ntpdStarting NTP server in mgmt-vrf: ntpd.
ERR bgp#fpmsyncd: :- onRouteMsg: Invalid VRF name mgmt (ifindex 83)
ERR ntpd[5809]: bind(17) AF_INET6 ::1#123 flags 0x5 failed: Cannot assign requested address
ERR ntpd[5809]: unable to create socket on lo (1) for ::1#123

Environment: 

SONiC.201911.183-dirty-20201216.084731
Distribution: Debian 9.13
Kernel: 4.9.0-11-2-amd64
Build commit: 76d7c4be
Build date: Wed Dec 16 08:54:40 UTC 2020
Platform: x86_64-arista_7170_64c
laiguoliang-lyr commented 3 years ago

I also encountered a similar problem, is there a solution currently？ Only enable mgmt-vrf will appear.

Logs: sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.2#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet2 (346) for 10.0.0.2#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.2 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.4#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet3 (347) for 10.0.0.4#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.4 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.6#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet4 (348) for 10.0.0.6#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.6 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.8#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet5 (349) for 10.0.0.8#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.8 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.14#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet8 (350) for 10.0.0.14#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.14 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.16#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet9 (351) for 10.0.0.16#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.16 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.18#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet10 (352) for 10.0.0.18#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.18 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.20#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet11 (353) for 10.0.0.20#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.20 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.22#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet12 (354) for 10.0.0.22#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.22 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.24#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet13 (355) for 10.0.0.24#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.24 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.26#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet14 (356) for 10.0.0.26#123

bluecmd commented 2 years ago

I've done some debugging on this. This error is because SONiC VRFs, and more specifically the mgmt VRF does not add a IPv6 loopback address.

It seems that before kernel 4.20 it was not possible to add a IPv6 loopback ::1 to a VRF. Since 202111 is the first release to use a kernel with the fix in (namely 5.10.46 at this writing) we should be able to fix this issue for good in 202111.

I have tested on a Linux machine with 5.14.0 kernel and the following works fine:

ip link add vrf-blue type vrf table 10
ip link set dev vrf-blue up
ip addr add ::1/128 dev vrf-blue
ip addr add fd00::/64 dev vrf-blue
ip -6 addr show vrf-blue
ip -6 ro show vrf vrf-blue

For older versions one could possibly figure out a way for NTPd to ignore listening to ::1, but I was unable to quickly find a way to do so.

hubblezzhu commented 2 years ago

I also encountered a similar problem, is there a solution currently？ Only enable mgmt-vrf will appear.

Logs: sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.2#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet2 (346) for 10.0.0.2#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.2 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.4#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet3 (347) for 10.0.0.4#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.4 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.6#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet4 (348) for 10.0.0.6#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.6 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.8#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet5 (349) for 10.0.0.8#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.8 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.14#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet8 (350) for 10.0.0.14#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.14 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.16#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet9 (351) for 10.0.0.16#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.16 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.18#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet10 (352) for 10.0.0.18#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.18 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.20#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet11 (353) for 10.0.0.20#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.20 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.22#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet12 (354) for 10.0.0.22#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.22 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.24#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet13 (355) for 10.0.0.24#123 sonic INFO ntpd[6227]: failed to init interface for address 10.0.0.24 sonic ERR ntpd[6227]: bind(55) AF_INET 10.0.0.26#123 flags 0x19 failed: Cannot assign requested address sonic ERR ntpd[6227]: unable to create socket on Ethernet14 (356) for 10.0.0.26#123

meet same problem. is there any way to solve?

1uc1fer commented 1 year ago

I've done some debugging on this. This error is because SONiC VRFs, and more specifically the mgmt VRF does not add a IPv6 loopback address.

It seems that before kernel 4.20 it was not possible to add a IPv6 loopback ::1 to a VRF. Since 202111 is the first release to use a kernel with the fix in (namely 5.10.46 at this writing) we should be able to fix this issue for good in 202111.

I have tested on a Linux machine with 5.14.0 kernel and the following works fine:

ip link add vrf-blue type vrf table 10
ip link set dev vrf-blue up
ip addr add ::1/128 dev vrf-blue
ip addr add fd00::/64 dev vrf-blue
ip -6 addr show vrf-blue
ip -6 ro show vrf vrf-blue

For older versions one could possibly figure out a way for NTPd to ignore listening to ::1, but I was unable to quickly find a way to do so.

I did some tests on the 202111 release and the problem still exists when enable mgmt VRF

copyandrun commented 1 year ago

meet same problem. is there any way to solve?

bluecmd commented 1 year ago

We have this patch that we use but hasn't been upstreamed. Feel free to try to upstream it.

files/image_config/rsyslog/rsyslog.conf.j2:

# Define a custom template
$template SONiCFileFormat,"%timegenerated%.%timegenerated:::date-subseconds% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
# Use RFC3164 compatible format for remote syslog
$template SONiCRemoteFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat

#Set remote syslog server
{% for server in SYSLOG_SERVER %}
*.* action(type="omfwd"
  template="SONiCRemoteFormat"
  target="{{ server }}"
  port="{{SYSLOG_SERVER[server].get("port", 514)}}"
  protocol="{{SYSLOG_SERVER[server].get("protocol", "udp")}}"
{% if 'vrf' in SYSLOG_SERVER[server] %}
  device="{{SYSLOG_SERVER[server]["vrf"]}}"
{% endif %}
  )
{% endfor %}

And then you can configure rsyslog per VRF:

{
    "SYSLOG_SERVER": {
        "10.x.y.z": {
            "vrf": "mgmt"
        }
    }
}