search

magma / magma

Platform for building access networks and modular network services
https://www.magmacore.org/
Other
1.69k stars 607 forks source link

LTE INTEG DEBIAN issue #15310

Open akhilamoyila9 opened 1 year ago

akhilamoyila9 commented 1 year ago

Your Environment

Some of the Magma core services are failing:

Execute the below script to fetch the systemd service status:

magma_status.sh.txt

systemd_svc_error

vagrant@magma-deb:~/build$ sudo tail -f /var/log/syslog
Sep 25 15:07:06 magma-deb sctpd[4085789]: ovs-restart:  magma-bridge-reset.sh -n uplink_br0
Sep 25 15:07:06 magma-deb systemd-networkd[2322]: gtp_br0: Gained IPv6LL
Sep 25 15:07:06 magma-deb sctpd[4086800]: ==4086800==Sanitizer CHECK failed: ../../../../src/libsanit(0) (0, 0)
Sep 25 15:07:06 magma-deb systemd[1]: sctpd.service: Main process exited, code=exited, status=23/n/a
Sep 25 15:07:06 magma-deb systemd-networkd[2322]: mtr0: Gained IPv6LL
Sep 25 15:07:06 magma-deb systemd[1]: sctpd.service: Failed with result 'exit-code'.
Sep 25 15:07:06 magma-deb systemd[1]: Failed to start Magma sctpd service.
Sep 25 15:07:06 magma-deb systemd-networkd[2322]: li_port: Gained IPv6LL
Sep 25 15:07:06 magma-deb smsd[240326]: INFO:root:mme non_eps_service_config is not SMS_ORC8R, sleepi
Sep 25 15:07:06 magma-deb systemd-networkd[2322]: ipfix0: Gained IPv6LL
Sep 25 15:07:11 magma-deb systemd[1]: sctpd.service: Scheduled restart job, restart counter is at 240
Sep 25 15:07:11 magma-deb systemd[1]: Stopped Magma sctpd service.
Sep 25 15:07:11 magma-deb systemd[1]: Starting Magma sctpd service...
Sep 25 15:07:11 magma-deb smsd[240326]: INFO:root:mme non_eps_service_config is not SMS_ORC8R, sleepi
Sep 25 15:07:11 magma-deb systemd[1]: systemd-sysctl.service: Succeeded.
Sep 25 15:07:11 magma-deb systemd[1]: Stopped Apply Kernel Variables.
Sep 25 15:07:11 magma-deb systemd[1]: Stopping Apply Kernel Variables...
Sep 25 15:07:11 magma-deb systemd[1]: Starting Apply Kernel Variables...
Sep 25 15:07:11 magma-deb systemd-sysctl[4086826]: Not setting net/ipv4/conf/all/promote_secondaries
Sep 25 15:07:11 magma-deb systemd-sysctl[4086826]: Not setting net/ipv4/conf/default/promote_secondar
Sep 25 15:07:11 magma-deb systemd[1]: Finished Apply Kernel Variables.
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --all destroy Flow_Samp
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: uplink_br0: Link DOWN
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: uplink_br0: Lost carrier
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: dhcp0: Link DOWN
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: dhcp0: Lost carrier
Sep 25 15:07:13 magma-deb multipathd[560]: sda: add missing path
Sep 25 15:07:13 magma-deb multipathd[560]: sda: failed to get udev uid: Invalid argument
Sep 25 15:07:13 magma-deb multipathd[560]: sda: failed to get sysfs uid: Invalid argument
Sep 25 15:07:13 magma-deb multipathd[560]: sda: failed to get sgio uid: No data available
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:13 magma-deb kernel: [270854.957239] device dhcp0 left promiscuous mode
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:13 magma-deb kernel: [270855.126566] device uplink_br0 left promiscuous mode
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: gtp_br0: Link DOWN
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: gtp_br0: Lost carrier
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:13 magma-deb kernel: [270855.342984] device gtpu_sys_2152 left promiscuous mode
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: gtpu_sys_2152: Link DOWN
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: gtpu_sys_2152: Lost carrier
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: mtr0: Link DOWN
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: mtr0: Lost carrier
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:13 magma-deb kernel: [270855.480900] device mtr0 left promiscuous mode
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: ipfix0: Link DOWN
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: ipfix0: Lost carrier
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:13 magma-deb kernel: [270855.597240] device ipfix0 left promiscuous mode
Sep 25 15:07:13 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:13 magma-deb systemd-networkd[2322]: proxy_port: Link DOWN
Sep 25 15:07:14 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:14 magma-deb kernel: [270855.807199] device proxy_port left promiscuous mode
Sep 25 15:07:14 magma-deb systemd-networkd[2322]: li_port: Link DOWN
Sep 25 15:07:14 magma-deb systemd-networkd[2322]: li_port: Lost carrier
Sep 25 15:07:14 magma-deb multipathd[560]: sdb: add missing path
Sep 25 15:07:14 magma-deb multipathd[560]: sdb: failed to get udev uid: Invalid argument
Sep 25 15:07:14 magma-deb multipathd[560]: sdb: failed to get sysfs uid: Invalid argument
Sep 25 15:07:14 magma-deb multipathd[560]: sdb: failed to get sgio uid: No data available
Sep 25 15:07:14 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:14 magma-deb kernel: [270855.914979] device li_port left promiscuous mode
Sep 25 15:07:14 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:07:14 magma-deb kernel: [270855.950934] device gtp_br0 left promiscuous mode
Sep 25 15:07:14 magma-deb kernel: [270855.977592] device ovs-system left promiscuous mode
Sep 25 15:07:14 magma-deb sctpd[4087244]: ifdown: interface patch-up not configured
Sep 25 15:07:15 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-ex
Sep 25 15:07:15 magma-deb kernel: [270857.085808] device ovs-system entered promiscuous mode
Sep 25 15:07:15 magma-deb kernel: [270857.087305] No such timeout policy "ovs_test_tp"
Sep 25 15:07:15 magma-deb kernel: [270857.087306] Failed to associated timeout policy `ovs_test_tp'
Sep 25 15:07:15 magma-deb networkd-dispatcher[743]: WARNING:Unknown index 192656 seen, reloading inte
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: Using default interface naming scheme 'v245'.
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: ethtool: autonegotiation is unset or enabled, the s
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: ovs-system: Could not generate persistent MAC: No d
Sep 25 15:07:15 magma-deb kernel: [270857.098210] device uplink_br0 entered promiscuous mode
Sep 25 15:07:15 magma-deb systemd-udevd[4086812]: Using default interface naming scheme 'v245'.
Sep 25 15:07:15 magma-deb systemd-udevd[4086812]: ethtool: autonegotiation is unset or enabled, the s
Sep 25 15:07:15 magma-deb networkd-dispatcher[743]: WARNING:Unknown index 192657 seen, reloading inte
Sep 25 15:07:15 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-exal -- set interface dhcp0 ofport_request=3
Sep 25 15:07:15 magma-deb kernel: [270857.185228] device dhcp0 entered promiscuous mode
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: ethtool: autonegotiation is unset or enabled, the s
Sep 25 15:07:15 magma-deb networkd-dispatcher[743]: WARNING:Unknown index 192658 seen, reloading inte
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: dhcp0: Could not generate persistent MAC: No data a
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: dhcp0: Link UP
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: dhcp0: Gained carrier
Sep 25 15:07:15 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-exe=patch options:peer=patch-up -- set interface patch-agw ofport_request=2
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: uplink_br0: Link UP
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: uplink_br0: Gained carrier
Sep 25 15:07:15 magma-deb sctpd[4087428]: RTNETLINK answers: File exists
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: proxy_port: Link UP
Sep 25 15:07:15 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-ex
Sep 25 15:07:15 magma-deb kernel: [270857.438453] device gtp_br0 entered promiscuous mode
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: ethtool: autonegotiation is unset or enabled, the s
Sep 25 15:07:15 magma-deb networkd-dispatcher[743]: WARNING:Unknown index 192659 seen, reloading inte
Sep 25 15:07:15 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-exs:remote_ip=flow options:key=flow -- set interface gtp0 ofport_request=32768
Sep 25 15:07:15 magma-deb kernel: [270857.526566] device gtpu_sys_2152 entered promiscuous mode
Sep 25 15:07:15 magma-deb networkd-dispatcher[743]: WARNING:Unknown index 192660 seen, reloading inte
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: ethtool: autonegotiation is unset or enabled, the s
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: gtpu_sys_2152: Link UP
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: gtpu_sys_2152: Gained carrier
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: gtpu_sys_2152: Gained IPv6LL
Sep 25 15:07:15 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-ex set interface mtr0 ofport_request=15577
Sep 25 15:07:15 magma-deb networkd-dispatcher[743]: WARNING:Unknown index 192661 seen, reloading inte
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: ethtool: autonegotiation is unset or enabled, the s
Sep 25 15:07:15 magma-deb systemd-udevd[4086817]: mtr0: Could not generate persistent MAC: No data av
Sep 25 15:07:15 magma-deb kernel: [270857.645328] device mtr0 entered promiscuous mode
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: mtr0: Link UP
Sep 25 15:07:15 magma-deb systemd-networkd[2322]: mtr0: Gained carrier

vagrant@magma-deb:~/build$ sudo tail -f /var/log/mme.log
tail: cannot open '/var/log/mme.log' for reading: No such file or directory
tail: no files remaining
vagrant@magma-deb:~/build$ sudo tail -f /var/log/syslog
Sep 25 15:11:24 magma-deb sctpd[4110563]: ==4110563==Sanitizer CHECK failed: ../../../../src/libsanit(0) (0, 0)
Sep 25 15:11:24 magma-deb systemd[1]: sctpd.service: Main process exited, code=exited, status=23/n/a
Sep 25 15:11:24 magma-deb systemd-networkd[2322]: li_port: Gained IPv6LL
Sep 25 15:11:24 magma-deb systemd[1]: sctpd.service: Failed with result 'exit-code'.
Sep 25 15:11:24 magma-deb systemd[1]: Failed to start Magma sctpd service.
Sep 25 15:11:24 magma-deb multipathd[560]: sdb: add missing path
Sep 25 15:11:24 magma-deb multipathd[560]: sdb: failed to get udev uid: Invalid argument
Sep 25 15:11:24 magma-deb multipathd[560]: sdb: failed to get sysfs uid: Invalid argument
Sep 25 15:11:24 magma-deb multipathd[560]: sdb: failed to get sgio uid: No data available
Sep 25 15:11:25 magma-deb systemd-networkd[2322]: gtp_br0: Gained IPv6LL
Sep 25 15:11:28 magma-deb smsd[240326]: INFO:root:mme non_eps_service_config is not SMS_ORC8R, sleepi
Sep 25 15:11:28 magma-deb multipathd[560]: sda: add missing path
Sep 25 15:11:28 magma-deb multipathd[560]: sda: failed to get udev uid: Invalid argument
Sep 25 15:11:28 magma-deb multipathd[560]: sda: failed to get sysfs uid: Invalid argument
Sep 25 15:11:28 magma-deb multipathd[560]: sda: failed to get sgio uid: No data available
Sep 25 15:11:29 magma-deb multipathd[560]: sdb: add missing path
Sep 25 15:11:29 magma-deb multipathd[560]: sdb: failed to get udev uid: Invalid argument
Sep 25 15:11:29 magma-deb multipathd[560]: sdb: failed to get sysfs uid: Invalid argument
Sep 25 15:11:29 magma-deb multipathd[560]: sdb: failed to get sgio uid: No data available
Sep 25 15:11:29 magma-deb systemd[1]: sctpd.service: Scheduled restart job, restart counter is at 241
Sep 25 15:11:29 magma-deb systemd[1]: Stopped Magma sctpd service.
Sep 25 15:11:29 magma-deb systemd[1]: Starting Magma sctpd service...
Sep 25 15:11:30 magma-deb systemd[1]: systemd-sysctl.service: Succeeded.
Sep 25 15:11:30 magma-deb systemd[1]: Stopped Apply Kernel Variables.
Sep 25 15:11:30 magma-deb systemd[1]: Stopping Apply Kernel Variables...
Sep 25 15:11:30 magma-deb systemd[1]: Starting Apply Kernel Variables...
Sep 25 15:11:30 magma-deb systemd-sysctl[4110590]: Not setting net/ipv4/conf/all/promote_secondaries
Sep 25 15:11:30 magma-deb systemd-sysctl[4110590]: Not setting net/ipv4/conf/default/promote_secondar
Sep 25 15:11:30 magma-deb systemd[1]: Finished Apply Kernel Variables.
Sep 25 15:11:31 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --all destroy Flow_Samp
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: uplink_br0: Link DOWN
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: uplink_br0: Lost carrier
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: dhcp0: Link DOWN
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: dhcp0: Lost carrier
Sep 25 15:11:31 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:11:31 magma-deb kernel: [271113.366034] device dhcp0 left promiscuous mode
Sep 25 15:11:31 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:11:31 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:11:31 magma-deb kernel: [271113.500010] device uplink_br0 left promiscuous mode
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: gtp_br0: Link DOWN
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: gtp_br0: Lost carrier
Sep 25 15:11:31 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:11:31 magma-deb kernel: [271113.727635] device gtpu_sys_2152 left promiscuous mode
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: gtpu_sys_2152: Link DOWN
Sep 25 15:11:31 magma-deb systemd-networkd[2322]: gtpu_sys_2152: Lost carrier
Sep 25 15:11:32 magma-deb systemd-networkd[2322]: mtr0: Link DOWN
Sep 25 15:11:32 magma-deb systemd-networkd[2322]: mtr0: Lost carrier
Sep 25 15:11:32 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:11:32 magma-deb kernel: [271113.855170] device mtr0 left promiscuous mode
Sep 25 15:11:32 magma-deb systemd-networkd[2322]: ipfix0: Link DOWN
Sep 25 15:11:32 magma-deb systemd-networkd[2322]: ipfix0: Lost carrier
Sep 25 15:11:32 magma-deb ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exi
Sep 25 15:11:32 magma-deb kernel: [271113.986263] device ipfix0 left promiscuous mode

Below is the detailed syslog record: syslog.txt

To Reproduce

The s1ap test cases are failing after the magma dev environment was setup was to reproduce the error

Expected behavior

All the services should come up LTE INTEG DEBIAN should be green in magma ci dashboard

Here is the log file for the test cases output.txt

tmdzk commented 1 year ago

It seems like the gtp_br0 interface is down, that's why all services are crashing one by one. can you try to up it manually? @akhilamoyila9