redis vnet counters stop updating, redis subscribe show no vnet activity

[fszyang]

Not consistently reproducible and unclear the trigger. Observed sometimes in demo testbed with multiple invaders, many BGP containers.

Symptoms:

• goes subscribe and goes hdelta shows no vnet activites even though there are packets going through; sys.x are still getting published
• hget still return values locally as well as remotely, but vnet counters aren't incrementing; redis demon show OK.
• goes vnet show ha show counter incrementing still; vnet demon show OK
• packets are still forwarding; fib tables still getting updating on new changes
• recovers with a goes restart

[jignesh045]

@fszyang can you please add those commands which are not incrementing counters so that i can verify it with latest master branch build?

[fszyang]

goes subscribe should publish vnet interface counter stats bracketed by ".start" and ".stop" at regular intervals. If there are packet/activities on any interfaces, the counter stats will be sandwiched between them. In the issue above, this periodic publishing stops for unknown reason.

Example: after goes subscribe platina-mk1 vnet.poll.start.time: Jul 25 14:56:19.252 vnet.poll.start.channel-length: 0 vnet.poll.stop.time: Jul 25 14:56:19.264

[jignesh045]

Issue verification status: Can not reproduced

Issue verified in the following build:

root@invader44:/# goes version
github.com/platinasystems/go: v0.41-332-g3d923a20
github.com/platinasystems/fe1: v0.2-529-ga5f1fd4
github.com/platinasystems/firmware-fe1a: v0.2
root@invader44:/#
root@invader44:/# goes hget platina-mk1 packages | grep version:
    version: 3d923a20338fc8460211756d67efae0ef8f0408a
    version: a5f1fd4d4e7965f9efd954131f232670b59779ea
    version: 60f39141fbbf78ddb2260dba74c68f2789374f18
root@invader44:/#
root@invader44:/# dpkg --list | grep kernel
ii  kmod                           18-3                                          amd64        tools for managing Linux kernel modules
ii  libdrm2:amd64                  2.4.58-2                                      amd64        Userspace interface to kernel DRM services -- runtime
ii  linux-image-4.13.0-platina-mk1 4.13-134-g39fbcf8c984f                        amd64        Linux kernel, version 4.13.0-platina-mk1
ii  rsyslog                        8.4.2-1+deb8u2                                amd64        reliable system and kernel logging daemon
root@invader44:/#

Steps:

Scenario 1: Using docker container setup.

1. Executed below docker playbook

abhishek@invader28:~/master/go/ansible/regtest$ pwd
/home/abhishek/master/go/ansible/regtest
abhishek@invader28:~/master/go/ansible/regtest$
abhishek@invader28:~/master/go/ansible/regtest$ ansible-playbook -i hosts_testbed1 playbooks/ospf/quagga_ospf_traffic_reset.yml -K

2. At the same time (during playbook execution) monitor the output of "goes subscribe platina-mk1" in all invaders. Attached herewith the output of monitored command.

Attachment: inv32_subscribe.txt inv31_subscribe.txt inv30_subscribe.txt inv29_subscribe.txt

Scenario 2: Using fast ping (through hping3) between directly connected interface.

1. Initiated fast ping (with hping3 command option like timeout 5, ttl 1) in between inv30 & inv32 using directly connected interface eth-5-1. Found same count for .start & .stop periodic publishing.

Attachment: inv30_subscribe_hping3.txt

Observation: Did not found any problem for periodic publishing of ".start" & ".stop" however not sure about how many .start & .stop should get published for specific time period.