search

openconfig / gnmic

gNMIc is a gNMI CLI client and collector
https://gnmic.openconfig.net
Apache License 2.0
191 stars 57 forks source link

gNMIc is unable to scrape data on Port 9273 #528

Open arbabali999 opened 1 month ago

arbabali999 commented 1 month ago

Hi,

I have successfully integrated Nokia and Juniper Routers with gnmic and successfully visualizing telemetry stats on Grafana. But with Huawei, i am unable to scrap data on port 9273 for Prometheus, hence, i am not able to visualize it on Grafana.

gnmic.yml:

10.1.1.1: username: admin password: admin subscriptions:

subscriptions: dplc_interface_stats: paths:

Result:

sudo gnmic -a 10.1.1.1:57400 --insecure -u admin -p admin -d get --path openconfig-interfaces:interfaces/interface[name=100GE1/0/2]/state/counters

2024/10/10 11:11:49.187280 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel Connectivity change to READY [ { "source": "10.1.1.1:57400", "timestamp": 1728540709375863000, "time": "2024-10-10T11:11:49.375863+05:00", "updates": [ { "Path": "openconfig-interfaces:interfaces/interface[name=100GE1/0/2]/state/counters", "values": { "openconfig-interfaces:interfaces/interface/state/counters": { "in-broadcast-pkts": "3753", "in-discards": "0", "in-errors": "2252", "in-fcs-errors": "1595", "in-multicast-pkts": "12466908", "in-octets": "8740684804016284", "in-unicast-pkts": "17446988040687", "in-unknown-protos": "0", "last-clear": "1724223418253000000", "out-broadcast-pkts": "3396", "out-discards": "28727127", "out-errors": "0", "out-multicast-pkts": "15091273", "out-octets": "23302406015041723", "out-unicast-pkts": "20895639374896" } } } ] } ]

Here i can see the statistics but they are not scrapping on port 9273 i.e., gnmic.:9273/metrics where Nokia and Juniper stats are successfully exposing. Refer below snapshot.

image

Note: I have hide the device IPs that are scrapping data successfully for security reasons.

Can you please assist in resolving the issue.

hellt commented 1 month ago

your path in the gnmic config differs from the one you use in the cli example

Try removing the leading forward slash:

paths:
- "openconfig-interfaces:/interfaces/interface[name=100GE1/0/2]/state/counters"
arbabali999 commented 1 month ago

That doesn't seem to be the issue, but I replicated the it regardless, and the problem still persists.

pperesbr commented 3 weeks ago

Hello everyone!!!

I'm also having problems getting the information collection to work correctly with Huawei equipment.

Based on the tests I've done, using the openconfig models doesn't work, so I downloaded the models that Huawei provides from the repository: https://github.com/Huawei/yang.

To do this collection, I managed to use their model, which is from the file: huawei-ifm.yang, as follows:

(.venv) g0004830@p-br-pr-cta-gt2-mk8s-01:~/huawei-yang/yang/network-router/8.22.0/ne40e-x8x16$ pyang -f tree huawei-ifm.yang module: huawei-ifm +--rw ifm +--rw global | +--rw statistic-interval? uint32 | +--rw ipv4-ignore-primary-sub? boolean | +--rw sub-interface-link-trap-enable? boolean | +--rw ipv4-conflict-enable! | | +--rw preempt-enable? boolean | +--rw ipv6-conflict-enable! | +--rw preempt-enable? boolean +--rw damp!

gnmic --timeout 60s -a 10.10.10.10:57400 -u admin -p admin --insecure get --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]

However, it is not very complete, like the IP counters part is not included.

I don't really understand how YANG models work when they extend from another model using the augment option. There is another yang file in this same repository which is: huawei-ip-statistics.yang

Which has the following content:

module: huawei-ip-statistics

augment /ifm:ifm/ifm:interfaces/ifm:interface/ip:ipv4/ip:state: +--ro statistics +--ro in-receives? uint64 +--ro in-octets? uint64 +--ro in-rate-pkts? uint64 +--ro in-rate-bits? uint64 +--ro in-rate-byte-bandwidth-usage? string +--ro in-ucast-pkts? uint64 +--ro in-ucast-octets? uint64 +--ro in-mcast-pkts? uint64 +--ro in-mcast-octets? uint64 +--ro in-bcast-pkts? uint64 +--ro in-bcast-octets? uint64 +--ro out-ucast-pkts? uint64 +--ro out-ucast-octets? uint64 +--ro out-mcast-pkts? uint64 +--ro out-mcast-octets? uint64 +--ro out-bcast-pkts? uint64 +--ro out-bcast-octets? uint64 +--ro out-rate-pkts? uint64 +--ro out-rate-bits? uint64 +--ro out-rate-byte-bandwidth-usage? string +--ro out-transmits? uint64 +--ro out-octets? uint64 +--ro out-frag-requires? uint64 +--ro out-frag-fails? uint32 +--ro out-frag-oks? uint64 +--ro out-frag-creates? uint64 augment /ifm:ifm/ifm:interfaces/ifm:interface/ip:ipv6/ip:state: +--ro statistics +--ro in-receives? uint64 +--ro in-octets? uint64 +--ro in-rate-pkts? uint64 +--ro in-rate-bits? uint64 +--ro in-rate-byte-bandwidth-usage? string +--ro in-ucast-pkts? uint64 +--ro in-ucast-octets? uint64 +--ro in-mcast-pkts? uint64 +--ro in-mcast-octets? uint64 +--ro out-ucast-pkts? uint64 +--ro out-ucast-octets? uint64 +--ro out-mcast-pkts? uint64 +--ro out-mcast-octets? uint64 +--ro out-rate-pkts? uint64 +--ro out-rate-bits? uint64 +--ro out-rate-byte-bandwidth-usage? string +--ro out-transmits? uint64 +--ro out-octets? uint64

I understand that it is extending what comes from huawei-ifm.yang, but what I couldn't do was define how the path is passed to query what is in the IP statistics, that is, how to use a model with augment.

I used gnmic generate but the path it provided me didn't work, example (path start with ifm example, but I need pass huawei-ifm before like yang file):

/ifm/interfaces/interface[name=]/ipv4/state/statistics/in-bcast-octets /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-bcast-pkts /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-mcast-octets /ifm/interfaces/interface[name=]/ipv 4/state/statistics/in-mcast-pkts /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-octets / ifm/interfaces/interface[name=]/ipv4/state/statistics/in-rate-bits /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-rate-byte-bandwidth-usage /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-rate-pkts /ifm/interfaces/interface[name=]/ipv4 /state/statistics/in-receives /ifm/interfaces/interface[name=]/ipv4/state/statistics/in-ucast-octets

In summary I don't set path using a file with augment, the other files with no augment from Huawei working.

pperesbr commented 3 weeks ago

I got api from huawei... this example with rpc, how to convert to gnmic get path:

<?xml version="1.0" encoding="utf-8"?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="">
  <get>
    <filter type="subtree">
      <ifm:ifm xmlns:ifm="urn:huawei:yang:huawei-ifm"
               xmlns:ip-stats="urn:huawei:yang:huawei-ip-statistics"
               xmlns:ip="urn:huawei:yang:huawei-ip">
        <ifm:interfaces>
          <ifm:interface>
            <ip:ipv4>
              <ip:state>
                <ip-stats:statistics/>
              </ip:state>
            </ip:ipv4>
          </ifm:interface>
        </ifm:interfaces>
      </ifm:ifm>
    </filter>
  </get>
</rpc>

Tks,

pperesbr commented 3 weeks ago

guys,

I managed to advance a little, I got to the huawei-ip module which is an augment of huawei-ifm, doing it like this:

gnmic --timeout 60s -a 10.10.10.10:57400 -u admin -p admin --insecure get --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv4/state/

But, this module huawei-ip-statistics not work, this status is:

gnmic --timeout 60s -a 10.10.10.10: 57400 -u admin -p admin --insecure get --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv4/state/huawei-ip-statistics:statistics/in-receives target "200.204.204.13:57400" Get request failed: "200.204.204.13:57400" GetRequest failed: rpc error: code = NotFound desc = Not found.

pperesbr commented 3 weeks ago

Hello,

After several hours trying to understand and solve the problem, I would like to share here how I managed to do it. The path only works when the device's interface configuration has statistics enabled and in forward mode, which is not standard on Huawei, in fact, something that only Huawei does... lol.

Here is the interface configuration and the command to get the counters for both IPv4 and IPv6:

interface GigabitEthernet0/3/6 mtu 9100 undo shutdown set flow-stat interval 10 control-flap damp-interface enable damp-interface level manual 20 2000 750 40 ipv6 enable ip address x.x.x.x 255.255.255.254 ipv6 address x.x.x.x.x./127 ipv6 mtu 9100 trust upstream default isis enable 1 isis ipv6 enable 1 isis circuit-type p2p isis circuit-level level-2 isis authentication-mode keychain isis-pwd isis ipv6 cost 600 isis cost 600 isis timer lsp-throttle 33 isis padding-hello isis ldp-sync isis timer ldp-sync hold-down 10 isis bfd enable isis bfd min-tx-interval 100 min-rx-interval 100 isis suppress-reachability isis ipv6 suppress-reachability mpls mpls mtu 9088 mpls ldp mpls ldp timer hello-send 5 mpls ldp timer keepalive-hold 30 mpls ldp timer keepalive-send 10 undo dcn port-queue-template QOS-TLF outbound poison-reverse enable statistic enable << default is not enabled statistic mode forward << default is mac mpls poison-reverse enable clock synchronization enable clock priority 69 clock bundle 1 clock ssm prc ptp notslave disable ptp enable # return

Huawei's doc.

The statistic mode forward command enables a device's all main physical interfaces and logical interfaces to collect traffic statistics based on IPv4 and IPv6 forwarding.

The undo statistic mode forward command restores the default traffic statistics collection mode.

By default, an interface collects traffic statistics based on the destination MAC address in packets.

gnmic --timeout 60s -a 10.10.10.10:57400 -u admin -p admin--insecure subscribe --path huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv6/state/huawei-ip-statistics:statistics --stream-mode sample { "source": "200.204.204.13:57400", "subscription-name": "default-1730858018", "timestamp": 1730858031666000000, "time": "2024-11-06T01:53:51.666Z", "updates": [ { "Path": "huawei-ifm:ifm/interfaces/interface[name=GigabitEthernet0/3/6]/huawei-ip:ipv6/state/huawei-ip-statistics:statistics", "values": { "huawei-ifm:ifm/interfaces/interface/huawei-ip:ipv6/state/huawei-ip-statistics:statistics": { "in-mcast-octets": 0, "in-mcast-pkts": 0, "in-octets": 5413024333, "in-rate-bits": 56986872, "in-rate-byte-bandwidth-usage": "0.56%", "in-rate-pkts": 5646, "in-receives": 4192815, "in-ucast-octets": 5413024333, "in-ucast-pkts": 4192815, "out-mcast-octets": 0, "out-mcast-pkts": 0, "out-octets": 227212866, "out-rate-bits": 12645488, "out-rate-byte-bandwidth-usage": "0.12%", "out-rate-pkts": 2537, "out-transmits": 803007, "out-ucast-octets": 227212866, "out-ucast-pkts": 803007 } } } ], "deletes": [ "" ] } { "sync-response": true }

@arbabali999 I added you to my LinkedIn, if you want to exchange ideas about monitoring Huawei equipment, just call.

arbabali999 commented 3 weeks ago

Thank you for your help. Unfortunately, my Huawei CE8850 device lacks a command specifically for "statistic mode forwarding." I did enable "statistic enable," and I can view the statistics using the get command. However, when I add the subscription in gnmic.yml, the statistics briefly appear on gnmic:9273/metrics. After refreshing the page, though, the data stops reflecting within a minute.

sudo gnmic -a 10.1.1.1:57400 --insecure -u admin -p admin -d get --path huawei-ifm:ifm/interfaces/interface[ifName=Eth-Trunk5] 2024/11/06 17:21:41.200149 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel Connectivity change to READY [ { "source": "10.1.1.1:57400", "timestamp": 1730895701591116000, "time": "2024-11-06T17:21:41.591116+05:00", "updates": [ { "Path": "huawei-ifm:ifm/interfaces/interface[ifName=Eth-Trunk5]", "values": { "huawei-ifm:ifm/interfaces/interface": { "ifAdminStatus": "up", "ifCeIpStatiEnable": { "ifIpStatiEnable": "Disable", "ifIpStatiIpv4Enable": "Disable", "ifIpStatiIpv6Enable": "Disable" }, "ifClass": "mainInterface", "ifClearedStat": { "ifOperStatItvl": 300, "inBitRate": "856118334", "inByteRate": "107014791", "inPacketRate": "220038", "inUseRate": "0.85%", "outBitRate": "3290595639", "outByteRate": "411324454", "outPacketRate": "382442", "outUseRate": "3.29%", "rcvBroadPacket": "898", "rcvBroadPacketRate": "0", "rcvDropPacket": "0", "rcvErrorPacket": "1843", "rcvMutiPacket": "29068506", "rcvMutiPacketRate": "5", "rcvUniPacket": "1446641410941", "rcvUniPacketRate": "213124", "receiveByte": "1013787103317884", "receivePacket": "1538720216260", "sendBroadPacket": "2441", "sendBroadPacketRate": "0", "sendByte": "3433702400167292", "sendDropPacket": "0", "sendErrorPacket": "0", "sendMutiPacket": "23636849", "sendMutiPacketRate": "7", "sendPacket": "2796174853321", "sendUniPacket": "2732612965330", "sendUniPacketRate": "367698" }, "ifControlFlap": { "ifCtrlFlapEnbl": false }, "ifDescr": " connected to testing link ", "ifDf": false, "ifDynamicInfo": { "ifCtrlFlapDamp": false, "ifLinkStatus": "up", "ifOperMac": "e00c-e5bf-29a1", "ifOperSpeed": "100000000000", "ifOperStatus": "up", "ifOpertMTU": 1500, "ifPhyStatus": "up", "ifV4State": "up", "ifV6State": "down", "isOffline": false, "lineProtocolUpTime": "2024-11-05T03:48:31Z" }, "ifErrorDown": { "errorDownCause": "invalid", "recoveryTime": 0, "remainderTime": 0 }, "ifIndex": 76, "ifLinkProtocol": "ethernet", "ifMtu": 1500, "ifName": "Eth-Trunk5", "ifNetworkLayerStatus": "ipv4andipv6up", "ifNumber": "5", "ifPhyType": "Eth-Trunk", "ifRouterType": "Broadcast", "ifServiceType": "None", "ifStatistics": { "in-realtime-bit-rate": "0", "out-realtime-bit-rate": "0", "rcvBroadPacket": "898", "rcvDropPacket": "0", "rcvErrorPacket": "1843", "rcvMutiPacket": "29068493", "rcvUniPacket": "1446640468161", "receiveByte": "1013786975430492", "receivePacket": "1538719970988", "sendBroadPacket": "2441", "sendByte": "3433701965810814", "sendDropPacket": "0", "sendErrorPacket": "0", "sendMutiPacket": "23636839", "sendPacket": "2796174440593", "sendUniPacket": "2732611345111" }, "ifTrapEnable": true, "ipv4Config": { "addrCfgType": "config" }, "ipv6Config": { "autoLinkLocal": false, "enableFlag": false }, "isL2SwitchPort": true, "l2SubIfFlag": false, "vrfName": "public", "vsName": "Admin-VS" } } } ] } ]

arbabali999 commented 3 weeks ago

PFB the problem i am having.

The stats briefly appear on the port that gNMIc exposes for Prometheus to retrieve data, but they stop displaying shortly afterward. image

After a few moments, it stops displaying any data. image