search

toni-moreno / snmpcollector

A full featured Generic SNMP data collector with Web Administration Interface for InfluxDB
MIT License
288 stars 53 forks source link

Parsing hex-string to float [Feature request] #310

Closed aklockmann closed 6 years ago

aklockmann commented 6 years ago

.1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.35.28.137.0.0.0.0.0.0.0.0.2 = Hex-STRING: 00 00 00 53 34 55 2B D0 .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.35.28.137.0.0.0.0.0.0.0.0.3 = Hex-STRING: 00 00 02 71 8C E3 6B DC .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.35.28.137.0.0.0.0.0.0.0.0.4 = Hex-STRING: 00 00 00 0A 96 D5 FC 90

This hex string has to be transformed to a numeric value to be send to influxdb. Any idea to accomplish this?

toni-moreno commented 6 years ago

Hi @aklockmann we will try to investigate a little more about these Hex-STRINGS values.

The snmpwalk output you has sent me is from the integrated snmp console of the snmpcollector or the command in the "net-snmp" tools ? Could you show me both outputs to show me the difference with different tools?

aklockmann commented 6 years ago

Hi,

The output is from net-snmp snmpwalk. Output from snmpcollector gui looks like: OID Type Value .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.44.82.236.0.0.0.0.0.0.0.0.2 OctetString .�4 .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.44.82.236.0.0.0.0.0.0.0.0.3 OctetString 6�rL .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.44.82.236.0.0.0.0.0.0.0.0.4 OctetString �ٴ Please note that the results do not have the same values, as counter increment. I defined the Metric like:

SNMP Metrics: connUnitPortStatCountTxElements ID: connUnitPortStatCountTxElements FieldName: connUnitPortStatCountTxElements Description: The number of octets or bytes that have been transmitted by this port. There is an one-second periodic polling of the port. This value is saved and compared with the next polled value to compute net throughput. multiplied by four BaseOID: .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.44.82.236.0.0.0.0.0.0.0.0 DataSrcType: OCTETSTRING

toni-moreno commented 6 years ago

Hi @aklockmann I've been looking for information about how to decode Hex-STRING Based data to an "integer", and nothing found. ( the same problem here http://opennms-discuss.narkive.com/fqEoV0xH/hexstring-encoded-values-with-snmp)

I would like to add this kind of data as a new feature for snmpcollector but I will need an example of encoded and also decoded data .

Could you get this "cooked(integer/counter)" and also "raw(hex-string) data from other tools? the brocade console perhaps?

toni-moreno commented 6 years ago

Hi @aklockmann has you tried set type as a COUNTER64 ? what happens when yo do that?

aklockmann commented 6 years ago

Hi @toni-moreno,

.1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.35.28.137.0.0.0.0.0.0.0.0.10 = Hex-STRING: 00 00 01 46 76 6A 49 A4

On console you can use some of these: printf "d\n" 0x00000146766A49A4 echo "00 00 01 46 76 6A 49 A4" | gawk '{gsub(/"/,""); printf "%d\n",strtonum( "0x" $1$2$3$4$5$6$7$8) }' perl -le 'print hex("00000146766A49A4");' All should result in 1402146015652. But i have no clue how to do that in golang. I only come that far.

https://golang.org/pkg/encoding/hex/#Decode package main


import (
    "encoding/hex"
    "fmt"
    "log"
)

func main() {
    src := []byte("00000146766A49A4")

    dst := make([]byte, hex.DecodedLen(len(src)))
    n, err := hex.Decode(dst, src)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("%d\n", dst[:n])

-->[0 0 1 70 118 106 73 164] 
}

has you tried set type as a COUNTER64 ? what happens when yo do that?

Switching to COUNTER64 results in values of 0. Maybe because the first stirng are two zeros.

toni-moreno commented 6 years ago

Hi @aklockmann we can suppose that the value is a direct hexadecimal code value , in this case no problem on how get it. But I have other doubts .Does this Hex-Coded represent always a 64BIT UNSIGNED INTEGER value ? if yes are always counters? My question is related to the UINT64 influxdb problem (influxdb < 1.4.0 has not support for 64 bit unsigned integer values below version , and on that needs for special building), in this context the ¿ should I treat the Hex-STRING , once decoded then cook as COUNTER64 ?

If I create a new type for HEX-STRING based OCTECTSTRING, should I add the kind of decoding process waitting for other kind of decoding types in the future?

@aklockmann have you ever seen any other (non textual) Hex-STRING Based types?

aklockmann commented 6 years ago

Hi @toni-moreno,

Does this Hex-Coded represent always a 64BIT UNSIGNED INTEGER value ? if yes are always counters?

These values seem to be unsigned counters, as they just counting frames passed or droped (as error occure) through an interface.

I read your discussion on https://github.com/influxdata/influxdb/issues/7801. It would be great, if you could do a decode of values HEX->INT (64Bit) and than calcualte the difference and send them over to influx.

If I create a new type for HEX-STRING based OCTECTSTRING, should I add the kind of decoding process waitting for other kind of decoding types in the future?

I don't know if others would need such decoding or transforming in future. Snmp-monitoring is not my daily business.

@aklockmann have you ever seen any other (non textual) Hex-STRING Based types?

No, not yet.

topranks commented 6 years ago

I have run into this issue as well on our setup.

Came across it while trying to poll the CISCO-NTP-MIB. Some of the values are encoded as Hex-String as well:

cmooney@nms-01:~$ snmpwalk -v2c -c commname leaf-switch CISCO-NTP-MIB::cntpSystem
CISCO-NTP-MIB::cntpSysLeap.0 = INTEGER: noWarning(0)
CISCO-NTP-MIB::cntpSysStratum.0 = INTEGER: 3
CISCO-NTP-MIB::cntpSysPrecision.0 = INTEGER: -20
CISCO-NTP-MIB::cntpSysRootDelay.0 = Hex-STRING: 00 07 98 D4  seconds
CISCO-NTP-MIB::cntpSysRootDispersion.0 = Hex-STRING: 00 15 7C 6A  seconds
CISCO-NTP-MIB::cntpSysRefId.0 = Hex-STRING: AC 11 04 FE
CISCO-NTP-MIB::cntpSysRefTime.0 = Hex-STRING: 1B 88 80 DE CE 47 D0 D4
CISCO-NTP-MIB::cntpSysPoll.0 = INTEGER: 6
CISCO-NTP-MIB::cntpSysPeer.0 = INTEGER: 1
CISCO-NTP-MIB::cntpSysClock.0 = Hex-STRING: 40 89 80 DE 51 DD 7B 12

Reading the MIB itself it defines the formats for these elements as follows:

OID Format
cntpSysRootDelay NTPSignedTimeValue
cntpSysRootDispersion NTPUnsignedTimeValue
cntpSysRefId NTPRefID
cntpSysRefTime NTPTimeStamp
cntpSysClock NTPTimeStamp

In turn the MIB defines those formats as follows:

NTPTimeStamp ::= TEXTUAL-CONVENTION
    STATUS          current
    DESCRIPTION
            "NTP timestamps are represented as a 64-bit
            unsigned fixed-point number, in seconds relative to
            00:00 on 1 January 1900.  The integer part is in the
            first 32 bits and the fraction part is in the last
            32 bits."

    REFERENCE
            "D.L. Mills, 'Network Time Protocol (Version 3)',
                RFC-1305, March 1992, Section 3.1"
    SYNTAX          OCTET STRING (SIZE (8))

NTPSignedTimeValue ::= TEXTUAL-CONVENTION
    STATUS          current
    DESCRIPTION
            "The time in seconds that could represent signed
            quantities like time delay with respect to some
            source.  This textual-convention is specific to Cisco
            implementation of NTP where 32-bit integers are used
            for such quantities.  The signed integer part is in
            the first 16 bits and the fraction part is in the
            last 16 bits."

    REFERENCE
            "D.L. Mills, 'Network Time Protocol (Version 3)',
                RFC-1305, March 1992, Sections 2,  3.2.1"
    SYNTAX          OCTET STRING (SIZE (4))

NTPUnsignedTimeValue ::= TEXTUAL-CONVENTION
    STATUS          current
    DESCRIPTION
            "The time in seconds that could represent unsigned
            quantities like maximum error of the local clock
            with respect to some source.  This textual-convention
            is specific to Cisco implementation of NTP where
            32-bit integers are used for such quantities.  The
            unsigned integer part is in the first 16 bits and the
            fraction part is in the last 16 bits."

    REFERENCE
            "D.L. Mills, 'Network Time Protocol (Version 3)',
                RFC-1305, March 1992, Sections 2, 3.2.1"
    SYNTAX          OCTET STRING (SIZE (4))

NTPRefId ::= TEXTUAL-CONVENTION
    STATUS          current
    DESCRIPTION
            "The reference clock identifier.  In the case of
            stratum 0 (unspecified) or stratum 1 (primary
            reference source), this is a four-octet,
            left-justified, zero-padded ASCII string as defined
            in RFC-1305.  In the case of stratum 2 and greater
            (secondary reference) this is the four-octet Internet
            address of the peer selected for synchronization.

            Some examples of stratum 0 identifiers are,
            DCN, DCN routing protocol
            NIST, NIST public modem
            TSP, TSP time protocol
            DTS, Digital Time Service

            Some examples of stratum 1 identifiers are,
            ATOM, Atomic clock (calibrated)
            VLF, VLF radio (OMEGA,, etc.)
            LORC, LORAN-C radionavigation
            GOES, GOES UHF environment satellite
            GPS, GPS UHF satellite positioning"

    REFERENCE
            "D.L. Mills, Network Time Protocol (Version 3)',
                RFC-1305, March 1992, Section 3.2.1"
    SYNTAX          OCTET STRING (SIZE (4))

Not really expecting you to be able to deal with all these varied formats. Is it maybe a potential to have a 'HEX-STRING' as a new DataSrcType? So like for any of these if it could write a textual representation of the received data as pairs of hex digits separated by a space? Basically as the net-snmp tools display it?

The end user could then have fun trying to extract the real data from the likes of the above.

Anyway just adding for more info on this. I was adding that measure as a test so it is not critical to me right now.

p.s. I have a pcap of this I can send if that helps.

toni-moreno commented 6 years ago

Hi @aklockmann , @topranks , we are right now working on adding type conversion to snmpcollector. ( https://github.com/toni-moreno/snmpcollector/pull/368) after this PR merged we will implement the HEX-STRING to float/integer .

But We have a doubt on the best way to do that.

we can use the current OCTECSTRING type and add ability to convert the value to some other non textual type, or create a new one type "Non Textual OCTECSTRING/HexString" to add this new feature.

I think first case would be better. What do you think about?

Also another question I already did some months ago. What possible conversions should I implement ? In the issue use cases It seems to convert the OCTET STRING (HexString) always as integer.

toni-moreno commented 6 years ago

Done on 0.8

ut0f9t commented 5 years ago

Hi Toni I have the same requirement I use the version 0.8 by I cannot see HexString in the list of SNMP Metrics I used Version: 0.8.0 Commit: 9e2519a Build Date: 2018/11/5 15:11:36

Thks for your help

ut0f9t commented 5 years ago

Hi @aklockmann Im also interested to work with snmpcollector. As you, im facing the same problems (brocade switch) Did you succeeded in making the changes and get it into work? Thks

* What SnmpCollector version are you using?
  snmpcollector v0.7.5 (git: [7db4b67](https://github.com/toni-moreno/snmpcollector/commit/7db4b6758f6c8799f10606d1ecdd1f7307b1bcd4) ) built at [2017-12-16 06:54:07]

* What OS are you running snmpcollector on?
  Debian GNU/Linux 8 (jessie)

* What did you do?
  I want to collect metrics from brocade fibrechannel switches. Your examples for Cisco switches do work with brocade as theses OIDs are supported. I managed to collect ifHCInOctets from only active ports, but brocade does support more complex counters and errorcounters in other OIDs. These i'd like to gather. They are fromated as OCTET STRING ( SIZE ( 8 ) ) (i.e. see http://oid-info.com/get/1.3.6.1.3.94.4.5.1.6)
  output from snmpwalk:

.1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.35.28.137.0.0.0.0.0.0.0.0.2 = Hex-STRING: 00 00 00 53 34 55 2B D0 .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.35.28.137.0.0.0.0.0.0.0.0.3 = Hex-STRING: 00 00 02 71 8C E3 6B DC .1.3.6.1.3.94.4.5.1.6.16.0.0.39.248.35.28.137.0.0.0.0.0.0.0.0.4 = Hex-STRING: 00 00 00 0A 96 D5 FC 90

This hex string has to be transformed to a numeric value to be send to influxdb. Any idea to accomplish this?

toni-moreno commented 5 years ago

Hi @ut0f9t , you should select datasource type "OCTETSTRING" and conversion to Integer . Could you test Again please?

ut0f9t commented 5 years ago

Hi Toni sorry for the delay. The problem is that when I define the metric octetstring and conversion to integer it is look fine, but when I edit it again is showing conversation as string. And the query is already sent with wrong conversion as string (output in gui as hex-string)

sqlite> select * from snmp_metric_cfg; connUnitPortStatTabletest|connUnitPortStatCountRxElements||.1.3.6.1.3.94.4.5.1.7|OCTETSTRING|0|0.0|0.0|0||1