Failed to load .proto - Githubissues

Sugi275 commented 7 years ago

Hi, I am using stack_kafka and XRv 6.1.2. Logstash can store telemetry in kafka, but the JSON structure is nested and can not be visualized with kibana. Is the next JSON structure normal?

JSON structure:

{
    "@timestamp": "2017-03-16T08:59:00.319Z",
    "@version": "1",
    "collection_end_time": 1489654386353,
    "collection_id": 142581,
    "collection_start_time": 1489654386284,
    "content": [
        {
            "content": {
                "name": "tunnel-te1",
                "timest": 1489654386351
            },
            "timest": 1489654386351
        },
        {
            "content": {
                "content": {
                    "content": [
                        {
                            "index": 0,
                            "timest": 1489654386351
                        },
                        {
                            "content": [
                                {
                                    "index": 0,
                                    "timest": 1489654386351
                                },
                                {
                                    "timest": 1489654386351,
                                    "type": "iana-if-type:mplsTunnel"
                                },
                                {
                                    "admin-status": "UP",
                                    "timest": 1489654386351
                                },
                                {
                                    "oper-status": "UP",
                                    "timest": 1489654386351
                                },
                                {
                                    "mtu": 1500,
                                    "timest": 1489654386351
                                },
                                {
                                    "last-change": 871688,
                                    "timest": 1489654386351
                                },
                                {
                                    "description": "",
                                    "timest": 1489654386351
                                },
                                {
                                    "content": [
                                        {
                                            "in-octets": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "out-octets": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "in-multicast-pkts": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "in-broadcast-pkts": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "out-multicast-pkts": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "out-broadcast-pkts": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "in-unknown-protos": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "in-errors": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "out-errors": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "last-clear": "2017-02-27T07:39:11Z",
                                            "timest": 1489654386351
                                        },
                                        {
                                            "in-unicast-pkts": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "in-discards": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "out-unicast-pkts": 0,
                                            "timest": 1489654386351
                                        },
                                        {
                                            "out-discards": 0,
                                            "timest": 1489654386351
                                        }
                                    ],
                                    "timest": 1489654386351
                                }
                            ],
                            "timest": 1489654386351
                        }
                    ],
                    "timest": 1489654386351
                },
                "timest": 1489654386351
            },
            "timest": 1489654386351
        }
    ],
    "encoding_path": "openconfig-interfaces:interfaces/interface",
    "host": "10.44.160.206",
    "msg_timestamp": 1489654386284,
    "node_id_str": "XRv-Telemetry",
    "port": 45019,
    "subscription_id_str": "testTOtest",
    "timest": 1489654386351
}

And I found out error in stack_kafka 's logstash. I think that the proto file has failed to load.

/data/logstash.log

{:timestamp=>"2017-03-16T08:56:08.626000+0000", :message=>"Failed to load .proto Ruby binding source", :proto_binding_source=>"/data/proto/telemetry_bis.pb.rb", :exception=>#<RuntimeError: Field already exists for tag: 1>, :stacktrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/message.rb:524:in `define_field'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ruby-protocol-buffers-1.6.1/lib/protocol_buffers/runtime/message.rb:536:in `optional'", "/data/proto/telemetry_bis.pb.rb:15:in `Telemetry'", "/data/proto/telemetry_bis.pb.rb:12:in `(root)'", "org/jruby/RubyKernel.java:1059:in `load'", "/opt/logstash/vendor/local_gems/a404edd4/logstash-codec-telemetry-0.9.0/lib/logstash/codecs/telemetry.rb:1:in `(root)'", "org/jruby/RubyDir.java:252:in `glob'", "/opt/logstash/vendor/local_gems/a404edd4/logstash-codec-telemetry-0.9.0/lib/logstash/codecs/telemetry.rb:614:in `register'", "/opt/logstash/vendor/local_gems/a404edd4/logstash-codec-telemetry-0.9.0/lib/logstash/codecs/telemetry.rb:609:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/codecs/base.rb:15:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/codecs/base.rb:48:in `clone'"], :level=>:warn}

Is there a problem on the XRv side? runnning config of XRv 6.1.2.

telemetry model-driven
 destination-group test
  address family ipv4 10.44.160.188 port 5556
   encoding self-describing-gpb
   protocol tcp
  !
 !
 sensor-group ospf
  sensor-path Cisco-IOS-XR-ipv4-ospf-oper-sub2:OSPF-SH-INTERFACE-UP/interface-neighbor-count
 !
 sensor-group test
  sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters
 !
 sensor-group test2
  sensor-path openconfig-interfaces:interfaces/interface
 !
 sensor-group if-oper
  sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface
 !
 sensor-group dial-in-test01
  sensor-path openconfig-interfaces:interfaces/interface
 !
 subscription testTOtest
  sensor-group-id test2 sample-interval 10000
  destination-id test
 !
 subscription sub-dial-in-test01
  sensor-group-id dial-in-test01 sample-interval 30000
 !
!

scadora commented 7 years ago

You didn't miss anything. The hierarchical nature of YANG-modeled data is not especially ELK-friendly (outside a few simple examples). We're working on a new tools to make this more consumable (which also addresses the Kafka issue). Stay tuned, we're almost there.

Sugi275 commented 7 years ago

Hi, scadora

Thank you for your reply. We are waiting for new tools.

and, I have two questions.

one: Is there a timing for the release of new tools? I am not rushing, but I'd like to know a guide

two: stack_kafka, stack_elk only support cisco sensor-path? I understand the telemetry codec is decoding it.

 sensor-group test
  sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters
 !

I understand that openConfig can not decode with telemetry codec.

 sensor-group test2
  sensor-path openconfig-interfaces:interfaces/interface
 !

best regards

scadora commented 7 years ago

At long last, here's Pipeline, the new tool I was talking about:

https://github.com/cisco/bigmuddy-network-telemetry-pipeline

Check it out! It should be much more powerful and flexible. It will work with OpenConfig models as well.

Sugi275 commented 7 years ago

Hi, scadora

Thank you for informing us. I'll give it a try.

CiscoDevNet / bigmuddy-network-telemetry-stacks

Failed to load .proto #7