logstash-plugins / logstash-integration-snmp

Logstash Integration Plugin for SNMP, including SNMP input and SNMP Trap Plugins
Apache License 2.0
0 stars 3 forks source link

Maximum size fields and Index Fields when doing walks #25

Open eperry opened 6 years ago

eperry commented 6 years ago

While doing very liberal SNMP Walks on a server I came across this issue

Field Names can get exceedingly long and SNMP Walks can create a Huge amount of field names

[2018-11-04T18:58:32,704][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"snmp-2018.11.04", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x22bdd815>], :response=>{"index"=>{"_index"=>"snmp-2018.11.04", "_type"=>"logs", "_id"=>"lcAp4WYBIR0n9FiBt-hA", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of mapping depth [20] in index [snmp-2018.11.04] has been exceeded due to object field [iso.org.dod.internet.mgmt.mib-2.dismanEventMIB.dismanEventMIBObjects.mteEvent.mteEventTable.mteEventEntry.mteEventEntryStatus.6.95.115.110.109.112.100.95.109.116.101.84.114]"}}}}
[2018-11-04T18:47:28,169][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"snmp-2018.11.04", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x78eb628b>], :response=>{"index"=>{"_index"=>"snmp-2018.11.04", "_type"=>"logs", "_id"=>"dpof4WYBf0IKXA2OUQXI", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [6000] in index [snmp-2018.11.04] has been exceeded"}}}}

I thought I would report it, though I am sure it will need some thought, on how to deal with very long mibs

I was thinking the following

to something like

  {
      oid_name: "OID"
      value: oid_value
  }

Not sure how you would do it with one large document

geekpete commented 1 year ago

I wonder if Flattened field type could be used here for some benefit? https://www.elastic.co/guide/en/elasticsearch/reference/current/flattened.html