Not able to store all data into InfluxDB

[mohsin106]

Hi,

I'm able to see all the data on STD-OUT but not all of it is being stored inside my InfluxDB. For example, I'm only seeing the following fields being stored in my InfluxDB:

show field keys from "jnpr.jvision"
name: jnpr.jvision
fieldKey         fieldType
--------         ---------
device           string
egress_queue     integer
interface        string
interface_parent string
type             string
value            integer
value_float      float
value_string     string

When data is scrolling to STD-OUT I see JSON formatted text which has the data that I need, but for some reason, it does not get stored in my InfluxDB.

This is what my fluent.conf looks like:

$ cat /etc/fluent/fluent.conf
<source>
  @type forward
  @id forward_input
</source>

#############
## INPUT ##
#############

<source>
   @type udp
   tag jnpr.jvision
   #tag debug
   format juniper_jti
   #format juniper_analyticsd
   output_format structured
   port 50051
   bind 0.0.0.0
</source>

##############
## OUTPUT ##
##############

<match jnpr.**>
   @type copy

   <store>
      @type influxdb
      host localhost
      port 8086
      dbname testfluent
      #measurement interfaces
      #user juniper
      #password juniper
      #value_keys ["value"]
      #value_keys ["jti_msg_json"]
      value_keys ["jnpr_interface_ext"]
      buffer_type memory
      # buffer_chunk_limit 524288 # 512 * 1024
      # buffer_queue_limit 1024
      flush_interval 2
      auto_tags enable
      # retry_limit 17
      # retry_wait 1.0
      # num_threads 1
   </store>
</match>

## match tag=debug.** and dump to console
<match jnpr.**>
  @type stdout
  @id stdout_output
</match>

The data that I'm trying to push info my InfluxDB prints out like this to STD-OUT:

2019-09-12 14:48:51 +0000 [debug]: #0 fluent/log.rb:302:debug: Received JTI sensor data from device 'router1:10.10.10.10' at time '1568299731'
2019-09-12 14:48:51 +0000 [debug]: #0 fluent/log.rb:302:debug: Value of 'jti_msg_json': '{"system_id"=>"router1:10.10.10.10", "component_id"=>0, "sensor_name"=>"ifd:/junos/system/linecard/interface/:/junos/system/linecard/interface/:PFE", "sequence_number"=>51855, "timestamp"=>1568299731366, "version_major"=>1, "version_minor"=>1, "enterprise"=>{"juniperNetworks"=>{"jnpr_interface_ext"=>{"interface_stats"=>[{"if_name"=>"et-0/0/0", "init_time"=>1565024147, "snmp_if_index"=>651, "parent_ae_name"=>"ae13", "egress_queue_info"=>[{"queue_number"=>0, "packets"=>749989902, "bytes"=>708195314929, "tail_drop_packets"=>0, "rl_drop_packets"=>0, "rl_drop_bytes"=>0, "red_drop_packets"=>0, "red_drop_bytes"=>0, "avg_buffer_occupancy"=>15986, "cur_buffer_occupancy"=>0, "peak_buffer_occupancy"=>15986, "allocated_buffer_size"=>4294967295}, {"queue_number"=>1, "packets"=>39370, "bytes"=>4554094, "tail_drop_packets"=>0, "rl_drop_packets"=>0, "rl_drop_bytes"=>0, "red_drop_packets"=>0, "red_drop_bytes"=>0, "avg_buffer_occupancy"=>0, "cur_buffer_occupancy"=>0, "peak_buffer_occupancy"=>0, "allocated_buffer_size"=>4294967295}, {"queue_number"=>2, "packets"=>3430, "bytes"=>4374851, "tail_drop_packets"=>0, "rl_drop_packets"=>0, "rl_drop_bytes"=>0, "red_drop_packets"=>0, "red_drop_bytes"=>0, "avg_buffer_occupancy"=>0, "cur_buffer_occupancy"=>0, "peak_buffer_occupancy"=>0, "allocated_buffer_size"=>4294967295}, {"queue_number"=>3, "packets"=>0, "bytes"=>0, "tail_drop_packets"=>0, "rl_drop_packets"=>0, "rl_drop_bytes"=>0, "red_drop_packets"=>0, "red_drop_bytes"=>0, "avg_buffer_occupancy"=>0, "cur_buffer_occupancy"=>0, "peak_buffer_occupancy"=>0, "allocated_buffer_size"=>4294967295}, {"queue_number"=>4, "packets"=>0, "bytes"=>0, "tail_drop_packets"=>0, "rl_drop_packets"=>0, "rl_drop_bytes"=>0, "red_drop_packets"=>0, "red_drop_bytes"=>0, "avg_buffer_occupancy"=>0, "cur_buffer_occupancy"=>0, "peak_buffer_occupancy"=>0, "allocated_buffer_size"=>4294967295}, {"queue_number"=>5, "packets"=>73655, "bytes"=>13382896, 

So I want to be able to store all this data inside my InfluxDB. Any help would be greatly appreciated, thank you.