Interface traffic graph not displaying traffic_out datasource for certain devices

[Givo29]

Describe the bug We are graphing the network traffic for multiple devices using the `Interface - Traffic' Data source (monitoring both inbound and outbound network traffic). For the majority of these devices the graph is working as expected however some are only getting the data for the inbound traffic and not outbound.

Expected behavior The graph should be graphing both the inbound and outbound network traffic, however it is only displaying data for inbound.

Screenshots Graphing command for working graph:

Graph display for working graph:

Data for working graph:

The graphing command for broken graph:

Graph display for broken graph:

Data for broken graph:

Both graphs use the same data template (Interface - Traffic)

Debugging Steps

• I have checked the rrd files for the broken graph - There is data for both inbound and outbound traffic logged in the rrd file.
• Deleted the rrd file and let cacti recreate it
• Deleted the graph from the device and readded it

Desktop (please complete the following information):

• OS:

  $ cat /etc/redhat-release 
  CentOS Linux release 7.6.1810 (Core) 

• Browser:

• Cacti Version: 1.2.2

Additional context Originally the original graph was not displaying inbound or outbound but after deleting the rrdfile the first time and letting cacti recreate it the inbound started to work but the outbound did not.

[netniV]

If you are using Cacti 1.2, have you tried using the Troubleshooting -> Data Source function to see if it finds any issues?

[cigamit]

Edit the Data Query and the Data Query Graph association and verify that both traffic_in and traffic_out are both properly associated and the checkbox checked.

[Givo29]

Sorry for the late reply.. @netniV I checked the troubleshooting for the graph and the only thing it shows it:

Check: Were we able to convert the title?
Value: Missing Datasource
Results: -

However when I click missing Datasource nothing happens (broken link maybe?).

@cigamit I've checked everything: The data query, data source, graph template etc. Everything seems to be right.. The weird thing is that other graphs using the same graph template are logging properly but some are not.

[cigamit]

Next thing to edit the Graph. When you edit it, you will likely see one of the Graph Item Fields mapped to 'None'. You need to unlock the Graph and then set it to the correct value. Then save.

[Givo29]

@cigamit Do you mean edit the graph template? All of the graph items are mapped to the correct data source.

[cigamit]

No, the Graph.

[Givo29]

@cigamit right i see what you mean. They are also correctly mapped

[Givo29]

quick update: I've just found that the devices that are working (getting both inbound and outbound traffic) are all centos/redhat 6 machines while the devices that are only getting inbound traffic but not outbound are all centos/redhat 7 machines.

[Givo29]

Hi guys, just wondering if there has been any update on this issue

[cigamit]

Well, goto Console -> Utilities -> System Utilities -> View Poller cache and take a screen capture of the two entries for the specific graph and post. Second thing to do is go to Console -> Troubleshooting -> Data Debug and debug the data sources. Post the output from that as well.

[Givo29]

Ok, I did both these things and here are the outputs:

[cigamit]

Run a raw snmpwalk for those two OID's on one of the affected devices. Then, on the RRDfile, run the following:

rrdtool info myrrdfile

Post the output of the walks and the info command for the affected RRDfile.

[Givo29]

# snmpwalk -c public -v 2c 10.10.3.47 .1.3.6.1.2.1.2.2.1.10.2
IF-MIB::ifInOctets.2 = Counter32: 632022148

# snmpwalk -c public -v 2c 10.10.3.47 .1.3.6.1.2.1.2.2.1.16.2
IF-MIB::ifOutOctets.2 = Counter32: 311843279

# rrdtool info touriga_qad_channel_islands_traffic_in_987.rrd 
filename = "touriga_qad_channel_islands_traffic_in_987.rrd"
rrd_version = "0003"
step = 300
last_update = 1561416916
header_size = 5216
ds[traffic_in].index = 0
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+08
ds[traffic_in].last_ds = "631933359"
ds[traffic_in].value = 2.3633605351e+04
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 120
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+08                                                                                                                                                                                                            
ds[traffic_out].last_ds = "311835973"                                                                                                                                                                                                             
ds[traffic_out].value = NaN                                                                                                                                                                                                                       
ds[traffic_out].unknown_sec = 16                                                                                                                                                                                                                  
rra[0].cf = "AVERAGE"                                                                                                                                                                                                                             
rra[0].rows = 600                                                                                                                                                                                                                                 
rra[0].cur_row = 223                                                                                                                                                                                                                              
rra[0].pdp_per_row = 1                                                                                                                                                                                                                            
rra[0].xff = 5.0000000000e-01                                                                                                                                                                                                                     
rra[0].cdp_prep[0].value = NaN                                                                                                                                                                                                                    
rra[0].cdp_prep[0].unknown_datapoints = 0                                                                                                                                                                                                         
rra[0].cdp_prep[1].value = NaN                                                                                                                                                                                                                    
rra[0].cdp_prep[1].unknown_datapoints = 0                                                                                                                                                                                                         
rra[1].cf = "AVERAGE"                                                                                                                                                                                                                             
rra[1].rows = 700                                                                                                                                                                                                                                 
rra[1].cur_row = 251                                                                                                                                                                                                                              
rra[1].pdp_per_row = 6                                                                                                                                                                                                                            
rra[1].xff = 5.0000000000e-01                                                                                                                                                                                                                     
rra[1].cdp_prep[0].value = 1.2885037368e+04                                                                                                                                                                                                       
rra[1].cdp_prep[0].unknown_datapoints = 0                                                                                                                                                                                                         
rra[1].cdp_prep[1].value = 0.0000000000e+00                                                                                                                                                                                                       
rra[1].cdp_prep[1].unknown_datapoints = 5                                                                                                                                                                                                         
rra[2].cf = "AVERAGE"                                                                                                                                                                                                                             
rra[2].rows = 775                                                                                                                                                                                                                                 
rra[2].cur_row = 663                                                                                                                                                                                                                              
rra[2].pdp_per_row = 24                                                                                                                                                                                                                           
rra[2].xff = 5.0000000000e-01                                                                                                                                                                                                                     
rra[2].cdp_prep[0].value = 2.0334135875e+04                                                                                                                                                                                                       
rra[2].cdp_prep[0].unknown_datapoints = 0                                                                                                                                                                                                         
rra[2].cdp_prep[1].value = 0.0000000000e+00                                                                                                                                                                                                       
rra[2].cdp_prep[1].unknown_datapoints = 11                                                                                                                                                                                                        
rra[3].cf = "AVERAGE"                                                                                                                                                                                                                             
rra[3].rows = 797                                                                                                                                                                                                                                 
rra[3].cur_row = 478                                                                                                                                                                                                                              
rra[3].pdp_per_row = 288                                                                                                                                                                                                                          
rra[3].xff = 5.0000000000e-01                                                                                                                                                                                                                     
rra[3].cdp_prep[0].value = 3.6398440945e+05                                                                                                                                                                                                       
rra[3].cdp_prep[0].unknown_datapoints = 0                                                                                                                                                                                                         
rra[3].cdp_prep[1].value = 0.0000000000e+00                                                                                                                                                                                                       
rra[3].cdp_prep[1].unknown_datapoints = 275                                                                                                                                                                                                       
rra[4].cf = "MIN"                                                                                                                                                                                                                                 
rra[4].rows = 600                                                                                                                                                                                                                                 
rra[4].cur_row = 164                                                                                                                                                                                                                              
rra[4].pdp_per_row = 1                                                                                                                                                                                                                            
rra[4].xff = 5.0000000000e-01                                                                                                                                                                                                                     
rra[4].cdp_prep[0].value = NaN                                                                                                                                                                                                                    
rra[4].cdp_prep[0].unknown_datapoints = 0                                                                                                                                                                                                         
rra[4].cdp_prep[1].value = NaN                                                                                                                                                                                                                    
rra[4].cdp_prep[1].unknown_datapoints = 0                                                                                                                                                                                                         
rra[5].cf = "MIN"                                                                                                                                                                                                                                 
rra[5].rows = 700                                                                                                                                                                                                                                 
rra[5].cur_row = 461                                                                                                                                                                                                                              
rra[5].pdp_per_row = 6                                                                                                                                                                                                                            
rra[5].xff = 5.0000000000e-01                                                                                                                                                                                                                     
rra[5].cdp_prep[0].value = 1.4200973994e+03                                                                                                                                                                                                       
rra[5].cdp_prep[0].unknown_datapoints = 0                                                                                                                                                                                                         
rra[5].cdp_prep[1].value = inf                                                                                                                                                                                                                    
rra[5].cdp_prep[1].unknown_datapoints = 5
rra[6].cf = "MIN"
rra[6].rows = 775
rra[6].cur_row = 14
rra[6].pdp_per_row = 24
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = 1.1283437323e+03
rra[6].cdp_prep[0].unknown_datapoints = 0
rra[6].cdp_prep[1].value = inf
rra[6].cdp_prep[1].unknown_datapoints = 11
rra[7].cf = "MIN"
rra[7].rows = 797
rra[7].cur_row = 198
rra[7].pdp_per_row = 288
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = 7.3285333392e+02
rra[7].cdp_prep[0].unknown_datapoints = 0
rra[7].cdp_prep[1].value = inf
rra[7].cdp_prep[1].unknown_datapoints = 275
rra[8].cf = "MAX"
rra[8].rows = 600
rra[8].cur_row = 102
rra[8].pdp_per_row = 1
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = NaN
rra[8].cdp_prep[0].unknown_datapoints = 0
rra[8].cdp_prep[1].value = NaN
rra[8].cdp_prep[1].unknown_datapoints = 0
rra[9].cf = "MAX"
rra[9].rows = 700
rra[9].cur_row = 485
rra[9].pdp_per_row = 6
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = 4.2655169099e+03
rra[9].cdp_prep[0].unknown_datapoints = 0
rra[9].cdp_prep[1].value = -inf
rra[9].cdp_prep[1].unknown_datapoints = 5
rra[10].cf = "MAX"
rra[10].rows = 775
rra[10].cur_row = 244
rra[10].pdp_per_row = 24
rra[10].xff = 5.0000000000e-01
rra[10].cdp_prep[0].value = 4.2655169099e+03
rra[10].cdp_prep[0].unknown_datapoints = 0
rra[10].cdp_prep[1].value = -inf
rra[10].cdp_prep[1].unknown_datapoints = 11
rra[11].cf = "MAX"
rra[11].rows = 797
rra[11].cur_row = 137
rra[11].pdp_per_row = 288
rra[11].xff = 5.0000000000e-01
rra[11].cdp_prep[0].value = 4.2655169099e+03
rra[11].cdp_prep[0].unknown_datapoints = 0
rra[11].cdp_prep[1].value = -inf
rra[11].cdp_prep[1].unknown_datapoints = 275
rra[12].cf = "LAST"
rra[12].rows = 600
rra[12].cur_row = 197
rra[12].pdp_per_row = 1
rra[12].xff = 5.0000000000e-01
rra[12].cdp_prep[0].value = NaN
rra[12].cdp_prep[0].unknown_datapoints = 0
rra[12].cdp_prep[1].value = NaN
rra[12].cdp_prep[1].unknown_datapoints = 0
rra[13].cf = "LAST"
rra[13].rows = 700
rra[13].cur_row = 30
rra[13].pdp_per_row = 6
rra[13].xff = 5.0000000000e-01
rra[13].cdp_prep[0].value = 1.6358176056e+03
rra[13].cdp_prep[0].unknown_datapoints = 0
rra[13].cdp_prep[1].value = NaN
rra[13].cdp_prep[1].unknown_datapoints = 5
rra[14].cf = "LAST"
rra[14].rows = 775
rra[14].cur_row = 694
rra[14].pdp_per_row = 24
rra[14].xff = 5.0000000000e-01
rra[14].cdp_prep[0].value = 1.6358176056e+03
rra[14].cdp_prep[0].unknown_datapoints = 0
rra[14].cdp_prep[1].value = NaN
rra[14].cdp_prep[1].unknown_datapoints = 11
rra[15].cf = "LAST"
rra[15].rows = 797
rra[15].cur_row = 270
rra[15].pdp_per_row = 288
rra[15].xff = 5.0000000000e-01
rra[15].cdp_prep[0].value = 1.6358176056e+03
rra[15].cdp_prep[0].unknown_datapoints = 0
rra[15].cdp_prep[1].value = NaN
rra[15].cdp_prep[1].unknown_datapoints = 275

[cigamit]

The max values appear to be incorrect on your RRDfiles. You need to use the RRDtool tune in order to resolve this issue. Take a look at the max_value and the last values. The last values are larger than the max, which means the RRDtool will clip them (make them zero).

[Givo29]

@cigamit by last values, do you mean: ds[traffic_out].last_ds = "311835973"? If that's what you meant it isn't the issue as the inbound and outbound have the same max value and the last ds for both the inbound and outbound are larger than the maximum values for them, yet the inbound traffic is working and the outbound is not. To make sure of this, i set the outbound max value to above the last ds value and it didn't help.

[netniV]

That is unlikely as the RRDtool should never update any source with a value greater than the max. Doing so would be a big bug in the tool.

[Givo29]

@netniV

> ds[traffic_in].max = 1.0000000000e+08
> ds[traffic_in].last_ds = "631933359"

This is the reading I posted above for the inbound traffic and this data source is working for this device, it is traffic_out that is not working, which displayed similar values:

> ds[traffic_out].max = 1.0000000000e+08                                                                                                                                                                                                            
> ds[traffic_out].last_ds = "311835973" 

[netniV]

And that was directly from the RRDTool?

[Givo29]

Yes, the command that was run is rrdtool info touriga_qad_channel_islands_traffic_in_987.rrd