Script Output is not accepted

[eschoeller]

I feel like I keep running into strange problems with script output and spine. I'm getting these errors:

2018/10/20 19:00:00 - SPINE: Poller[1] Device[713] ERROR: Empty result [172.x.x.x]: '/usr/bin/perl /cacti/cacti-1.2.0-beta1-prod/scripts/query_generex.pl -H '172.x.x.x' -C 'community' --temperature'
2018/10/20 19:00:00 - SPINE: Poller[1] WARNING: Invalid Response, Device[713] HT[1] DS[62268] SCRIPT: /usr/bin/perl /cacti/cacti-1.2.0-beta1-prod/scripts/query_generex.pl -H '172.x.x.x' -C 'community' --temperature, output: U

The script outputs this from the command line: TEMP_1:696 TEMP_2:725 TEMP_3:691 TEMP_4:693 TEMP_5:712 TEMP_6:703 TEMP_7:721 TEMP_8:703 TEMP_9:702 TEMP_10:689 TEMP_11:712 TEMP_12:720 TEMP_13:698 TEMP_14:714 TEMP_15:725 TEMP_16:709 TEMP_17:707 TEMP_18:693 TEMP_19:698 TEMP_20:700 TEMP_21:707 TEMP_22:709 TEMP_23:707 TEMP_24:711 TEMP_25:716 TEMP_26:716 TEMP_27:707 TEMP_28:725 TEMP_29:698 TEMP_30:712 TEMP_31:707 TEMP_32:716 TEMP_33:698 TEMP_34:720 TEMP_35:725 TEMP_36:725 TEMP_37:725 TEMP_38:725 TEMP_39:727 TEMP_40:707

This was intermittently working in 1.1.38 and I was able to "kick" it by disabling the device, or disabling boost, or something. But it was erratic.

Now it's consistently not working. Here is a snippet of the script:

if ($temperature) {
   my $output;
   my $results = $session->gettable($BACS_TEMPERATURE_TABLE, noindexes => 1, columns => ['bacsModuleTemperature']);
   my $i = 1;
   foreach my $key (sort { $a <=> $b } keys %{$results}) {
      (my $temp = $results->{$key}{'bacsModuleTemperature'}) =~ s/ 0.1 degrees celcius//g;
      if ($i ==1) { $output .= "TEMP_$key:$temp"; }
      else { $output .= " TEMP_$key:$temp"; }
      $i++;
   }
   print $output;
   exit;
}

I originally would print each TEMP_ value during the loop, but I recall spine doesn't like a trickle of output, rather all in one shot. So I switched to using an $output variable, and got super aggressive and threw in an exit right after the output. Still no dice. But, if I change the script to this:

if ($temperature) {
   my $output;
   my $results = $session->gettable($BACS_TEMPERATURE_TABLE, noindexes => 1, columns => ['bacsModuleTemperature']);
   my $i = 1;
   foreach my $key (sort { $a <=> $b } keys %{$results}) {
      (my $temp = $results->{$key}{'bacsModuleTemperature'}) =~ s/ 0.1 degrees celcius//g;
      if ($i ==1) { $output .= "TEMP_$key:$temp"; }
      else { $output .= " TEMP_$key:$temp"; }
      $i++;
   }
   #print $output;
   print "TEMP_1:693 TEMP_2:725 TEMP_3:702 TEMP_4:694 TEMP_5:716 TEMP_6:707 TEMP_7:725 TEMP_8:707 TEMP_9:702 TEMP_10:689 TEMP_11:714 TEMP_12:721 TEMP_13:700 TEMP_14:716 TEMP_15:725 TEMP_16:712 TEMP_17:711 TEMP_18:696 TEMP_19:698 TEMP_20:707 TEMP_21:711 TEMP_22:709 TEMP_23:707 TEMP_24:714 TEMP_25:716 TEMP_26:712 TEMP_27:707 TEMP_28:725 TEMP_29:698 TEMP_30:711 TEMP_31:707 TEMP_32:716 TEMP_33:705 TEMP_34:725 TEMP_35:725 TEMP_36:721 TEMP_37:725 TEMP_38:725 TEMP_39:729 TEMP_40:707";
   exit;
}

it works just fine. Obviously defeats the purpose though. I think this is different than what I experienced in #1611 .

[cigamit]

Try compiling with popen and testing and then niftypopen and testing. You might want to ensure that the script does not encounter any standard errors by redirecting those to /dev/null in the script.

[eschoeller]

After getting more devices up and running I found 3 other scripts running across 150 other devices that are all failing in a similar manner. These have been running for years with no real problems. So I don't think it's really an issue with this newer script I had recently deployed but something larger. Still looking.

[eschoeller]

I am getting scads of these messages in the cacti_stderr.log

Insecure dependency in `` while running setgid at

In each case this is occurring in when I build an output array or variable in these scripts, but it's not happening when I actually print the data. So, has something changed in the way scripts are called? Specifically perl scripts? Does this have anything to do with the input validation feature?

[eschoeller]

And I can see the difference very clearly here in the way I call this from the command line:

perl /cacti/cacti-1.2.0-beta1-prod/scripts/sarfetch.pl 'x.colorado.edu' '2' 'community' '161' '1500'
CPU:all user:0.00 nice:0.00 system:0.00 iowait:0.00 steal:0.00 idle:100.00 procs:0.00 cswchs:178.00 pgpgins:0.00 pgpgouts:0.00 faults:62.00 majflts:0.00 pgfrees:41.00 pgscanks:0.00 pgscands:0.00 pgsteals:0.00 vmeff:0.00 kbmemfree:434532 kbmemused:1595584 memused:78.60 kbbuffers:8320 kbcached:932328 kbswpused:1613484 swpused:39.09 kbactive:745796 kbinact:535452 kbdirty:20 dentunusd:361784 file_sz:2432 inode_sz:39143 pty_sz:0 runq_sz:1 plist_sz:399 ldavg_1:0.00 ldavg_5:0.14 ldavg_15:0.18 blocked:0 totsck:652 tcpsck:7 udpsck:4 rawsck:0 ip_frag:0 tcp_tw:4

Versus:

perl -T /cacti/cacti-1.2.0-beta1-prod/scripts/sarfetch.pl 'x.colorado.edu' '2' 'community' '161' '1500'
Insecure dependency in `` while running with -T switch at /cacti/cacti-1.2.0-beta1-prod/scripts/sarfetch.pl line 53.

The stderr output is slightly different, but I am getting the feeling this is a tainted variable issue, and maybe this code just sucks or there are some ownership/permissions problems.

[netniV]

The insecure dependency is a perl issued response. That will be fed back to Spine if it's on the stdout, or if stderr isn't redirected, thus causing issues. Why that would be intermittent I couldn't say, but what is at line 53 in your script is the cause of the issue by the looks of it.

[cigamit]

The insecure dependency, google it. It's your perl script. Likely setting something like the PATH or not. not a Cacti issue.

[eschoeller]

Yes, I agree it is something with the script. But what interests me the most is that this became a problem after the upgrade. Before the upgrade I had been running these scripts for years with no problems. The remote pollers are debian boxes and I also did a system upgrade of about 30 packages just a couple hours before the Cacti upgrade (introducing another variable, which was silly) which included some python, apache, php and perl packages so something could have changed at the system level to trip me up on this.

[eschoeller]

OK, I think I made some headway. I actually found this: https://forums.cacti.net/viewtopic.php?f=2&t=41534 And I had erroneously done a 'chomd +s spine' instead of 'chmod u+s spine' (it's difficult to remember this). What's interesting to me is that the spine binary which was +s on an RHEL system caused no problems. But on the debian systems I got all those insecure dependency messages. This error wasn't terribly obvious to me. I'll put in a request for spine to check the setgid bit and throw a warning/error in the log ... maybe that might prevent people from getting tripped up on this again.

[netniV]

It’s does. But only if you are using ICMP as that requires it. Scripts aren’t known if it requires it since it depends on what the script does.