search

seren / freenas-temperature-graphing

Scripts to graph FreeNAS CPU and drive temperatures
Other
77 stars 14 forks source link

FreeNAS 11.2-U6 update broke HDD/CPU temp graph #22

Open s0n1kpt opened 4 years ago

s0n1kpt commented 4 years ago

Hi, I use your temp graphs since FreeNAS 9.10 and i really like it. Now i update from 11.2-U2 to 11.2-U6 and they stop working properly, i only received I bunch off emails with

ERROR: could not lock RRD

Can i do something to solve this?

Thanks

( And sorry for my English )

seren commented 4 years ago

Have you tried running it manually with the -v flag? This should show where in the script it's failing and what command it was trying to run.

A brief Google search shows that a ERROR: could not lock RRD message can be caused by missing binaries or libraries, or due to incorrect permissions. Is your scheduled task still set to run as root?

s0n1kpt commented 4 years ago

I tried to run it manually several times with the -v flag and nothing showed. I think it's because it is not "sequencial" for example I received one error at 07:25, other at 08:33 and the next at 09:59.

Now I have two scripts running the first every 1min and the second every 5min, the problem only occur on the 1min.

The full email report is: Cron root@CoreCortex PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/sysVault/sysData/rrd-graph.sh /mnt/sysVault/sysData/temps-1min.rrd > /dev/null

ERROR: could not lock RRD

Thanks

If you have any idea what I can do next?!?

Thanks again

seren commented 4 years ago

What's the full output when you run: /mnt/sysVault/sysData/rrd-graph.sh -v /mnt/sysVault/sysData/temps-1min.rrd and /mnt/sysVault/sysData/rrd-graph.sh -d /mnt/sysVault/sysData/temps-1min.rrd

Also, what's the output of ls -l /mnt/sysVault/sysData?

s0n1kpt commented 4 years ago

Sorry for the delay, i tried to start from zero and do everything again. But the problem remains, i really can't understand why during a day i receive 18/20 errors, a day have 1440 minutes, the script runs 1440 times, why only fail 18/20 times ?!? The 5min script is working 100%.

I will keep trying to understand this, and maybe with your help we can solve this. Already cross my mind rollback to 11.2-U2.1, everything was working 100%, but i keep thinking i can help others.

Thanks

ls -l /mnt/sysVault/sysDataSet/bin/

total 16082
-rwxr-xr-x  1 root  wheel      1673 Jun 20  2017 configtomail.sh
-rw-r--r--  1 root  wheel  96579517 Oct 16 01:00 fan_control.log
-rwxr--r--  1 root  wheel     21224 Jun 19  2017 hybrid_fan_controller.pl
-rwxr-xr-x  1 root  wheel      8267 Jun 25  2017 rrd-graph.sh
-rwxr-xr-x  1 root  wheel      8492 Jun 25  2017 rrd.sh
-rwxr-xr-x  1 root  wheel      4438 Jun 20  2017 smart_report.sh
-rwxr--r--  1 root  wheel       158 Jun 19  2017 start_fan_controller.sh
-rwxr-xr-x  1 root  wheel      1483 Jun 20  2017 temps_report.sh
-rw-r--r--  1 root  wheel    207956 Oct 16 01:02 temps-1min-cpus.png
-rw-r--r--  1 root  wheel     38993 Oct 16 01:02 temps-1min-drives.png
-rw-r--r--  1 root  wheel    389264 Oct 16 01:02 temps-1min.rrd
-rw-r--r--  1 root  wheel    140208 Oct 16 01:00 temps-5min-cpus.png
-rw-r--r--  1 root  wheel     33786 Oct 16 01:00 temps-5min-drives.png
-rw-r--r--  1 root  wheel    389264 Oct 16 01:00 temps-5min.rrd
-rwxr-xr-x  1 root  wheel       491 Jun 22  2017 temps-graph-email-report.sh
-rwxr-xr-x  1 root  wheel      2413 Jun 22  2017 temps-rrd-format.sh
-rwxr-xr-x  1 root  wheel       753 Jun 22  2017 temps-simple.sh
-rwxrw-r--  1 root  wheel       490 Jun 19  2017 test.sh
drwxr-xr-x  2 root  wheel         2 Oct 13 06:00 tmp
-rwxr-xr-x  1 root  wheel      3933 Jun 21  2017 zpool_report.sh
 /mnt/sysVault/sysDataSet/bin/rrd.sh -v /mnt/sysVault/sysDataSet/bin/temps-1min.rrd

Rrdtool database filename: /mnt/sysVault/sysDataSet/bin/temps-1min.rrd
Current working directory is: /mnt/sysVault/sysDataSet/bin
Running script: '/mnt/sysVault/sysDataSet/bin/temps-rrd-format.sh'

31:31:32:32:33:33:31:31:22:22:30:32:34:32:29:30

(running script again non-verbosely)
Data: 32:32:31:31:32:33:34:34:22:22:30:32:34:32:29:30

Updating the db: '/usr/local/bin/rrdtool update /mnt/sysVault/sysDataSet/bin/temps-1min.rrd N:32:32:31:31:32:33:34:34:22:22:30:32:34:32:29:30'
Added data
 /mnt/sysVault/sysDataSet/bin/rrd.sh -d /mnt/sysVault/sysDataSet/bin/temps-1min.rrd

+ verbose=1
+ '[' -n '' ']'
++ id -u
+ '[' 0 '!=' 0 ']'
+ '[' -z /mnt/sysVault/sysDataSet/bin/temps-1min.rrd ']'
+ '[' -n 1 ']'
+ echo 'Rrdtool database filename: /mnt/sysVault/sysDataSet/bin/temps-1min.rrd'
Rrdtool database filename: /mnt/sysVault/sysDataSet/bin/temps-1min.rrd
+++ dirname /mnt/sysVault/sysDataSet/bin/rrd.sh
++ cd /mnt/sysVault/sysDataSet/bin
++ pwd
+ CWD=/mnt/sysVault/sysDataSet/bin
+ '[' -n 1 ']'
+ echo 'Current working directory is: /mnt/sysVault/sysDataSet/bin'
Current working directory is: /mnt/sysVault/sysDataSet/bin
+ '[' -e /mnt/sysVault/sysDataSet/bin/temps-1min.rrd ']'
+ func_test_writable /mnt/sysVault/sysDataSet/bin/temps-1min.rrd
+ '[' -e /mnt/sysVault/sysDataSet/bin/temps-1min.rrd ']'
+ '[' -e /mnt/sysVault/sysDataSet/bin/temps-1min.rrd ']'
+ '[' -f /mnt/sysVault/sysDataSet/bin/temps-1min.rrd ']'
+ touch /mnt/sysVault/sysDataSet/bin/temps-1min.rrd
+ return 0
+ '[' -n 1 ']'
+ echo 'Running script: '\''/mnt/sysVault/sysDataSet/bin/temps-rrd-format.sh'\'''
Running script: '/mnt/sysVault/sysDataSet/bin/temps-rrd-format.sh'
+ '[' -n 1 ']'
+ echo ''

+ '[' -n 1 ']'
+ echo ''

+ '[' -n 1 ']'
+ /mnt/sysVault/sysDataSet/bin/temps-rrd-format.sh
32:33:32:33:32:32:31:31:22:22:30:32:34:32:29:30
+ '[' -n 1 ']'
+ echo ''

+ '[' -n 1 ']'
+ echo ''

+ '[' -n 1 ']'
+ echo '(running script again non-verbosely)'
(running script again non-verbosely)
++ /mnt/sysVault/sysDataSet/bin/temps-rrd-format.sh
+ data=35:36:36:35:38:38:36:36:22:22:30:32:34:32:29:30
+ '[' -n 1 ']'
+ echo 'Data: 35:36:36:35:38:38:36:36:22:22:30:32:34:32:29:30'
Data: 35:36:36:35:38:38:36:36:22:22:30:32:34:32:29:30
+ '[' -n 1 ']'
+ echo ''

+ '[' -n 1 ']'
+ echo 'Updating the db: '\''/usr/local/bin/rrdtool update /mnt/sysVault/sysDataSet/bin/temps-1min.rrd N:35:36:36:35:38:38:36:36:22:22:30:32:34:32:29:30'\'''
Updating the db: '/usr/local/bin/rrdtool update /mnt/sysVault/sysDataSet/bin/temps-1min.rrd N:35:36:36:35:38:38:36:36:22:22:30:32:34:32:29:30'
+ /usr/local/bin/rrdtool update /mnt/sysVault/sysDataSet/bin/temps-1min.rrd N:35:36:36:35:38:38:36:36:22:22:30:32:34:32:29:30
+ '[' 0 == 0 ']'
+ '[' -n 1 ']'
+ echo 'Added data'
Added data
+ exit 0
 /mnt/sysVault/sysDataSet/bin/rrd-graph.sh -v /mnt/sysVault/sysDataSet/bin/temps-1min.rrd

+ '[' -n '' ']'
+ '[' -z /mnt/sysVault/sysDataSet/bin/temps-1min.rrd ']'
+ '[' -n 1 ']'
+ echo 'Rrdtool database filename: /mnt/sysVault/sysDataSet/bin/temps-1min.rrd'
Rrdtool database filename: /mnt/sysVault/sysDataSet/bin/temps-1min.rrd
+ '[' -v PS1 ']'
+ MAXGRAPHTEMP=70
+ MINGRAPHTEMP=20
+ SAFETEMPLINE=40
+ LINECOLORS=(000000 00FF00 0000FF FF0000 01FFFE FFA6FE FFDB66 006401 010067 95003A 007DB5 FF00F6 FFEEE8 774D00 90FB92 0076FF D5FF00 FF937E 6A826C FF029D FE8900 7A4782 7E2DD2 85A900 FF0056 A42400 00AE7E 683D3B BDC6FF 263400 BDD393 00B917 9E008E 001544 C28C9F FF74A3 01D0FF 004754 E56FFE 788231 0E4CA1 91D0CB BE9970 968AE8 BB8800 43002C DEFF74 00FFC6 FFE502 620E00 008F9C 98FF52 7544B1 B500FF 00FF78 FF6E41 005F39 6B6882 5FAD4E A75740 A5FFD2 FFB167 009BFF E85EBE)
+ NUMCOLORS=64
+ colorindex=0
+++ dirname /mnt/sysVault/sysDataSet/bin/rrd-graph.sh
++ cd /mnt/sysVault/sysDataSet/bin
++ pwd
+ CWD=/mnt/sysVault/sysDataSet/bin
+ '[' -n 1 ']'
+ echo 'Current working directory is: /mnt/sysVault/sysDataSet/bin'
Current working directory is: /mnt/sysVault/sysDataSet/bin
+ outputprefix=/mnt/sysVault/sysDataSet/bin/temps-1min
+ outputprefix=temps-1min
++ /sbin/sysctl -n hw.ncpu
+ numcpus=8
+ drivedevs=
++ /sbin/sysctl -n kern.disks
++ awk '{for (i=NF; i!=0 ; i--) if(match($i, /da/)) print $i }'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada0
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=22
+ [[ 22 == '' ]]
+ drivedevs=' ada0'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada1
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=22
+ [[ 22 == '' ]]
+ drivedevs=' ada0 ada1'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada2
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=30
+ [[ 30 == '' ]]
+ drivedevs=' ada0 ada1 ada2'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada3
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=32
+ [[ 32 == '' ]]
+ drivedevs=' ada0 ada1 ada2 ada3'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada4
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=34
+ [[ 34 == '' ]]
+ drivedevs=' ada0 ada1 ada2 ada3 ada4'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada5
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=32
+ [[ 32 == '' ]]
+ drivedevs=' ada0 ada1 ada2 ada3 ada4 ada5'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada6
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=29
+ [[ 29 == '' ]]
+ drivedevs=' ada0 ada1 ada2 ada3 ada4 ada5 ada6'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/ada7
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=30
+ [[ 30 == '' ]]
+ drivedevs=' ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7'
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/da0
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=
+ [[ '' == '' ]]
+ for i in $(/sbin/sysctl -n kern.disks | awk '{for (i=NF; i!=0 ; i--) if(match($i, '/da/')) print $i }' )
++ /usr/local/sbin/smartctl -a /dev/da1
++ awk '/194 Temperature_Celsius/{print $0}'
++ awk '{print $10}'
+ DevTemp=
+ [[ '' == '' ]]
+ '[' -n 1 ']'
+ echo 'numcpus: 8'
numcpus: 8
+ '[' -n 1 ']'
+ echo 'drivedevs:  ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7'
drivedevs:  ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7
+ title=Temps
++ echo /mnt/sysVault/sysDataSet/bin/temps-1min.rrd
++ sed 's/.*temps-\(.*\)min.rrd/\1/'
+ interval=1
+ '[' -z 1 ']'
+ '[' -n 1 ']'
+ echo 'Interval minutes: 1'
Interval minutes: 1
+ timespan=86400
+ '[' -n 1 ']'
+ echo 'Graph timespan seconds: 86400'
Graph timespan seconds: 86400
+ '[' -n 1 ']'
+ echo 'Output a combined graph of all cpus'
Output a combined graph of all cpus
+ outputfilename=cpus
+ defsandlines=
+ title='Temperature: All CPUs, 1 minute interval'
+ guidrule=
+ (( i=0 ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 0'
cpu: 0
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 1'
cpu: 1
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 2'
cpu: 2
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 3'
cpu: 3
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 4'
cpu: 4
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3 DEF:cpu4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu4:MAX LINE1:cpu4#01FFFE:cpu4'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 5'
cpu: 5
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3 DEF:cpu4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu4:MAX LINE1:cpu4#01FFFE:cpu4 DEF:cpu5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu5:MAX LINE1:cpu5#FFA6FE:cpu5'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 6'
cpu: 6
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3 DEF:cpu4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu4:MAX LINE1:cpu4#01FFFE:cpu4 DEF:cpu5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu5:MAX LINE1:cpu5#FFA6FE:cpu5 DEF:cpu6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu6:MAX LINE1:cpu6#FFDB66:cpu6'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'cpu: 7'
cpu: 7
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3 DEF:cpu4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu4:MAX LINE1:cpu4#01FFFE:cpu4 DEF:cpu5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu5:MAX LINE1:cpu5#FFA6FE:cpu5 DEF:cpu6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu6:MAX LINE1:cpu6#FFDB66:cpu6 DEF:cpu7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu7:MAX LINE1:cpu7#006401:cpu7'
+ (( i++  ))
+ (( i < 8 ))
+ '[' -n 1 ']'
+ echo 'Definitions and lines:  DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3 DEF:cpu4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu4:MAX LINE1:cpu4#01FFFE:cpu4 DEF:cpu5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu5:MAX LINE1:cpu5#FFA6FE:cpu5 DEF:cpu6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu6:MAX LINE1:cpu6#FFDB66:cpu6 DEF:cpu7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu7:MAX LINE1:cpu7#006401:cpu7'
Definitions and lines:  DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3 DEF:cpu4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu4:MAX LINE1:cpu4#01FFFE:cpu4 DEF:cpu5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu5:MAX LINE1:cpu5#FFA6FE:cpu5 DEF:cpu6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu6:MAX LINE1:cpu6#FFDB66:cpu6 DEF:cpu7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu7:MAX LINE1:cpu7#006401:cpu7
+ write_graph_to_disk
++ date
+ /usr/local/bin/rrdtool graph /mnt/sysVault/sysDataSet/bin/temps-1min-cpus.png -w 1280 -h 317 -a PNG --slope-mode --start end-86400 --end now --font DEFAULT:7: --title 'Temperature: All CPUs, 1 minute interval' --watermark 'Wed Oct 16 01:13:47 WEST 2019' --vertical-label Celcius --right-axis-label Celcius DEF:cpu0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu0:MAX LINE1:cpu0#000000:cpu0 DEF:cpu1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu1:MAX LINE1:cpu1#00FF00:cpu1 DEF:cpu2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu2:MAX LINE1:cpu2#0000FF:cpu2 DEF:cpu3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu3:MAX LINE1:cpu3#FF0000:cpu3 DEF:cpu4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu4:MAX LINE1:cpu4#01FFFE:cpu4 DEF:cpu5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu5:MAX LINE1:cpu5#FFA6FE:cpu5 DEF:cpu6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu6:MAX LINE1:cpu6#FFDB66:cpu6 DEF:cpu7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:cpu7:MAX LINE1:cpu7#006401:cpu7 --right-axis 1:0 --alt-autoscale --lower-limit 20 --upper-limit 70 --rigid
+ '[' -n 1 ']'
+ echo 'Output a combined graph of all drives'
Output a combined graph of all drives
+ outputfilename=drives
+ defsandlines=
+ title='Temperature: All Drives, 1 minute interval'
+ guidrule=HRULE:40#FF0000:Max-safe-temp:dashes
+ i=0
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada0'
drive device: ada0
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0'
+ ((  i = i + 1  ))
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada1'
drive device: ada1
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1'
+ ((  i = i + 1  ))
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada2'
drive device: ada2
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2'
+ ((  i = i + 1  ))
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada3'
drive device: ada3
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3'
+ ((  i = i + 1  ))
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada4'
drive device: ada4
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3 DEF:ada4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada4:MAX LINE1:ada4#01FFFE:ada4'
+ ((  i = i + 1  ))
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada5'
drive device: ada5
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3 DEF:ada4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada4:MAX LINE1:ada4#01FFFE:ada4 DEF:ada5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada5:MAX LINE1:ada5#FFA6FE:ada5'
+ ((  i = i + 1  ))
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada6'
drive device: ada6
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3 DEF:ada4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada4:MAX LINE1:ada4#01FFFE:ada4 DEF:ada5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada5:MAX LINE1:ada5#FFA6FE:ada5 DEF:ada6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada6:MAX LINE1:ada6#FFDB66:ada6'
+ ((  i = i + 1  ))
+ for drdev in ${drivedevs}
+ '[' -n 1 ']'
+ echo 'drive device: ada7'
drive device: ada7
+ ((  colorindex = i % NUMCOLORS  ))
+ defsandlines=' DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3 DEF:ada4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada4:MAX LINE1:ada4#01FFFE:ada4 DEF:ada5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada5:MAX LINE1:ada5#FFA6FE:ada5 DEF:ada6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada6:MAX LINE1:ada6#FFDB66:ada6 DEF:ada7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada7:MAX LINE1:ada7#006401:ada7'
+ ((  i = i + 1  ))
+ '[' -n 1 ']'
+ echo 'Definitions and lines:  DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3 DEF:ada4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada4:MAX LINE1:ada4#01FFFE:ada4 DEF:ada5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada5:MAX LINE1:ada5#FFA6FE:ada5 DEF:ada6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada6:MAX LINE1:ada6#FFDB66:ada6 DEF:ada7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada7:MAX LINE1:ada7#006401:ada7'
Definitions and lines:  DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3 DEF:ada4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada4:MAX LINE1:ada4#01FFFE:ada4 DEF:ada5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada5:MAX LINE1:ada5#FFA6FE:ada5 DEF:ada6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada6:MAX LINE1:ada6#FFDB66:ada6 DEF:ada7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada7:MAX LINE1:ada7#006401:ada7
+ write_graph_to_disk
++ date
+ /usr/local/bin/rrdtool graph /mnt/sysVault/sysDataSet/bin/temps-1min-drives.png -w 1280 -h 317 -a PNG --slope-mode --start end-86400 --end now --font DEFAULT:7: --title 'Temperature: All Drives, 1 minute interval' --watermark 'Wed Oct 16 01:13:48 WEST 2019' --vertical-label Celcius --right-axis-label Celcius HRULE:40#FF0000:Max-safe-temp:dashes DEF:ada0=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada0:MAX LINE1:ada0#000000:ada0 DEF:ada1=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada1:MAX LINE1:ada1#00FF00:ada1 DEF:ada2=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada2:MAX LINE1:ada2#0000FF:ada2 DEF:ada3=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada3:MAX LINE1:ada3#FF0000:ada3 DEF:ada4=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada4:MAX LINE1:ada4#01FFFE:ada4 DEF:ada5=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada5:MAX LINE1:ada5#FFA6FE:ada5 DEF:ada6=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada6:MAX LINE1:ada6#FFDB66:ada6 DEF:ada7=/mnt/sysVault/sysDataSet/bin/temps-1min.rrd:ada7:MAX LINE1:ada7#006401:ada7 --right-axis 1:0 --alt-autoscale --lower-limit 20 --upper-limit 70 --rigid

/mnt/sysVault/sysDataSet/bin/rrd-graph.sh -d /mnt/sysVault/sysDataSet/bin/temps-1min.rrd

The last one as no output.

Thanks again

seren commented 4 years ago

That's odd. The -d option to rrd-graph.sh should definitely produce output. Could you try downloading the latest version and see if -d works then?

The fact that this error is intermittent makes me think that there may be contention between the rrd.sh and rrd-graph.sh scripts. It's possible that the rrd.sh script is still writing data to the temps-1min.rrd file when the rrd-graph.sh script tries to read it.

A quick hack to work around this would be to put sleep 10 && at the beginning of your cron command that runs rrd-graph.sh. This would cause the cron job to sleep for 10 seconds before doing the graphing, and probably give rrd.sh time to finish up (assuming that timing is the problem). It's odd that this hasn't happened for me, but I'd say try this work-around and see if the errors stop.

darby427 commented 4 years ago

same issue here with the could not lock rrd.. sleep 10&& fixed it for me on freenas 11.3u1 .. however, now my problem is I cant email the images like I used to.. the files are created and when I envoke the email script, it sends an email.. but its not attached as an image anymore.. its just code... 99.00% sure its a freenas problem and not a script problem.

s0n1kpt commented 4 years ago

@seren thank you , that really solved the issue. Sorry just being able to answer now but I had some problems with the server and it's been off since , not a lot of free time. @darby427 last weekend during some tests this also happened to me, i had to change the email sending script, i will try to post here as soon as possible.

darby427 commented 4 years ago

thanks! uuencode worked fine until 11.3