end of night script doesn't support large file numbers

[maphilli14]

I have an ASI290MM and shoot with these settings

Dec 13 12:55:02 allsky allsky.sh[26948]: ZWO ASI290MM Information:
Dec 13 12:55:02 allsky allsky.sh[26948]:   - Native Resolution: 1936x1096
Dec 13 12:55:02 allsky allsky.sh[26948]:   - Pixel Size: 2.9microns
Dec 13 12:55:02 allsky allsky.sh[26948]:   - Supported Bins: 1 2
Dec 13 12:55:02 allsky allsky.sh[26948]:   - Mono camera
Dec 13 12:55:02 allsky allsky.sh[26948]: Camera does not support serialNumber
Dec 13 12:55:02 allsky allsky.sh[26948]: - Sensor temperature: 41.10
Dec 13 12:55:02 allsky allsky.sh[26948]: Capture Settings:
Dec 13 12:55:02 allsky allsky.sh[26948]:  Image Type: ASI_IMG_RAW8
Dec 13 12:55:02 allsky allsky.sh[26948]:  Resolution (before any binning): 1936x1096
Dec 13 12:55:02 allsky allsky.sh[26948]:  Quality: 95
Dec 13 12:55:02 allsky allsky.sh[26948]:  Daytime capture: Yes
Dec 13 12:55:02 allsky allsky.sh[26948]:  Exposure (day): 0.500ms, Auto: Yes
Dec 13 12:55:02 allsky allsky.sh[26948]:  Exposure (night): 1,000ms, Auto: Yes
Dec 13 12:55:02 allsky allsky.sh[26948]:  Max Auto-Exposure (day): 60,000ms (60.0s)
Dec 13 12:55:02 allsky allsky.sh[26948]:  Max Auto-Exposure (night): 4,000ms (4.0s)
Dec 13 12:55:02 allsky allsky.sh[26948]:  Delay (day): 5,000ms
Dec 13 12:55:02 allsky allsky.sh[26948]:  Delay (night): 100ms
Dec 13 12:55:02 allsky allsky.sh[26948]:  Gain (night only): 250, Auto: No, max: 200
Dec 13 12:55:02 allsky allsky.sh[26948]:  Gain Transition Time: 15.0 minutes
Dec 13 12:55:02 allsky allsky.sh[26948]:  Brightness (day): 50
Dec 13 12:55:02 allsky allsky.sh[26948]:  Brightness (night): 50

Nearing the Northern Hemisphere's winter solstice, with long nights I get this many files:

allsky:~/allsky/images/20211211 $ ls -1 | wc -l
12335

In most cases the keogram and startrails are working well, but the timelapse does not produce a video.

In trying to clean them up with... /home/pi/allsky/scripts/removeBadImages.sh .

it throws this error:

/home/pi/allsky/scripts/removeBadImages.sh: line 93: /usr/bin/which: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 131: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 139: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 143: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 153: /usr/bin/bc: Argument list too long

So I have 2 issues in one.

1) The timelapse NEVER generates a result over the past 6 nights 2) My inability to clean up any bad images via the included script.

[linuxkidd]

Hi @maphilli14 We can certainly fix the high image count issues in removeBadImages.sh and the timelapse generation (actually, timelapse may not be fixable on the Pi due to memory requirements of ffmpeg...) However, I suspect there's a mix-up in your config that is causing the high file counts...

Dec 13 12:55:02 allsky allsky.sh[26948]:  Max Auto-Exposure (day): 60,000ms (60.0s)
Dec 13 12:55:02 allsky allsky.sh[26948]:  Max Auto-Exposure (night): 4,000ms (4.0s)

Max auto-exposure: DAY - 60s NIGHT - 4s

With these settings, I'm surprised if you're not getting only the very brightest stars in your night exposures. Is this intentional?

[maphilli14]

I was poking around a few nights back and am getting down to Mag 7+ even with a gibbous moon nearby! That camera is super sensitive, especially with the 150degree lens on it! I have the daytime set at default but it never goes that high, typically the daytime is in microSeconds in duration with no gain. Typical nighttime are 4seconds and 250gain which gets me well past visual star limiting magnitude. I'll pull a sample later to show you.

[maphilli14]

let me know how this comes across...

[EricClaeys]

@linuxkidd could you please take a look at removeBadImages.sh on lines 93 and 139 and let me know why you think they are producing the error message? They aren't even using the argument list.

@maphilli14 when I am back from vacation in a week I will fix removeBadImages.sh to work with any number of images. I only take about 2000 a day so have never seen this problem. A future enhancement will be to run it for every image as it is being processed rather than once a day for all images. In the meantime you may want to increase your exposure time and decrease gain to decrease the graininess of your images. For example, 8 sec exposure and 125 gain.

What do your log files say regarding the timelapse not working?

[jimjackii]

Hi,

I would have the problem elsewhere. Increasing the ulimit helped.

ulimit -s 65535 or ulimit -s 131071

Set this parameter at the begin in the shell command file.

[maphilli14]

for use of the scripts running at startup would you need to set this as some env var or where would you include? Thanks @jimjackii

I have an older copy of allsky that creates the symlinks and I'm running it on a WSL in Win10's Ubuntu shell after scp'ing the files across the network. Will see if that gets me a full mp4 or not!

[maphilli14]

yah, that didn't work, it removed all of them....

removeBadImages.sh: Removed './image-20211216004023.jpg' (bad threshold: MEAN=32)
removeBadImages.sh: Removed './image-20211216012113.jpg' (bad threshold: MEAN=41)
removeBadImages.sh: Removed './image-20211215180128.jpg' (bad threshold: MEAN=39)
removeBadImages.sh: Removed './image-20211215200035.jpg' (bad threshold: MEAN=37)
removeBadImages.sh: Removed './image-20211216005831.jpg' (bad threshold: MEAN=37)
removeBadImages.sh: 13115 bad file(s) found and removed.

[EricClaeys]

@maphilli14 what are your values for REMOVE_BAD_IMAGES_THRESHOLD_LOW and REMOVE_BAD_IMAGES_THRESHOLD_HIGH in config.sh? Can you attach one of the files it removed so I can try?

[jimjackii]

@maphhilli14 I set the parameter "ulimit" at the begin of my shell script like this

#!/bin/sh
ulimit -s 65535
... script code ...
exit 0

[maphilli14]

@jimjackii - Is end of night a sufficient place to put it?

Can you attach one of the files it removed so I can try?

I'm not sure I want to mix old and new versions across different platforms. Let me try another set and get back with you.

[jimjackii]

@jimjackii - Is end of night a sufficient place to put it?

@maphilli14 I've set this parameter in every file that has a large number of files being edited.

[maphilli14]

@jimjackii I tried using your ulimit in the remove bad images... I don't think it helped...

pi@allsky:~ $ cd allsky/images/20211209.new/
pi@allsky:~/allsky/images/20211209.new $ /home/pi/allsky/scripts/removeBadImages.sh .
/home/pi/allsky/scripts/removeBadImages.sh: line 94: /usr/bin/which: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 144: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 145: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 154: /usr/bin/bc: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 155: [: -eq: unary operator expected
/home/pi/allsky/scripts/removeBadImages.sh: line 132: /usr/bin/egrep: Argument list too long
/home/pi/allsky/scripts/removeBadImages.sh: line 140: /usr/bin/bc: Argument list too long
^C
pi@allsky:~/allsky/images/20211209.new $ more /home/pi/allsky/scripts/removeBadImages.sh
#!/bin/bash
ulimit -s 65535

ME="$(basename "${BASH_ARGV0}")"

source "${ALLSKY_HOME}/variables.sh"
source "${ALLSKY_CONFIG}/config.sh"

Per your example I changed from bash to sh and it didn't fix much...

pi@allsky:~/allsky/images/20211209.new $ nano /home/pi/allsky/scripts/removeBadImages.sh
pi@allsky:~/allsky/images/20211209.new $ /home/pi/allsky/scripts/removeBadImages.sh .
/home/pi/allsky/scripts/removeBadImages.sh: 6: source: not found
/home/pi/allsky/scripts/removeBadImages.sh: 7: source: not found
/home/pi/allsky/scripts/removeBadImages.sh: 8: source: not found
jq: error: Could not open file : No such file or directory
/home/pi/allsky/scripts/removeBadImages.sh: 91: cannot create /removeBadImages.log: Permission denied
/home/pi/allsky/scripts/removeBadImages.sh: 93: typeset: not found
/home/pi/allsky/scripts/removeBadImages.sh: 109: [: -gt: unexpected operator
/home/pi/allsky/scripts/removeBadImages.sh: 176: [: -eq: unexpected operator
:  bad file(s) found and removed. See /removeBadImages.log.
pi@allsky:~/allsky/images/20211209.new $ sudo /home/pi/allsky/scripts/removeBadImages.sh .
/home/pi/allsky/scripts/removeBadImages.sh: 6: source: not found
/home/pi/allsky/scripts/removeBadImages.sh: 7: source: not found
/home/pi/allsky/scripts/removeBadImages.sh: 8: source: not found
jq: error: Could not open file : No such file or directory
/home/pi/allsky/scripts/removeBadImages.sh: 93: typeset: not found
/home/pi/allsky/scripts/removeBadImages.sh: 109: [: -gt: unexpected operator
/home/pi/allsky/scripts/removeBadImages.sh: 176: [: -eq: unexpected operator
:  bad file(s) found and removed. See /removeBadImages.log.
pi@allsky:~/allsky/images/20211209.new $

[EricClaeys]

@maphilli14 the Allsky scripts use some bash-specific commands like "source" so won't work with sh.

How many images are in 20211209.new?

[EricClaeys]

Setting a higher ulimit likely won't fix this as it's not stack-related. I just submitted a simple fix.

[EricClaeys]

@maphilli14, @jimjackii, please grab the latest scripts/removeBadImages.sh and let me know how it goes. It fixed the problem on my test machine.

@maphilli14, What do your log files say regarding the timelapse not working?

[maphilli14]

pi@allsky:~/allsky/images/20211209.new $ ls -1 | wc -l
13114

and it does seem to be at least running now, thanks! Any other changes to running for each image save?

[maphilli14]

Fixes to the removebadimages worked great! The genforday did not result in a proper timelapse....

ln -s /home/pi/allsky/images/20211209.new/image-20211210063956.jpg /home/pi/allsky/tmp/sequence-20211209.new/12758.jpg
ln -s /home/pi/allsky/images/20211209.new/image-20211210064000.jpg /home/pi/allsky/tmp/sequence-20211209.new/12759.jpg
ln -s /home/pi/allsky/images/20211209.new/image-20211210064008.jpg /home/pi/allsky/tmp/sequence-20211209.new/12760.jpg
pi@allsky:~/allsky/images/20211209.new $ cat /home/pi/allsky/tmp/timelapseTMP.txt | grep err
pi@allsky:~/allsky/images/20211209.new $ cat /home/pi/allsky/tmp/timelapseTMP.txt | grep ERR
pi@allsky:~/allsky/images/20211209.new $ 
pi@allsky:~/allsky/images/20211209.new $ less /home/pi/allsky/tmp/timelapseTMP.txt
pi@allsky:~/allsky/images/20211209.new $ /home/pi/allsky/scripts/generateForDay.sh t 20211209.new

================
If you want to upload the file(s) you just created, execute 'uploadForDay.sh  20211209.new'
================
pi@allsky:~/allsky/images/20211209.new $ /home/pi/allsky/scripts/generateForDay.sh -t 20211209.new
===== Generating Timelapse
/home/pi/allsky/scripts/timelapse.sh: line 128: 24965 Killed                  ffmpeg -y -f image2 -loglevel ${FFLOG} -r ${FPS} -i "${SEQUENCE_DIR}/%04d.${EXTENSION}" -vcodec ${VCODEC} -b:v ${TIMELAPSE_BITRATE} -pix_fmt ${PIX_FMT} -movflags +faststart $SCALE ${TIMELAPSE_EXTRA_PARAMETERS} ${TIMELAPSE_PARAMETERS} "${OUTPUT_FILE}" >> "${TMP}" 2>&1

*** : ERROR: ffmpeg failed.
Error log is in '/home/pi/allsky/tmp/timelapseTMP.txt'.

Links in '/home/pi/allsky/tmp/sequence-20211209.new' left for debugging.
Remove them when the problem is fixed.
[timelapseTMP.txt](https://github.com/thomasjacquin/allsky/files/7764863/timelapseTMP.txt)

Command Failed: '/home/pi/allsky/scripts/timelapse.sh' 20211209.new 

================
If you want to upload the file(s) you just created, execute 'uploadForDay.sh  -t 20211209.new'
================
pi@allsky:~/allsky/images/20211209.new $ 

[maphilli14]

timelapseTMP.txt

[EricClaeys]

@maphilli14 I'm 99% sure timelapse is failing due to memory or swap being too small. See this Wiki Page. The ffmpeg command that creates the video does a lousy job with memory. Let me know if the suggestions on the Wiki page helped.

[maphilli14]

@EricClaeys the default swap is 100 and I bumped to max 2048 and yes htop is much better! wish me luck!