Nightly Segmentation Fault

[amiles347]

I've a recurring segmentation fault. It happens nightly but at different times:

● allsky.service - All Sky Camera
   Loaded: loaded (/lib/systemd/system/allsky.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-09-24 06:24:27 BST; 2h 45min ago
  Process: 833 ExecStart=/home/pi/allsky/allsky.sh (code=exited, status=139)
 Main PID: 833 (code=exited, status=139)

Sep 23 15:17:10 allsky systemd[1]: Started All Sky Camera.
Sep 24 06:24:27 allsky allsky.sh[833]: /home/pi/allsky/allsky.sh: line 25:   932 Segmentation fault      ./capture $ARGUMENTS
Sep 24 06:24:27 allsky systemd[1]: allsky.service: Main process exited, code=exited, status=139/n/a
Sep 24 06:24:27 allsky systemd[1]: allsky.service: Unit entered failed state.
Sep 24 06:24:27 allsky systemd[1]: allsky.service: Failed with result 'exit-code'.

settings.json:

{"width":"0","height":"0","exposure":"10000","maxexposure":"20000","autoexposure":"1","gain":"50","maxgain":"200","autogain":"0","gamma":"50","brightness":"50","wbr":"53","wbb":"90","bin":"1","delay":"10","daytimeDelay":"5000","type":"1","quality":"95","usb":"40","filename":"image.jpg","flip":"0","text":"text","textx":"15","texty":"30","fontname":"0","fontcolor":"255 255 255","fonttype":"0","fontsize":"0.7","fontline":"1","latitude":"51N","longitude":"2W","angle":"-6","time":"1","darkframe":"0","showDetails":"1"}

config.sh:

#!/bin/bash
source /home/pi/allsky/scripts/ftp-settings.sh

# Set to true to upload current image to your website
UPLOAD_IMG=true

# Set to true to upload timelapse video to your website at the end of each night
UPLOAD_VIDEO=true

# Set to true to send data to your server at the end of each night
POST_END_OF_NIGHT_DATA=false

# Set to true to generate a timelapse at the end of the night
TIMELAPSE=true

# Set to true to generate a keogram at the end of the night (image summary of the night)
KEOGRAM=true

# Set to true to upload the keogram after it has been generated
UPLOAD_KEOGRAM=true

# Set to true to generate a startrails image of the night. Will skip brighter images to avoid over-exposure
STARTRAILS=true

# Images with a brightness higher than the threshold will be skipped for startrails image generation
BRIGHTNESS_THRESHOLD=0.1

# Set to true to upload the startrails after it has been generated
UPLOAD_STARTRAILS=true

# Set to true to enable automatic deletion of archived data (images + videos)
AUTO_DELETE=true

# Set this value to the number of archived nights you want to keep. Needs AUTO_DELETE=true to work
NIGHTS_TO_KEEP=14

# Path to the dark frame for hot pixels subtraction. Can be jpg or png.
DARK_FRAME="dark.png"

# Set to 0 to disable Daytime Capture
DAYTIME="1"

# Path to the camera settings (exposure, gain, delay, overlay, etc)
CAMERA_SETTINGS="/var/www/html/settings.json"

Night 1 log file (Crashed early on in the night):

Saving image-20180922220443.jpg

Resizing image.jpg

Uploading image-resize.jpg

13828109
Saving image-20180922220454.jpg

Resizing image.jpg

Uploading image-resize.jpg

20000000

Night 2 log file (Appears to crash on transition from night to day. However, no end of night data generated):

Saving image-20180924062407.jpg

Resizing image.jpg

Uploading image-resize.jpg

20000000
Resizing image.jpg

Uploading image-resize.jpg

Using an ASI224MC on a Pi 3 B+. Could it be an unexpected exit from the ftp server causing some issue?

Andy

[thomasjacquin]

Hi Andy, I am using the same camera and it works fine on my side so we'll need to investigate a bit deeper. Did you fill the info in ftp-settings.sh? Currently, if you have UPLOAD_IMG set to true but ftp-settings.sh not filled, it will create a lot of lftp threads and it can kill the service.

Thomas

[amiles347]

Yes - ftp is working. Last night, I reduced the image size for ftp transfer and it didn't crash. So I think you have almost the right diagnosis. Each ftp transfer was overlapping until they eventually got too many and killed the service. However, wouldn't that produce an over memory limit rather than a seg fault?

[thomasjacquin]

I had a similar issue with lftp when it snowed a lot and the wifi connection got really bad. I have seen some segfaults as well during timelapse creation. I believe the segfault happens when one of the threads dies unexpectedly (lack of memory for example). I will have to do some research to see if it can be handled more nicely.

[amiles347]

Not a fix but you can use systemd to restart the service should it go down for any reason:

[Service] User=pi ExecStart=/home/pi/allsky/allsky.sh StandardOutput=null StandardError=syslog SyslogFacility=local5 Restart=on-failure

Andy

[thomasjacquin]

Thanks Any, that's a good idea. For some reason, I have a similar version on my camera but forgot to commit the changes.

Restart=always
RestartSec=5

I read something about the fact that if it tries to restart too many times with 10 seconds, it won't start at all. That's why I added the delay.

[EricClaeys]

This issue is from a much older release of the AllSky software. If you think it might still be relevant, please test it with the newest version and submit a new issue if needed. Users have not reported this problem in a while.