search

aaronwmorris / indi-allsky

Software to manage a Linux-based All Sky Camera.
GNU General Public License v3.0
216 stars 35 forks source link

Stopped working at midnight #1113

Closed mshetzer closed 6 months ago

mshetzer commented 7 months ago

Love this application. Its great to see what is going one outside when its freezing out, and I want to image. Thanks for putting sure a huge project together.

Everything was working great yesterday until midnight. This morning, I am unable to get the latest image to produce. I have many errors in my log. See below. Any help is greatly appreciated !!! Matt

Jan 26 08:50:51 allsky [WARNING] MainProcess-5624/MainThread miscDb.addNotification() #780: Not adding existing notification Jan 26 08:50:51 allsky [WARNING] MainProcess-5624/Upload-1 uploader.saferun() #117: Goodbye Jan 26 08:50:51 allsky [INFO] MainProcess-5624/MainThread allsky._fileUploadWorkerStop() #532: Stopping Upload worker Jan 26 08:50:51 allsky [WARNING] Video-1-5634/MainThread video.saferun() #173: Goodbye Jan 26 08:50:51 allsky [WARNING] Video-1-5634/MainThread video.sigterm_handler_worker() #127: Caught TERM signal Jan 26 08:50:51 allsky [INFO] MainProcess-5624/MainThread allsky._stopVideoWorker() #457: Stopping Video worker Jan 26 08:50:51 allsky [INFO] MainProcess-5624/MainThread allsky._stopVideoWorker() #454: Terminating Video worker Jan 26 08:50:51 allsky [WARNING] MainProcess-5624/MainThread allsky.run() #550: Shutting down Jan 26 08:50:43 allsky [WARNING] Capture-1-5631/MainThread capture.saferun() #353: Goodbye Jan 26 08:50:43 allsky [INFO] Capture-1-5631/MainThread indi.serverDisconnected() #350: Server disconnected (exit code = 0, localhost, 7624 Jan 26 08:50:43 allsky [WARNING] Capture-1-5631/MainThread indi.disableCcdCooler() #900: Cooling not supported Jan 26 08:50:41 allsky [WARNING] Capture-1-5631/MainThread indi.disableCcdCooler() #895: Disabling CCD cooling Jan 26 08:50:41 allsky [WARNING] Capture-1-5631/MainThread capture.saferun() #347: Shutting down Jan 26 08:50:41 allsky [WARNING] Video-1-5634/MainThread video.sigterm_handler_worker() #127: Caught TERM signal Jan 26 08:50:41 allsky [WARNING] Capture-1-5631/MainThread capture.sigterm_handler_worker() #148: Caught TERM signal Jan 26 08:50:41 allsky [WARNING] MainProcess-5624/MainThread allsky.sigterm_handler_main() #190: Caught TERM signal, shutting down Jan 26 08:50:41 allsky [WARNING] MainProcess-5624/MainThread allsky.sigterm_handler_main() #190: Caught TERM signal, shutting down Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread stretch.mode1_apply_gamma() #65: Applying gamma correction Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread stretch.main() #43: Using image stretch mode 1 Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread stretch._generateNumpyMask() #189: Using central ROI for blob calculations Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread stretch._generateNumpyMask() #175: Generating mask based on SQM_ROI Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread processing._calculate_8bit_adu() #835: ADU average: 64373.3 (4023) Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread processing._generateAduMask() #2576: Using central ROI for ADU calculations Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread processing._generateAduMask() #2561: Generating mask based on ADU_ROI Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread image.processImage() #373: Image: 1296 x 976 Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread sqm.calculate() #42: Weighted SQM average: 755344.67 Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread sqm.calculate() #37: Raw SQM average: 64559.87 Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread sqm._generateSqmMask() #63: Using central ROI for SQM calculations Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread sqm._generateSqmMask() #48: Generating mask based on SQM_ROI Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread sqm.calculate() #21: Exposure: 0.001000, gain: 101 Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread processing._calibrate() #742: Dark not found: ccd1 16bit 0.0010000s gain 101 bin 1 20.50c Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread processing._calibrate() #723: Temperature matched dark not found: 20.50c Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread processing._calibrate() #705: Searching for dark frame: gain 101, exposure >= 0.0, temp >= 20.5c Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread processing._calibrate() #693: Bad Pixel Map not found: ccd1 16bit 0.0010000s gain 101 bin 1 20.50c Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread processing._calibrate() #674: Temperature matched bad pixel map not found: 20.50c Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread processing._calibrate() #656: Searching for bad pixel map: gain 101, exposure >= 0.0, temp >= 20.5c Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread processing._detectBitDepth() #604: Overriding bit depth to 12 bits Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread processing._detectBitDepth() #602: DETECTED BIT DEPTH (16) IS DIFFERENT FROM CONFIGURED BIT DEPTH (12) Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread processing._detectBitDepth() #571: Image max value: 65535 Jan 26 08:50:38 allsky [INFO] Image-37-6123/MainThread processing.add() #485: Image bits: 16, cfa: RGGB Jan 26 08:50:38 allsky [INFO] MainProcess-5624/MainThread allsky._queueManualTasks() #998: Checking for manually submitted tasks Jan 26 08:50:38 allsky [WARNING] MainProcess-5624/MainThread processing._load_detection_mask() #2471: No detection mask defined Jan 26 08:50:38 allsky [INFO] MainProcess-5624/MainThread allsky._startImageWorker() #349: Starting Image-37 worker Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: IndexError: index 31681 is out of bounds for axis 0 with size 4096 Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: gamma_data = lut.take(data, mode='raise') Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: File "/home/mshetzer/indi-allsky/indi_allsky/stretch.py", line 82, in mode1_apply_gamma Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: data = self.mode1_apply_gamma(data, image_bit_depth) Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: File "/home/mshetzer/indi-allsky/indi_allsky/stretch.py", line 52, in mode1_stretch Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: return self.mode1_stretch(data, image_bit_depth), True Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: File "/home/mshetzer/indi-allsky/indi_allsky/stretch.py", line 44, in main Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: stretched_image, is_stretched = self._stretch.main(self.image, self.max_bit_depth) Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: File "/home/mshetzer/indi-allsky/indi_allsky/processing.py", line 2416, in stretch Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: self.image_processor.stretch() Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: File "/home/mshetzer/indi-allsky/indi_allsky/image.py", line 482, in processImage Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: self.processImage(i_dict) Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: File "/home/mshetzer/indi-allsky/indi_allsky/image.py", line 267, in saferun Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: self.saferun() Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: File "/home/mshetzer/indi-allsky/indi_allsky/image.py", line 238, in run Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() #342: Image worker exception: Traceback (most recent call last): Jan 26 08:50:35 allsky [INFO] Capture-1-5631/MainThread capture.saferun() #327: Exposure received in 0.6039 s (0.6029) Jan 26 08:50:35 allsky [INFO] Capture-1-5631/MainThread capture.saferun() #409: Total time since last exposure 10.0365 s Jan 26 08:50:35 allsky [INFO] Capture-1-5631/MainThread indi.set_number() #1302: Setting Duration (s) = 0.0010000000474974513 Jan 26 08:50:34 allsky [INFO] Capture-1-5631/MainThread capture.shoot(

aaronwmorris commented 7 months ago

Based on your previous tickets and the resolution, I am assuming you are using a ASI224MC.

Jan 26 08:50:38 allsky [WARNING] Image-37-6123/MainThread processing._detectBitDepth() DETECTED BIT DEPTH (16) IS DIFFERENT FROM CONFIGURED BIT DEPTH (12)

I would set your camera bit depth back to Auto Detect or 16-bit. Even though the IMX224 is a 12-bit camera, the ZWO drivers pre-scale the data up to 16-bits. The instructions are not clear in config and I will update the text.

Jan 26 08:50:38 allsky [ERROR] MainProcess-5624/MainThread allsky._startImageWorker() Image worker exception: IndexError: index 31681 is out of bounds for axis 0 with size 4096

I am not exactly sure what caused this, but I suspect it might be due to the bit depth setting above. 4096 is max value for 12 bits.

mshetzer commented 7 months ago

That worked perfectly. Thank you.

I have it currently running now testing it, and it is in Moon Mode. Not sure why. Any ideas?

aaronwmorris commented 7 months ago

You mentioned midnight in the first post, but the times in the logs are 8 hours after midnight. Is the time and time zone correct on the system?

mshetzer commented 7 months ago

The 8 hours are when I got up, and looked at the system. The whole system was down and I could not connect, and I had to restart it. Not sure what the cause was (Ubuntu system), but I'll see if it happens again tonight.

Matt

aaronwmorris commented 7 months ago

I recommend setting up the watchdog. Even if you are not using a Raspberry Pi, this can work for you.

https://github.com/aaronwmorris/indi-allsky/wiki/Raspberry-Pi-Watchdog

mshetzer commented 7 months ago

Aaron, I had an error in my location. Forgot the '-' in my longitude.