New configuration problems

[blacc01]

Hello,

I have a new software configuration, which means the Raspberry OS is fully reinstalled, and also the AllSky software is brand new. Hardware (Raspberry 4B+ 2G, ZWO AS 120 MC, etc is old, not changed). Before reinstallation my system was working fine, just needed to upgrade as the software was old. Since this new configuration i have several problems. Its running since two days, but it stops sometimes, and at night only about the 12th or 13rd image is ok, others are only black. See the MP4 file of the last night here: http://[mcllserver.gotdns.com/allsky-20220608.mp4](http://mcllserver.gotdns.com/allsky-20220608.mp4)

I attach all the needed files, sorry, the log file is a little big, but i had to let it run as the problem is coming randomly.

allsky.log.txt config.sh.txt

pi@allsky:~ $ cd ~/allsky; git log | head commit fea3f2e03992b79e95b0845532482ce11af285ff Merge: 861e464 a6da0cf Author: EricClaeys 83164203+EricClaeys@users.noreply.github.com Date: Sun May 29 20:48:51 2022 -0700

Merge pull request #1214 from thomasjacquin/postData.sh-bug-fix

postData.sh: Bug fix: Allow to be run by WebUI

commit 861e464c237a1a8667ed8ad95b29d4fc1f19f6e6

[blacc01]

And again, it stopped at 9:24 at the morning... Last entry in the log is: Jun 9 09:23:57 allsky allsky.sh[1369]: > Sleeping 5.0 sec from histogram exposure Jun 9 09:23:57 allsky allsky.sh[1369]: > Saving DAY image 'image-20220609092356.jpg' Jun 9 09:24:02 allsky allsky.sh[1369]: STARTING EXPOSURE at: 2022-06-09 09:24:02 @ 175 us Jun 9 09:24:03 allsky allsky.sh[1369]: === next exposure=214 (multiply by 1.225) [last_exposure_us=175, minAcceptable=118, mean=106] Jun 9 09:24:03 allsky allsky.sh[1369]: === next exposure=263 (multiply by 1.225) [last_exposure_us=215, minAcceptable=118, mean=106] Jun 9 09:24:03 allsky allsky.sh[1369]: === next exposure=206 (multiply by 0.781) [last_exposure_us=265, maxAcceptable=134, mean=156] Jun 9 09:24:03 allsky allsky.sh[1369]: >xxx mean was 106 and went from 12 below min of 118 to 22 above max of 134, is now at 156; OK to set temp max to current_exposure_us of 265 Jun 9 09:24:03 allsky allsky.sh[1369]: === next exposure=167 (multiply by 0.781) [last_exposure_us=215, maxAcceptable=134, mean=156] Jun 9 09:24:03 allsky allsky.sh[1369]: > Sleeping 5.0 sec from histogram exposure Jun 9 09:24:03 allsky allsky.sh[1369]: > Saving DAY image 'image-20220609092402.jpg' Jun 9 09:24:08 allsky allsky.sh[1369]: STARTING EXPOSURE at: 2022-06-09 09:24:08 @ 215 us

No new entries...

However it says, that its running: pi@allsky:~ $ service allsky status ● allsky.service - All Sky Camera Loaded: loaded (/etc/systemd/system/allsky.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-06-09 08:34:29 CEST; 4h 40min ago Main PID: 696 (allsky.sh) Tasks: 7 (limit: 3720) CPU: 30min 47.318s CGroup: /system.slice/allsky.service ├─ 696 /bin/bash /home/pi/allsky/allsky.sh └─1369 /home/pi/allsky/capture -debuglevel 1 -tty 0 -aggression 75 -alwaysshowadvanced 1 -angle -6 -autousb 0 -a>

Jun 09 09:23:57 allsky allsky.sh[1369]: > Saving DAY image 'image-20220609092356.jpg' Jun 09 09:24:02 allsky allsky.sh[1369]: STARTING EXPOSURE at: 2022-06-09 09:24:02 @ 175 us Jun 09 09:24:03 allsky allsky.sh[1369]: === next exposure=214 (multiply by 1.225) [last_exposure_us=175, minAcceptable=118, m> Jun 09 09:24:03 allsky allsky.sh[1369]: === next exposure=263 (multiply by 1.225) [last_exposure_us=215, minAcceptable=118, m> Jun 09 09:24:03 allsky allsky.sh[1369]: === next exposure=206 (multiply by 0.781) [last_exposure_us=265, maxAcceptable=134, m> Jun 09 09:24:03 allsky allsky.sh[1369]: >xxx mean was 106 and went from 12 below min of 118 to 22 above max of 134, is now a> Jun 09 09:24:03 allsky allsky.sh[1369]: === next exposure=167 (multiply by 0.781) [last_exposure_us=215, maxAcceptable=134, m> Jun 09 09:24:03 allsky allsky.sh[1369]: > Sleeping 5.0 sec from histogram exposure Jun 09 09:24:03 allsky allsky.sh[1369]: > Saving DAY image 'image-20220609092402.jpg' Jun 09 09:24:08 allsky allsky.sh[1369]: STARTING EXPOSURE at: 2022-06-09 09:24:08 @ 215 us

[blacc01]

I still have randomly stops with my camera. When it stops capturing, there is no new entry in the log, allsky software seems to be running, only the last captured image is old. Other problem is also still there, sometimes the nightly shots are "blinking", like in the video in my first post. What could cause those problems? With my old configuration (old allsky) i never had this problems.

[EricClaeys]

@blacc01 This is the only case of this behavior I am aware of. Try the following:

ps -ef | grep capture
# You should get two lines, one very long.
# Get the first number in the long line.
strace -p FIRST_NUMBER

Let it run for a minute and post the output. This will tell me if the program is actively running and what it is doing.

Why do you think the software is running if there are no new entries in the log file?

I am investigating the nighttime flickering / blinking. I don't know why it didn't do that in the past but does it now.

[blacc01]

Hi Eric,

Thanks for your reply. The last day there was no stop in the software process, but as it will stop again, i'll do the steps you mentioned. Interesting is, that when there is no new captures, so it looks like the software is frozen, service status sasy its running, no errors. I can also go in the Web UI to the menus, only capture process is stopped, and also there is no new entry in the logs. But we will see after the next "stop". :)

Flickering problem: In the past there was no such "effect", but i'd used an old version of the AllSky, and also old Raspberri OS. Flickering was present in my last night video. BUT its not only in the video, but the captured jpegs are also affected. Here the video: https://youtu.be/HAsUZ0WANdE

And here an example JPEG which is dark (interesting that exposure is the same on both images):

And here the NEXT captured image which is ok:

Here the logs in the time when the two frames above was captured, maybe you see something. But i think there is something wrong with the timestamps. I see a lot of entries with starting exposure at 30sec, but the timestamps are only a few seconds.

Here some part of the log: Jun 14 22:13:06 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:06 @ 30.0 sec Jun 14 22:13:07 allsky allsky.sh[1305]: WARNING: time to take image (553863) differs from requested exposure time (29989578) by -29435715, threshold=14994789 Jun 14 22:13:07 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:07 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221306.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:08 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:08 @ 30.0 sec Jun 14 22:13:09 allsky allsky.sh[1305]: WARNING: time to take image (634893) differs from requested exposure time (29997394) by -29362501, threshold=14998697 Jun 14 22:13:09 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:09 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221308.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:10 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:10 @ 30.0 sec Jun 14 22:13:11 allsky allsky.sh[1305]: WARNING: time to take image (603262) differs from requested exposure time (29999348) by -29396086, threshold=14999674 Jun 14 22:13:11 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:11 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221310.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:12 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:12 @ 30.0 sec Jun 14 22:13:13 allsky allsky.sh[1305]: WARNING: time to take image (640067) differs from requested exposure time (29999837) by -29359770, threshold=14999918 Jun 14 22:13:13 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:13 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221312.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:14 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:14 @ 30.0 sec Jun 14 22:13:15 allsky allsky.sh[1305]: WARNING: time to take image (620886) differs from requested exposure time (29999959) by -29379073, threshold=14999979 Jun 14 22:13:15 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:15 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221314.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:16 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:16 @ 30.0 sec Jun 14 22:13:17 allsky allsky.sh[1305]: WARNING: time to take image (593645) differs from requested exposure time (29999989) by -29406344, threshold=14999994 Jun 14 22:13:17 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:17 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221316.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:18 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:18 @ 30.0 sec Jun 14 22:13:19 allsky allsky.sh[1305]: WARNING: time to take image (622165) differs from requested exposure time (29999997) by -29377832, threshold=14999998 Jun 14 22:13:20 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:20 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221318.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:21 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:21 @ 30.0 sec <--- START OF DARK IMAGE CAPTURE Jun 14 22:13:22 allsky allsky.sh[1305]: WARNING: time to take image (645140) differs from requested exposure time (29999999) by -29354859, threshold=14999999 Jun 14 22:13:22 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:22 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221321.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:23 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:23 @ 30.0 sec <--- START OF OK IMAGE CAPTURE Jun 14 22:13:52 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:13:52 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221323.jpg' <--- OK PICTURE Jun 14 22:13:53 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:53 @ 28.0 sec Jun 14 22:13:54 allsky allsky.sh[1305]: WARNING: time to take image (467150) differs from requested exposure time (28031250) by -27564100, threshold=14015625 Jun 14 22:13:54 allsky allsky.sh[1305]: > Sleeping: 3.0 sec Jun 14 22:13:54 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221353.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:57 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:57 @ 29.5 sec Jun 14 22:13:58 allsky allsky.sh[1305]: WARNING: time to take image (671491) differs from requested exposure time (29507812) by -28836321, threshold=14753906 Jun 14 22:13:58 allsky allsky.sh[1305]: > Sleeping: 1.5 sec Jun 14 22:13:58 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221357.jpg' <--- BAD, DARK PICTURE Jun 14 22:13:59 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:13:59 @ 29.9 sec Jun 14 22:14:00 allsky allsky.sh[1305]: WARNING: time to take image (620810) differs from requested exposure time (29876953) by -29256143, threshold=14938476 Jun 14 22:14:00 allsky allsky.sh[1305]: > Sleeping: 1.1 sec Jun 14 22:14:00 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221359.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:02 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:02 @ 30.0 sec Jun 14 22:14:02 allsky allsky.sh[1305]: WARNING: time to take image (462611) differs from requested exposure time (29969238) by -29506627, threshold=14984619 Jun 14 22:14:02 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:02 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221402.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:04 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:04 @ 30.0 sec Jun 14 22:14:05 allsky allsky.sh[1305]: WARNING: time to take image (615271) differs from requested exposure time (29992309) by -29377038, threshold=14996154 Jun 14 22:14:05 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:05 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221404.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:06 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:06 @ 30.0 sec Jun 14 22:14:07 allsky allsky.sh[1305]: WARNING: time to take image (592152) differs from requested exposure time (29998077) by -29405925, threshold=14999038 Jun 14 22:14:07 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:07 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221406.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:08 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:08 @ 30.0 sec Jun 14 22:14:09 allsky allsky.sh[1305]: WARNING: time to take image (646980) differs from requested exposure time (29999519) by -29352539, threshold=14999759 Jun 14 22:14:09 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:09 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221408.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:10 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:10 @ 30.0 sec Jun 14 22:14:11 allsky allsky.sh[1305]: WARNING: time to take image (596114) differs from requested exposure time (29999879) by -29403765, threshold=14999939 Jun 14 22:14:11 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:11 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221410.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:12 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:12 @ 30.0 sec Jun 14 22:14:13 allsky allsky.sh[1305]: WARNING: time to take image (644668) differs from requested exposure time (29999969) by -29355301, threshold=14999984 Jun 14 22:14:13 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:13 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221412.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:14 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:14 @ 30.0 sec Jun 14 22:14:15 allsky allsky.sh[1305]: WARNING: time to take image (603473) differs from requested exposure time (29999992) by -29396519, threshold=14999996 Jun 14 22:14:15 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:15 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221414.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:16 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:16 @ 30.0 sec Jun 14 22:14:17 allsky allsky.sh[1305]: WARNING: time to take image (648729) differs from requested exposure time (29999998) by -29351269, threshold=14999999 Jun 14 22:14:17 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:17 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221416.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:18 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:18 @ 30.0 sec Jun 14 22:14:19 allsky allsky.sh[1305]: WARNING: time to take image (614219) differs from requested exposure time (29999999) by -29385780, threshold=14999999 Jun 14 22:14:19 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:19 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221418.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:20 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:20 @ 30.0 sec Jun 14 22:14:49 allsky allsky.sh[1305]: > Sleeping: 1.0 sec Jun 14 22:14:49 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221420.jpg' <--- OK PICTURE Jun 14 22:14:50 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:50 @ 28.4 sec Jun 14 22:14:51 allsky allsky.sh[1305]: WARNING: time to take image (499830) differs from requested exposure time (28425000) by -27925170, threshold=14212500 Jun 14 22:14:51 allsky allsky.sh[1305]: > Sleeping: 2.6 sec Jun 14 22:14:51 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221450.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:54 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:54 @ 29.6 sec Jun 14 22:14:55 allsky allsky.sh[1305]: WARNING: time to take image (562864) differs from requested exposure time (29606250) by -29043386, threshold=14803125 Jun 14 22:14:55 allsky allsky.sh[1305]: > Sleeping: 1.4 sec Jun 14 22:14:55 allsky allsky.sh[1305]: > Saving NIGHT image 'image-20220614221454.jpg' <--- BAD, DARK PICTURE Jun 14 22:14:56 allsky allsky.sh[1305]: STARTING EXPOSURE at: 2022-06-14 22:14:56 @ 29.9 sec Jun 14 22:14:58 allsky allsky.sh[1305]: *** WARNING: time to take image (728105) differs from requested exposure time (29901562) by -29173457, threshold=14950781

Here the content of my settings: settings_ZWO.json.txt

And my config.sh: config.sh.txt

[EricClaeys]

@blacc01 Please increase your Debug Level to 4 so next time you attach the log file it will have more info. And please attach the file instead of copy/paste the contents which are hard to read on a phone.

The WARNING messages in the log file are basically saying "I asked for a 30-second exposure but only got a half-second one". I don't know what causes that but it results in a very dark image.

[blacc01]

Okay, sorry for that. Now i stopped allsky cleared the log file, set the debug level to 4 and atrted allsky. Wi will than see. I'll send result at the morning.

[blacc01]

So, there was no "stopping" of the server sincs that, but i have again and again the "flickering" effect. Here the video: https://youtu.be/x8nizTMNSUY

And here the full log with debuf level 4: allsky.log.20220616.txt

I was checked the timeframe starting from Jun 16 03:07:31. At this pont the picture was ok, and the picture from Jun 16 03:08:01 is ok aswell. The problem comes after that, the pictures from Jun 16 03:08:31, Jun 16 03:08:34, etc are "underexposed". What i see in the logs, when the picture is ok i see like this: xxxxxxx exposure_time_us=29999999, estimated timeToTakeImage_us=28622904

But when its dask, i see like this: xxxxxxx exposure_time_us=29100000, estimated timeToTakeImage_us=420007

Expose time "420007" which is about half a sec and not 29-30 secs. And of course there is a warning line below, like: *** WARNING: time to take image (420007) differs from requested exposure time (29100000) by -28679993, threshold=14550000

Why is this? In the previous version of AllSky software i never had this problem.

[blacc01]

Ok, now i got the allsky stopping problem. Now its 18:22, and on the Allsky WEB UI i see its showing the image capture from 17:17:19

The logging is stopped at the same time, at 17:20. Regarding systemctl status, the software is running: status.txt

And here attached the full log: allsky.log_20220616_1821.txt

Whats going on here?

When i go simply to the "Camera settings" menu and press "Save changes" (did not change everything), AllSky is running again.

[blacc01]

Today morning i saw, that the software was stopped yesterday evening at 21:40, with a big red error message on the Web UI, saying too much asi errors.. No change was made in the settings, so i don't know the reasons. Here the logs: allsky.log.20220617.txt

Is there any option that checks ASI_ERROR_TIMEOUT and if it reaches the number 5 it restarts the Allsky software and not stopping it? Because now no image was taken all the night, loosing one full night...

[EricClaeys]

@blacc01, any idea why the time went backwards after the 2nd line below, then jumped back up after the 5th line?

Jun 16 21:37:17 allsky allsky.sh[29070]:   > Camera set auto-exposure to 9.9 sec, timeout: 24817 ms
Jun 16 21:37:17 allsky allsky.sh[29070]:   > [Cleared buffer frame, next exposure: 9908697, auto=
Jun 16 21:17:08 allsky allsky.sh[704]:      ***** Starting AllSky *****
Jun 16 21:17:09 allsky allsky.sh[704]: CAMERA: ZWO
Jun 16 21:17:09 allsky allsky.sh[704]: CAMERA_SETTINGS: /etc/raspap/settings_ZWO.json
Jun 16 21:38:21 allsky allsky.sh[1269]: *** WARNING: Could not set locale to en_US.UTF-8 ***
Jun 16 21:38:21 allsky allsky.sh[1269]: **********************************************

Next time it hangs, please do the following BEFORE you restart Allsky:

ps -ef | grep capture
# You should get two lines, one very long.
# Get the first number in the long line.
strace -p FIRST_NUMBER

If there are 5 contiguous ASI_ERROR_TIMEOUT errors the program stops because if it kept going it would very likely keep getting more errors. However, if you have uhubctl installed and configured in config/config.sh then when too many ASI_ERROR_TIMEOUTs occur, it will reset the USB bus which often fixes the problem. See point 2 at this Wiki page.

Prior to the hang, things were going ok although there were several cases where the time to take an exposure was significantly less than the requested time. There were no ASI_ERROR_TIMEOUTs until after you restarted, and it never got a good frame. I highly suggest getting uhubctl.

Do you have a powered USB 3.0 hub? It's possible your issues are due to the camera not getting enough power from the Pi. Another thing to try is a more powerful power supply and/or thicker wire for the Pi.

[EricClaeys]

@blacc01 I forgot to mention. The WebUI just shows the last picture, even if that picture is extremely old. When you click on "Save changes", it restarts Allsky, even if you had no changes. That's why there's that checkbox that saves the changes but does NOT restart Allsky.

[blacc01]

@blacc01, any idea why the time went backwards after the 2nd line below, then jumped back up after the 5th line?

No didn't noticed that, just now as you mentioned it. Its strange, don't know the reason.

Next time it hangs, please do the following BEFORE you restart Allsky:

ps -ef | grep capture
# You should get two lines, one very long.
# Get the first number in the long line.
strace -p FIRST_NUMBER

okay, thanks, will do it so!

If there are 5 contiguous ASI_ERROR_TIMEOUT errors the program stops because if it kept going it would very likely keep getting more errors. However, if you have uhubctl installed and configured in config/config.sh then when too many ASI_ERROR_TIMEOUTs occur, it will reset the USB bus which often fixes the problem. See point 2 at this Wiki page.

Thanks will give a try.

Do you have a powered USB 3.0 hub? It's possible your issues are due to the camera not getting enough power from the Pi. Another thing to try is a more powerful power supply and/or thicker wire for the Pi.

No don't have. But before (with the older Allsky software) i did never had this problem. But i'll look and buy a powered USB 3.0 hub. It will be not easy to mount it in the box, as the space there is limited. A hub and also a power distributor for the two outlets... ehh.

[blacc01]

Prior to the hang, things were going ok although there were several cases where the time to take an exposure was significantly less than the requested time. There were no ASI_ERROR_TIMEOUTs until after you restarted, and it never got a good frame. I highly suggest getting uhubctl. Do you have a powered USB 3.0 hub? It's possible your issues are due to the camera not getting enough power from the Pi. Another thing to try is a more powerful power supply and/or thicker wire for the Pi.

No i could get a powered USB HUB, but not a 3.0 than a 2.0. But i think its okay, as my Asi 120MC is a USB 2.0 version (black USB connector on the side, not the blue one). I installed it, now we will see if its helps or not.

About the uhubctl: i checked the link, readed more time, but did not really understand it. I have not a smart USB hub, so it cannot be controlled (switched off/on by PI). Or have to setup for the PI's USB connectors? In that case which connector is which? The USB hub is connected to the USB 3.0 connector, the lower one. But if the software resets the PI's USB hub, that will not reset the ASI CAM, as it is powered now from the USB HUB. Another question. If i know the port which should be resetted, how can i implement it in the AllSky so, that if 5 ASI_ERROR_TIMEOUTS are coming, it should reset or power off than on the desired port?

[blacc01]

Some new info. Till now there was no allsky stopping, and found only one ASI_ERROR_TIMEOUT event in the log: 20220618_asierror.txt

Flickering still there, interesting, that i see flickering only between 22:00-22:20 and 03:15-03:50, see video: https://youtu.be/XdDUO4tCBFU

[EricClaeys]

@blacc01 Let's watch it over the next couple days and if you only get an occasional ASI_ERROR_TIMEOUT and no hangs, then we can probably assume the problem was due to insufficient power to the camera.

I have the following in my config.sh file. My camera is on a USB 3 port on the Pi.

# Specify the path to the command and the USB bus number (on a Pi 4, bus 1 is USB 2.0 and
# bus 2 is the USB 3.0 ports).  If you don't have 'uhubctl' installed set UHUBCTL_PATH="".
UHUBCTL_PATH="/home/pi/uhubctl/uhubctl"
UHUBCTL_PORT=2

The uhubctl command will control hubs in many different devices, including the USB hubs in a Pi 2B, 3B, 3B+, and 4B. The Pi Zero isn't listed so I don't know if it'll work. This has nothing to do with the USB 2.0 powered hub you just added. To use uhubctl, copy the 10 or so files to a directory on your Pi. I created /home/pi/uhubctl and put my files there. Then cd into that directory and type make. That will create the executable. Then update config.sh.

To see what USB port your camera is on:

cd /home/pi/uhubctl    # or where ever you put it
sudo ./uhubctl

Look for the line with your camera. I get:

pi:~/uhubctl $ sudo ./uhubctl
Current status for hub 2 [1d6b:0003 Linux 5.15.45-v7l+ xhci-hcd xHCI Host Controller 0000:01:00.0, USB 3.00, 4 ports, ppps]
  Port 1: 02a0 power 5gbps Rx.Detect
  Port 2: 0203 power 5gbps U0 enable connect [03c3:290b ZWO ASI290MM]
  Port 3: 02a0 power 5gbps Rx.Detect
  Port 4: 02a0 power 5gbps Rx.Detect
Current status for hub 1-1 [2109:3431 USB2.0 Hub, USB 2.10, 4 ports, ppps]
  Port 1: 0100 power
  Port 2: 0100 power
  Port 3: 0100 power
  Port 4: 0100 power
Current status for hub 1 [1d6b:0002 Linux 5.15.45-v7l+ xhci-hcd xHCI Host Controller 0000:01:00.0, USB 2.00, 1 ports, ppps]
  Port 1: 0507 power highspeed suspend enable connect [2109:3431 USB2.0 Hub, USB 2.10, 4 ports, ppps]

So my cameras in on hub 2, port 2. The port number is what you put in config.sh.

Re: flickering: I'm struggling with how to fix this since I don't understand what's causing the problem. Can you attach the log file with the 22:00-22:20 and 03:15-03:50 flickering?

[EricClaeys]

@blacc01 It looks like most (or all?) the flickering is due to the light in the house going on and off. When it's dark outside and the lights go on, that frame will be very bright. When the lights go off the next frame will be dark. Have you looked at each picture to see what's happening?

[blacc01]

@blacc01 It looks like most (or all?) the flickering is due to the light in the house going on and off. When it's dark outside and the lights go on, that frame will be very bright. When the lights go off the next frame will be dark. Have you looked at each picture to see what's happening?

@EricClaeys I've checked that, but its nothing todo with the light in the house, as it is not changed during that time, so its "static". Another interesting thing, that even on the dark frames the onlay text says 30secs of exposure time, which is not the truth. There are some events when the light changes in the house (for example at 22:28:04, when additional lights were going on), but it does not affect the exposure time. Another interesting thing, when the "flickering effect" is "on" than every 13th or 14th frame is ok, others are black.

[blacc01]

And just see that since i use this new version of Allsky my Startrails images are not correct. I mean it does not have trails. All of my startrails are like the attached images, however there was no rain, no moon, only fine starry images all the nights. So there should be created a good strartrail image of them. Maybe the "flickering" images affect this also.

Startrails image:

Here the keogram :

[blacc01]

Next time it hangs, please do the following BEFORE you restart Allsky:

ps -ef | grep capture
# You should get two lines, one very long.
# Get the first number in the long line.
strace -p FIRST_NUMBER

Today Allsky stooped again, so i did what you mentioned. Its NOT the ASI_ERROR_TIMEOUT problem, just the "pausing" of Allsky! Before restarting, i do de strace, as below. But after FUTEX_BITSET_MATCH_ANY it was only waiting, did nothing: allsky-strace-01.txt

Than simple did a save config in the Web UI, so the AllSky was restarting. Inmediately after the save i saw that in the cli the waiting strace command continued from FUTEX_BITSET_MATCH_ANY. See: allsky-strace-02.txt

Maybe that will help to get why Allsky sometimes stopping, or better like the process is paused. :)

[blacc01]

Last night nearly the full night was "flickering"... Pictures so unuseable... Here the keogram, you can see it:

[EricClaeys]

@blacc01 Please try the attached file. It probably won't fix the flickering problem but may give me more information about what's happening. I can't explain why some exposures are only taking a half second when they should be taking much longer.

To use this file:

cd ~/allsky/src
mv capture.cpp capture-ORIGINAL.cpp
# copy the attached file here, removing the ".txt" from the name
make capture
systemctl stop allsky
cp capture ..
sudo truncate -s 0 /var/log/allsky.log
systemctl start allsky

Wait until you get the flickering, then attach the log file.

Regarding the stopping/pausing: The capture program has a separate piece of it that waits until a new image is captured, the it writes the image to a file. That can happen the same time the main part of the capture program is doing something else. The separate piece uses "futex()" to determine when a new image is captured. The strace showed that the futex() is waiting for 1655708282 seconds (about 52 years), so if you wait long enough, it would continue :-)

Next time it stops/hangs, would you please

ps -ef | grep capture
# Get the first number in the long line.
strace -p FIRST_NUMBER
gdb --pid=FIRST_NUMBER

Thanks. capture.cpp.txt

[blacc01]

@EricClaeys thanks. I did what you wrote, but i get error:

pi@allsky:~/allsky/src $ make capture
2022-06-21 15:54:07 Building capture program...
capture.cpp: In function ‘ASI_ERROR_CODE takeOneExposure(int, long int, unsigned char*, long int, long int, ASI_IMG_TYPE, int*, int*)’:
capture.cpp:511:29: error: ‘exposureTime_us’ was not declared in this scope; did you mean ‘exposure_time_us’?
  511 |    if (timeToTakeImage_us < exposureTime_us && exposureTime_us > (5 * US_IN_SEC))
      |                             ^~~~~~~~~~~~~~~
      |                             exposure_time_us
make: *** [Makefile:132: capture] Error 1

Of course before that i moved top a backup cpp file the original one, than moved the file you sent to the empty capture.cpp file, and started make, which is failed.

[EricClaeys]

@blacc01 Sorry about that. I'm running a much newer version in test mode so didn't have a good way to compile the version I sent. Please try this updated version.

FYI, when you have multiple lines you want to highlight, you can put three tildas (~~~) on a line, then add the lines you want to highlight, then add another line with three tildas. It formats code correctly, whereas back ticks don't. I edited your previous reply.

capture.cpp.txt

[blacc01]

Thnks @EricClaeys, now the file was ok, could "make" it. We will see.

[EricClaeys]

@blacc01 You may want to take new dark frames. You have a LOT of noise and hot pixels in the images you attached above. The ASI120 is notorious for having lots of dark pixels, but yours are really bad. New darks should help a lot.

[blacc01]

@blacc01 You may want to take new dark frames. You have a LOT of noise and hot pixels in the images you attached above. The ASI120 is notorious for having lots of dark pixels, but yours are really bad. New darks should help a lot.

Yes i agree. But i think i need to have let the system keep darks for a whole night to get darks for all temperatures. Or not?

[EricClaeys]

@blacc01 The larger a range of temperatures you have, the better, but you don't need every temperature. Some people have had good luck with putting the camera in the freezer until its frozen, then taking it out and taking darks. If you can do that you probably don't need to take darks all night since you'll already have the colder darks. You'll somehow want to get darks when it's warmer. Maybe heat the camera then take darks until the camera is at ambient temperature.

[blacc01]

Thanks! Freezer is a good idea!

[blacc01]

I think my ZWO camera CCD is not the best, as the dark images have a lot of bad pixels (see image below):

But soon i'll change it to an RPi HQ cam, and hope i'll get better results.

[EricClaeys]

@blacc01 It looks like there are more bad pixels than good ones! The RPiHQ camera is pretty good, especially for the money. It's the camera I recommend to most people now.

[blacc01]

Next time it stops/hangs, would you please

ps -ef | grep capture
# Get the first number in the long line.
strace -p FIRST_NUMBER
gdb --pid=FIRST_NUMBER

@EricClaeys Today i saw that the Allsky stopped again (no errors, just waiting as before. It was "stopped" at 13:54.

Here the outputs of the commands you asked for. 20220624_stop.txt

Maybe you can find it why its just waiting.

[blacc01]

@blacc01 It looks like there are more bad pixels than good ones! The RPiHQ camera is pretty good, especially for the money. It's the camera I recommend to most people now.

@EricClaeys yeah. :( I think my ZWO is weared down. Maybe its not worth to take dark images, better to wait for the new RPiHQ cam and go ahead with it.

[EricClaeys]

@blacc01 Unfortunately you stopped the "capture" program while doing the strace but before doing the gdb command, so the program no longer existed and gdb didn't produce anything. Next time, don't bother with the strace since it'll produce the same output as the first two times. Just do the gdb, and then stop the process.

The noise in the images would drive me nuts so if it were my camera I'd take some darks, even if only for an hour.

[blacc01]

@EricClaeys since that no "stopping" seen, but the "flickering" effect is still there.

Today the RPi HQ CAM, the lense, adaptor, cables are received, so i think i can go and test this. Some question about that:

1. Do i need to reinstall AllSky, or just change some settings? Which should be changed to what?
2. Should i remove IR filter from the CCD or just leave it there?

I think thats all for now.

[EricClaeys]

@blacc01 From an Allsky perspective you only need to change CAMERA="RPiHQ" in config/config.sh, then restart. However, you may need to configure Linux to use enable the camera. The Wiki has a couple pages on the RPiHQ camera - I suggest you read them. Because the RPiHQ's sensor is fairly large, timelapse may fail. I set

TIMELAPSEWIDTH=2028
TIMELAPSEHEIGHT=1520

in config/config.sh to get around this. My images are still full-size, but the timelapse is 1/2 size, which is fine since I don't have a 4K monitor.

Are you ok with closing this Issue since the original problem no longer applies?

[blacc01]

@EricClaeys Thanks for the infos. But i suggest to wait with the topi closure until i installed new cam and have a night capture with it. I'll let you know the results. Thanks

[blacc01]

As ZWO camera is changed to RPiHQ Cam as the problems was not really (possible the ZWO cam is defect), we can close this thread. I have new problems now with the RPIHQ CAMm but for that i opened another thread. Thanks @EricClaeys for you big-big help. ;)

[EricClaeys]

@blacc01 My pleasure. I think you'll like the RPiHQ MUCH better than the 120!.

Do you live in the US? If so, and you'd like to sell your 120, I'd like to buy it as a test camera. A lot of people have problems with them that I can't reproduce.

[blacc01]

@EricClaeys Last night i tried a lot of settings, and as result i have the following video (the flickering in the video is because i set a lot of time the settings): https://youtu.be/mFuS2iaGwBY

I think thats the maximum i can get out from this system. Would be better to get more Milky Way, more stars, better contrast. Maybe the light pollution is too much. I don't know.

Here my stratrails:

What i don't like is the keogram picture. Its so narrow, i dont know why. How could i get a more wider (and not soooo long) image?

Here my settings:

[blacc01]

Okay, for my keogram question i got the answer on your wiki. Sollution is the --image-expand option. Now the image is much better!!

[EricClaeys]

@blacc01 I am glad you are making good progress

I can see lots of stars on my observatory camera in Bortle 2 skies but hardly any at home in the suburbs of Chicago. You may be in the same situation and the software can't really help with that.

I would prefer if you looked in the Wiki before asking questions. I am trying to improve it and need to know what information is confusing or missing.

Eric

[blacc01]

Yes, its possible what you say. So i think thats the maximum i can get here. Thanks for your help.

[EricClaeys]

@blacc01 If you run into a problem not in the Wiki, please either enter a new Issue if you think it is a bug, or add a Discussion item. If we determine it actually is a bug we can then create a new Issue. If you find something missing from the Wiki os something confusing, add a Discussion item so I can fix it.

Enjoy your allsky camera!