Error: one bright and one good image in alternation

[Tartouffe]

Fresh install today of v0.8.1 .. now got one bright and one good image in alternation for the first few images after reboot. Then its fine for some images, but when a darker or little brighter one follows, it restarts again with these extreme exposures. Seems like the auto exposure value calculated from every image (physically stored somewhere?) is going nuts.

Hint: i use a -very- large resolution with a 20 MPx camera (ASI183MC / 5496*3672 px) .. exceeding any limits?

Any idea how to fix or which setting causes this?

[EricClaeys]

@Tartouffe in the WebUI, please set the Debug Level to 3, turn On Show histogram (you may need to first click on the Show advanced settings box at the bottom), turn on Show histogram box, and unless you want to see the gain and brightness, turn them off. Now, before clicking the Save changes button, in a terminal window run sudo rm /var/log/allsky.log. NOW click the Save changes button and let the software run until you have sever back and forth frames.

At that point, attach the /var/log/allsky.log file (do not copy/paste it).

[Tartouffe]

Hi Eric,

thanks for the quick reply. It seems that the Histograms hits 254/254 for a short amount of time on the bright images. Gain and Brightness were okay (0 and 50). Historgram value of a following good one is e.g. '43'. Attached is the log and one cropped info part of the bright ones. If you need more data or i could test something?

--

Also i noticed a second bug within this 0.8.1. version: i set the night images (like at the older version) to 75 seconds (75000). But the longest exposure which was taken were 60s. For a quick overview over my settings --> see screenshot

--

Third one (just seen at the logs a minute ago): There is a slash trimming missing at the ftp upload (double slash) "upload.sh: Copying image.jpg to /home/SkyCamWeb/allsky//image.jpg"

Thanks :)

allsky.log

[EricClaeys]

@Tartouffe, ZWO limits auto exposures to 60 seconds. I don't know why. That's why it's not going past 60 seconds even though you set Max Exposure to 75.

[Tartouffe]

Eric, that can't be possible. a) at version 0.6 or 0.7 i had set the exposure (from start on) to 75s; at the beginning of my AllSky installation i tested it up to 120s, but then got visible star trails all above 80s. Works fine for months with the 75s and there were no limitation on any older version. b) every ZWO camera is a specialized 'Astronomy camera'. You can be sure its a MUST to be able to shoot unlimited time. Like my other Zwo camera (ASI2600MC) attached the ZWO ASIAir Raspi Linux computer which i had set for my astrophotos up to 600s (normally shooting around 450s) with the telescope.

So the 60s limit must be somewhere else applied new i think, between version 0.6/0.7 and 0.8.

---

Something other: Did the directory for the timelapse videos change? The Timelapse video is placed at the daily pictures directory, not (anymore) collected at the /videos dir (without daily separated subdirs). At my ftp-settings.sh 'VIDEOS_DIR' is set correct (.../allsky/videos , like main dir .../allsky; images at ../allsky/images) .

At the endofnight script is says about copy file to UPLOAD_FILE = ${DATE_DIR}/Videofilename. (line 79) I think $date_dir has to be removed with the absolute path from the dir variable at the ftp-settings?!

[EricClaeys]

@Tartouffe the limit only applies to auto exposure, not manual exposure. To see the camera capabilities set the Debug Level to 4 and restart allsky, then look at /var/log/allsky.log. Near the top of the new run's messages it lists all the capabilities including their min, max, and default values. You want the on called something like MaxAutoExposure.

I'll check on your second item, but in version 0.7 and 0.8 the timelapse video goes in ~/allsky/DATE/allsky-DATE.mp4. There is no ~/allsky/videos directory. I don't know if one existed in version 0.6. If you have allsky-website installed there is a /var/www/html/allsky/videos directory. Is this what you are referring to? Where is your .mp4 file?

[Tartouffe]

@EricClaeys

the limit only applies to auto exposure, not manual exposure.

Aaah. So when i turn autoexposure off i can use it up to 75s? ...wondering myself. Made screenshots of the settings screen before i made the update and there were autoexposure ON - and the 75s images worked. But when its only this option ... easy to switch off to get my 75s night images again. :)

Videos-Dir Had to check the file / rights and access first. Talking about the default /videos dir on the web server. Sorry not for mention this.

---

Did you find anything at the/my log about the jumping histogram value and too bright images? There must be some kind of variable amnesia 😂

[EricClaeys]

@Tartouffe, your ASI183 camera goes up to 2000 seconds in manual exposure mode.

If you are using the Pi as the webserver the "videos" directory should be /var/html/www/allsky/videos, and that location didn't change (but you need to set it to something in the newest ftp-settings.sh).

I haven't looked at the log yet.

[EricClaeys]

@Tartouffe, forgot to ask, did you display the histogram box on your images. If the Sun or Moon go through the box it'll throw off the brightness so you may need to move the box. I had to do that with my MC178MC.

[Tartouffe]

Oh thanks, good tip. Did not know that this switch now exists. Had switched it on, but didn't see a "box", but can set it by pixel value to a centered position. Had to do this by day when the bright/normal pictures happened - rarely to none happended at night. Playing with the parameter - could be outside the sky, because of the smaller 185 degrees view of my cam.

[EricClaeys]

@Tartouffe, the histogram box only appears during daytime since that's when the histogram exposure algorithm runs. Probably when you first turned it on it was nighttime. By default it's centered on the chip and is 500 pixels by 500 pixels big. If you crop the image, for example to get rid of the black on the right side, the histogram box won't look centered on the image because the center of the chip isn't centered in the image anymore.

You can also play with the size. When we implemented the box we all had 3000-ish pixel sensors. My ASI224 has about 1000-ish pixels wide so I had to shrink the box. In the future we'll change the size to be a percent of the sensor, rather than an absolute pixel size.

[EricClaeys]

@Tartouffe, you edited your message from October 17 - it used to have additional questions and issues. Do you still have those questions and issues?

[Tartouffe]

you edited your message from October 17 - it used to have additional questions and issues. Do you still have those questions and issues?

No, because of complete new install. Old thread can be closed (or should I ?)

By default it's centered on the chip and is 500 pixels by 500 pixels big. If you crop the image, for example to get rid of the black on the right side, the histogram box won't look centered on the image because the center of the chip isn't centered in the image anymore.

Doing no cropping. I leave the cam-pic as it is. Also no dark image usage.

I played around with the histo box, placed it to a spot where nearly no sky changes the brightness (see picture). Still the same. One image has a histo of 255, the next following a minute later has good exposure, and so on. There must be somewhere an uninitialized variable or smth...

[EricClaeys]

Hi @Tartouffe,

It would help me if you don’t edit your messages, other than for trivial things like fixing a misspelled word. I get confused when I see messages like what’s below in my email, but when I go to the website I see a somewhat different message. I prefer if you simply create a new message when you have new information. Thanks.

The histogram box and associated code was added in version 0.8 to work around a bug in the ZWO daylight exposure where it would jump all over the place and a picture with a long exposure could be darker than one with a shorter exposure (like in your pictures – 0.10 ms is brighter than 0.25 ms). The new algorithm worked better on all the cameras we tested. Clearly there is still an issue with the new histogram algorithm since some people still have images that vary widely in brightness for no apparent reason. Or, the ZWO bug is still appearing even though the histogram algorithm turns auto exposure off).

Is the green rectangle where the histogram box initially was, or where you moved it for testing? Where is it now?

Eric

[Tartouffe]

It would help me if you don’t edit your messages

Yes sorry. I misinterpreted the value at the histo box; read it too quick and thought it would be "x, y", then box "width and height", not percent. So my first post was rubbish. I will keep posted onesin future.

--

Back to the flashing problem: As seen two posts before, the yellow/green box is my new place of the histo box (manually drawn onto the camera shot, because it newer shows a rectangle or box with 'show histo box'.

I placed it at an mostly non brightness changing area (90% of the box at a part of the roof) to see, if the flashes are still this extreme (histo value up to 255). And they were. So my conclusion is, that the value is random or uninitialized, rather than a wrong read out or calculation. When the (remaining 10%) over the clouds changes only a little (even not really noticeable) the flashing is extreme as before, some pictures are totally white. Followed by - next day-picture in cueue a minute later - a perfect illuminated one, how it should. I placed the box more to the left too (X = under 1500 px), always the same result. So the box was not apparently outside of my used bigger image dimensions or something like that, where a random histogram value can be explained by using a camera with much higher reslution as usual.

[Apennine2]

Hello,

I experienced this same issue after upgrading to 0.8.1 yesterday. I was able to fix the issue by setting Version 0.8 exposure to "Yes".

[EricClaeys]

@Tartouffe, would you please try that attached capture file? Rename it to "capture.cpp" and move it to the allsky/src directory. Then

cd src
make capture
sudo service allsky stop
cp capture ..
sudo truncate -s 0 /var/log/allsky.log
sudo service allsky start

Let it go a couple minutes, then attach the log file. The version of "capture" has some changes that might help, but also has some more debugging info. Thanks.

capture.cpp.txt

[Tartouffe]

I will try immediately after work hours. Previously, should i reset to the default histogram values (1000 1000 50 50)?

[Tartouffe]

Hi Eric,

here is the data you need. File was compiled and i let it run for ~20min. Did you change something? Got -no- bright image with this compilation.

I noticed another hicup: some pictures are repeating themself after one minute - only a small pixel shift to the left can be noticed, but no cloud movement. Possible because of your 'debug-code'? My day settings are: every 59s one picture. See animation over the logged 20min and keep an eye on the time code top left.

I noticed a or this (?) stutter only on day images! since the 0.8+ version before, but couldn't verify this, because of the bright images with no content between. The Timelapse seems to stutter, wasn't smooth at run. This only happened on day pictures, not at the night. But with this compilation i can see that there is some kind of data hicup.

Hope it helps to locate/identify allsky.log .

[Tartouffe]

Additional information:. Still got these bright images, also with the compiled version. As the sun came out later and the overall brightness increases, the bright/while images in exchange with normal exposed ones came back. Possible that this info locates you more to the point in the code where...

Cheers.

[EricClaeys]

@Tartouffe, I suggest setting the histogram settings to whatever works best. Height and width should be around 15-20% of the sensor size. You don't want the Sun or Moon to go through the box. Having the box on your house may be a good idea since it probably doesn't change as much in brightness when clouds are out, versus having the box on the sky where clouds come and go.

[Tartouffe]

...had to reworke that. Even with the histo box >80% on a place where the sun didn't alter a lot of brightness, when the sun came back yesterday i got the white images between again. And the stutter (see animated GIF two posts before). There must be something out of bounds (small variable who then goes into negative...?) when a value was exceeded. Happened only at day on mid or bright weather conditions. Also the stutter (double image) does not happen at night. Since 0.8x did not have any timelapse or day sequence who can be shown. Looks like stroboscope psycho movie. Weird.

[Tartouffe]

Attached the log, when the bright images happen (yesterday it was too cloudy/no sun). Same settings as yesterday, around 20min runtime. Also take a look at the animated GIF, how it looks - one picture every 60s.

allsky.log

[jsanchez73]

Hello,

I am having the same issue here. Any updates on how to fix it?

[EricClaeys]

@jsanchez73 try the version of capture.cpp we just uploaded. If that still gives problems let me know (and include the log file). I'm working on a version that dampens the changes by implementing an "aggression" setting. Foe example, it will only make half the expected change in brightness rather than the full change. I'm also trying to detect wild swings in brightness and deal with them.

[Tartouffe]

Trying out the updated capture.cpp. But it is (very) bad weather. Had to wait for sun, to reach some brightness level. Could go a few days until I have a result. (Note: version 0.8 expure has the same problem)

[EricClaeys]

@Tartouffe, are you saying version 0.8 is causing your bad weather????? We are working on a cloud-removal filter :-)

Are you running with the version exposure mon ON or OFF?

Selse reported good results with 0.8 exposyee ON, but I realize not everyone can run with it ON.

[Tartouffe]

Sure, since version 0.8 the weather is always bad. Hope to see the sky clearance filter in action 😂

--

If you mean "daytime auto exposure ON', yes, that is how my daytime settings were set (including version 0.8 exposure OFF). If i should test with other parameters, tell me with which.

But after a day runtime, it still produces those bright images between, even on nearly no brightness change at the pictures on (still) very rainy and cloudy day. The newly compiled version does not fix this issue, sorry.

[EricClaeys]

@Tartouffe sorry for all the misspellings in my last post. It should have said: "Are you running with the version 0.8 exposure mode ON or OFF?

Someone else reported good results with 0.8 exposure ON, but I realize not everyone can run with it ON."

You just answered the question about version 0.8 exposure mode - you said it is OFF. Can you run with it ON? Or does that give ASI_ERROR_TIMEOUTs?

[Tartouffe]

"Version 0.8 Exposure" was OFF, now ON.

Had tested this (set ON) with the version before, no difference. Lets see what the latest capture.cpp will produce over the day tomorrow with now switched on.

[EricClaeys]

@Tartouffe, please increase your Debug Level to 3 in the WebUI. It will give me much-needed information, like what was happening with the dark/bright images.

The "stutter" in your GIF is because some images were being skipped due to ASI_ERROR_TIMEOUTs even though you are using the older version 0.7 exposure algorithm. Try decreasing your USB Bandwidth to 40.

The strangest image is the one that doesn't include the overlay. I have no idea how that can happen.

[Tartouffe]

First debug level 3 run with the parameters: USB Bandwidth: 40 / debug Level 3 / Version 0.8 Exposure: YES A too bright image was shot at timecode 15:17:59. Only one. I had to run it on brighter sky (when your "clear sky automation" is working error free ;) ).

I noticed when a bright image was produced, an image in the set Intervall is missing! E.g. every 1min one picture, the bright image is 2 minutes old/er, the one-minute-before it is missing. After the bright one, the time Intervall gets a few seconds longer. But also keep an eye on the time Intervall of the Images. But had to reproduce/prove this on better conditions. For now the sky makes no sense. For the next run at brighter weather i will set it to 60000ms per image for better calculation, wheather there is really a frame before the bright one missing. Then it could be a cumulation of some values or variables.

Attached the log. Timecode of the wrong one: see above.
Meanwhile: waiting for better sky.

allsky.log

[Tartouffe]

Eric, you know what? NOT A SINGLE overexposed picture since the modified parameters 3 day ago. I think the default USB value of 80 (if i remember right?) causes these bright images. Now with 40 the images are how they should be. Also no frames were missing. Had to prove this on bright weather. Will keep you updated (waiting for bright sky/sunshine).

[EricClaeys]

@Tartouffe, that's great to hear.

The brightness of the sky appears to have been changing quite a bit. The mean brightness of the picture before 15:17:59 was 81 which is below the acceptable range, then the 15:17:59 picture's mean was 129 which is inside the acceptable range. The next several pictures had mean of 124 - 134 which are all in the allowed range. If those pictures were way too bright you may want to move the histogram box to include more of the sky.

[EricClaeys]

@Tartouffe, is it still working well? If so, would you please close this issue? Thanks.

[Tartouffe]

Sunny weather didn't happend these days, so i can't proof it by 100%. But with reduced USB bandwidth no more brigthness issues. It has to be that. Thanks for the patience identifying this issue. :)

So if you got some too bright images, reduce the USB bandwidth. I will close here.