v0.8 - timelapse creation failed

[Jonk2]

The first night / morning after updating to 0.8, a timelapse was produced no problem. I noticed that it was heavily compressed and quite blocky in places, so I looked at how to improve this - my first thought was the bitrate.

In the timelapse script the bitrate is set to 2MB/s so I changed this line and only this line from :-b:v 2000k to :-b:v 10000k \ #bitrate default 2000k

This morning, there's a keogram etc but no timelapse.

In the log, there's nothing about the timelapse, it goes from taking the final night image to the first day image. The only message is 'Processing end of night data'. I could be wrong but on v0.7 wasn't the entire process of timelapse creation recorded in the log? I don't have a copy of the previous log to check but I'm sure I've seen it?

I'm going to ssh into the pi and see if I can run the script manually before I set the bitrate back to 2000k, but the main question I suppose, is what is the maximum bitrate I can set? Is there another was to improve the timelapse quality? I'm happy with a lossless video, space / bandwidth etc. isn't an issue.

From past experience, 20MB/s is a good bitrate for high def video files, but this timelapse is near 3K resolution (whatever the image resolution is).

[Jonk2]

Hmmm...running the script manually has returns this error:

I'm stuck now so will return the timelapse script to default (re download the file from github) and see what happens.

[Jonk2]

I've replace the timelapse.sh containing my small modification, with the default as per the original file (from 0.8).

Running manually I get the same error message as above.

I've also compared the scripts from v0.7 and v0.8 and the first few lines are the same, so I'm confused as to why it fails at the first line?

I suppose I could put the 0.7 timelapse.sh file in and see if that runs manually?

[Jonk2]

D'oh - I've just learnt that python3 is for .py files, not .sh.....

It needs to be made an executable sudo chmod +x timelapse.sh then run manually using sh timelapse.sh

Ignore me, I'm off to hide in shame!

But before I do that...

This is now the error it returns...

As you can see the directory does exist and it's full of last night's images:

So what next?

[jcauthen78]

Looks like you're close. Give this a go:

terminal cd allsky ./scripts/timelapse.sh 20210807

i've had issues runing it manually from the scripts folder in the past, found it needs to be run from the root of the Allsky folder. Let me know how that goes for you.

[Jonk2]

Ok I did what you suggested and it worked - the timelapse file was created.

I changed the bitrate in the script from 2000k to 10000k and ran it again...

...and it worked fine, resulting video a lot better (size approximately 45MB vs 8MB). I'll try maybe 20MB/s too, giving approx 100MB filesize.

I will also check tomorrow to see if it automatically creates one for tonight's images and report back, but manually, yes it seems to do as it should, even copying the video into the 'website location', in my case a local NAS.

Thanks for the tip.

[EricClaeys]

Jon, Running it as "sh timelapse.sh" runs it in the standard shell ("sh"), which doesn't know about the "source" command. Had you done "bash timelapse.sh" it would have worked, but no need to prepend the "bash" since the first line of the file is "#!/bin/bash" which tells the operating system to run the file using /bin/bash. "sh" also doesn't know about the "-en" option to "echo", which is why you saw "-en" in the log file as well.

At the end of every night, the "endOfNight.sh" script is run. It will write an entry to the log for every sub-command it executes, depending on what options you have in config.sh. You should see:

endOfNight.sh: Generating Keogram
endOfNight.sh: Processed 1000 keogram files       (or whatever number were on your system)
endOfNight.sh: Generating Startrails
eendOfNight.sh: Processed 1000 startrails files.  Summary:
Minimum .....
endOfNight.sh: Generating Timelapse
timelapse.sh: Creating symlinks to generate timelapse
timelapse.sh: Created 1000 links total                 (or whatever number were on your system)
timelapse.sh: Deleting sequence
timelapse.sh: Timelapse was created

What's tricky, is that there will likely be lots of other log entries between the entries above, so you may need to search for the script name in the log file (endOfNight.sh, timelapse.sh, etc.). These log entries could span dozens of minutes, depending on how many images you take.

Please look again in your log file for the timelapse.sh messages and let me know if you find them.

Here's what I plan to do to allow the user to specify the bitrate. In timelapse.sh change the line "-b:v 2000k \" to "-b:v ${TIMELAPSE_BITRATE:-2000k} \" In config.sh, add:

# Set the bitrate of the timelapse video.  Higher numbers will produce higher quality but larger files.
   TIMELAPSE_BITRATE="2000k"

Then change the 2000 in config.sh to whatever you want.

Let me know how that goes. Eric

[bbillp]

Would this error be USB related ? I upgraded the night of 8/7/3021, entered my ZWO ASI178 camera and Location, double checked the location then captured no images.
Log.txt shows Starting Night Time Capture Saving auto exposed night image ERROR Failed getting image ……. Status 11 ASI Error Timeout scripts/copy_notification_image.sh: Permission Denied

[jcauthen78]

It's a permissions error with one of the newer updated files that somehow didn't get the correct permissions on install.

Open the terminal

sudo chmod u+x /home/pi/allsky/scripts/copy_notification_image.sh

Not sure if the 'u' is needed, but it works for me so I've just run with it. 😅

That should get the permissions error fixed and let the rest of the script continue as normal.

[EricClaeys]

The permissions error was fixed yesterday so anyone getting the new version won't see the problem. For the "timeout" message, try increasing your USB Bandwidth from 40 to, for example, 90. Let me know if that fixes the problem. Eric

[Jonk2]

Eirc,

I've looked through the log again and this is what I find (I've removed lots of lines in between, take a note of the timestamps for the end of night.sh related lines):

Aug 8 05:46:08 allsky allsky.sh[20548]: Processing end of night data Aug 8 05:46:08 allsky allsky.sh[20548]: endOfNight.sh: Removing bad images

Aug 8 05:59:10 allsky allsky.sh[20548]: removeBadImages.sh: No bad files found.

This took 13 minutes, nothing to worry about there I guess?

Next...

Aug 8 05:59:10 allsky allsky.sh[20548]: endOfNight.sh: Generating Keogram Aug 8 06:04:18 allsky allsky.sh[20548]: endOfNight.sh: Processed 1063 /home/pi/allsky/tmp/keogramTMP.txt keogram files

This took 5 minutes. Again, nothing to worry about.

Next...

Aug 8 06:04:18 allsky allsky.sh[20548]: endOfNight.sh: Generating Startrails Aug 8 06:08:28 allsky allsky.sh[20548]: endOfNight.sh: Processed 1065 /home/pi/allsky/tmp/startrailsTMP.txt startrails files. Summary: Aug 8 06:08:28 allsky allsky.sh[20548]: Minimum: 0.112967 maximum: 0.386757 mean: 0.197037 median: 0.176065

This took 2.5 ish minutes. good, no problem.

Then...

Aug 8 06:08:28 allsky allsky.sh[20548]: timelapse.sh: #033[0;32mCreating symlinks to generate timelapse#033[0m Aug 8 06:08:31 allsky allsky.sh[20548]: STARTING EXPOSURE at: 2021-08-08 06:08:31 Aug 8 06:08:32 allsky allsky.sh[20548]: > Saving image 'image.jpg' that started at 2021-08-08 06:08:31 (11 µs) Aug 8 06:08:32 allsky allsky.sh[20548]: > Sleeping from histogram exposure: 5000 ms Aug 8 06:08:32 allsky allsky.sh[20548]: timelapse.sh: Created 1063 links total Aug 8 06:08:32 allsky allsky.sh[20548]: [NULL @ 0x18238b0] Unable to find a suitable output format for ' #bitrate' Aug 8 06:08:32 allsky allsky.sh[20548]: #bitrate: Invalid argument Aug 8 06:08:32 allsky allsky.sh[20548]: ./timelapse.sh: line 76: -pix_fmt: command not found Aug 8 06:08:32 allsky allsky.sh[20548]: ./timelapse.sh: line 77: -movflags: command not found Aug 8 06:08:32 allsky allsky.sh[20548]: #033[0;31m*** timelapse.sh: ERROR: ffmpeg failed with RET=127 Aug 8 06:08:32 allsky allsky.sh[20548]: Links in '/home/pi/allsky/images/20210807/sequence' left for debugging. Aug 8 06:08:32 allsky allsky.sh[20548]: Remove them when the problem is fixed.#033[0m

This clearly shows the timelapse creation failing, after all I did was change the bitrate and comment that the original was 2000k.

Was the # comment the issue? i.e. my fault?

:-b:v 10000k \ #bitrate default 2000k

If so, great, this can be closed. If not?

[EricClaeys]

Jon, Yes, the "#bitrate comment is the problem. Put it above or below the ffmpeg command. The "\" tells the shell that the next line is part of the current line. The errors below #bitrate are a result of the #bitrate error.

The other output is exactly as expected. Removing bad frames is pretty slow, which is why there's a comment in config.sh saying it's slow. Other than taking time and CPU, it doesn't hurt though.

Eric

[EricClaeys]

Jon, Did you change your USB bandwidth, and if so, did it fix the problem? What did you set it to?

[Jonk2]

I 'may' have set it to auto after the timeout issue was noticed, I don't remember but I haven't had the issuse since I reported. I don't really want to set it back to the default 40 now it's working just in case, but I will if it helps?

The only issue I've had since regarding the actual camera, is 2 frames last night, one was pure purple and the next was pure green:

I have seen this before, so not sure what would cause it. Bandwith maybe, could be a bug in the ASI driver.

I'll close this issue about the timelapse as that is now sorted.

[EricClaeys]

Jon, If your USB Bandwidth is working, leave it. Mine's at "auto" and I don't remember seeing a problem in a while, so I'm not going to change. I will put in an update to set the default to "auto" and "80".

I get a purple picture once or twice a day it seems. I'll have to check if I've seen one at night. Haven't seen a green one (yet). I'd really like to figure out a way for "removeBadImages.sh" to remove them. I assume it's a bug in the ASI driver. I may try setting auto-white balance and see if that works, but I think the last time I tried auto balance I didn't like the colors.