Possible FFMPEG error causes INDI-Allsky to fail after a few hours

[SubBass100]

I use an 4k IP camera and retrieve snapshots via directions from the wiki. I've installed on both fresh Ubuntu install as well as raspberry PI's with the same results. Everything appears to work and system is runs for a few hours (the time to failure is inconsistent). Upon looking through the logs I see that the camera exposure is BUSY. If I follow the log back long enough to where it is no longer BUSY, I can see an FFMPEG error that reads like this:

Oct 4 21:45:58 allsky [INFO] Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:57: [INFO] FFMPEG Error while clearing buffer: End of file. Oct 4 21:45:58 allsky [INFO] message repeated 1012 times: [ Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:57: [INFO] FFMPEG Error while clearing buffer: End of file. ] Oct 4 21:45:58 allsky [INFO] Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] FFMPEG Error while clearing buffer: End of file. Oct 4 21:45:58 allsky [INFO] message repeated 830 times: [ Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] FFMPEG Error while clearing buffer: End of file. ] Oct 4 21:45:58 allsky [INFO] Image-2/MainThread detectLines.detectLines() #78: Line detection in 0.1278 s Oct 4 21:45:58 allsky [INFO] Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] FFMPEG Error while clearing buffer: End of file. Oct 4 21:45:58 allsky [INFO] Image-2/MainThread detectLines.detectLines() #81: Detected 0 lines Oct 4 21:45:58 allsky [INFO] Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] FFMPEG Error while clearing buffer: End of file. Oct 4 21:45:58 allsky [INFO] message repeated 283 times: [ Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] FFMPEG Error while clearing buffer: End of file. ] Oct 4 21:45:58 allsky [INFO] Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] Buffer Cleared of 2538 stale frames. Oct 4 21:45:58 allsky [INFO] Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] FFMPEG Error: -541478725, End of file. Oct 4 21:45:58 allsky [INFO] message repeated 9 times: [ Capture-39/Dummy-5 indi.newMessage() #333: new Message 2023-10-05T01:45:58: [INFO] FFMPEG Error: -541478725, End of file. ]

The Keo and Startrails do generate so I think the system fails after or during that process but I'm at a loss why it keeps failing. Seeing how no one else on the forums is having this issue, I'm assuming it's because I'm using an IP camera / snapshot grab. I had a guess but no luck. My guess was that every once and a while the camera might not respond with a snapshot or a partial file? which breaks FFMPEG? to see if I could avoid this, I created a simple script that downloads an snapshot from the camera (or even just use the same image over and over) then places it in a directory Indi Allsky then would retrieve it from that location to avoid latency, however even with the file readily available it still fails after a few hours. I really like what the software can do, Does anything stand out as something I can do to correct this issue? Thank you in advance!

[aaronwmorris]

These messages are due to you see are related to the indi_webcam_ccd server, which is why nobody else is complaining. The ffmpeg errors are not related to generating videos. When indi_webcam_ccd downloads the remote image, it is actually using ffmpeg to download the image. This activity does not occur within the indi-allsky side, but in the indiserver.

I have seen issues where indi_webcam_ccd/indiserver would run out of memory. That is a possibility.

You may consider setting up a cronjob to stop indi-allsky, restart the indiserver, then start indi-allsky every 1-2 hours.

[SubBass100]

Thank you for the quick response, my current build is running in a vm running Ubuntu with 4cpu/4gb ram. I will try increasing the ram to 8gb. I'll also research indi_webcam_ccd for related issues on google while I am waiting to see if there is an error. I'll let you know how it goes, thanks

[SubBass100]

Sorry for the delay, I've ran some more tests with 8gb ram, it does look like your assumptions are correct with running out of memory. After about two hours almost all 8GB ram had been consumed, shortly after it crashed. I was out and about so I wasn't able to see if the swap had filled before it crashed.

Is there any other driver I can use? Is there a way to use wget in place of ffmpeg to grab?

Some additional things I've tried are converting the images to lower resolution incase the 4k images were somehow consuming more space (no change)

Based on a discussion forum here I've tried using webm instead of h.264, however, it didn't seem to make a difference either.

I'm still pursuing indi-webcam-ccd , To note the issues from: https://www.indilib.org/individuals/devices/cameras/indi-webcam.html

One known issue with this driver is that sometimes stale frames will be left in the buffer when starting a new image or a new live video stream. I added a buffer flushing method based on the buffer timeout to try to combat this issue, but it can still happen sometimes. Try setting the buffer timeout to change how stale frames are detected.

Another known issue is that sometimes FFMPEG randomly disconnects from the device. This could be due to a network interruption, error in the stream, or other issue. I have written the driver to try to detect these disconnections and then reconnect so that there is little to no interruption in imaging, but sometimes it does mess up an image or two. You can also hit the connect button to reconnect as well.

Hope this sheds some light, Thank again and if you would like me to try anything in particular please let me know and I'll drop it into my testing routine.

[aaronwmorris]

I will have to write a new camera "driver" to support this. It is on my todo list.

[SubBass100]

Thanks for your help, as a temporary solution I'll try your previous suggestion until a fix (new driver) is available.

You may consider setting up a cronjob to stop indi-allsky, restart the indiserver, then start indi-allsky every 1-2 hours.

[SubBass100]

I've created a bash script that can easily restart the two services you mention and made it executable. When I run it, it magically clears the memory and everything seems happy... I've been manually doing this for the last few hours and it indi-allsky runs well.

My script simply runs: `

!/bin/bash

systemctl --user restart indi-allsky systemctl --user restart indiserver `

I went to add it to crontab and for what ever reason it will not run. There is no error in syslog about it, just nothing. Do I need to define the location of the service or something similar? Do you know of a better way to restart indi-allsky and indiserver every hour? Thanks!

[aaronwmorris]

I am going to keep this issue open so I can notify you when the new camera input is available.

[SubBass100]

Thanks, Just for completeness in case anyone else finds this thread and tries to restart the services every hour.... It looks like although restarting those services keeps the system in a running state, It may break startrails and keogram. I'm not 100% sure if restarting indi-allsky and indiserver every hour kills those processes but mine are not rendering and showing broken pages where before I started cycling the services it would create an image and video up to the point it failed.

[aaronwmorris]

As of merge #991 I have added a new "camera" to download images from a web camera or any remote web server. You can select pyCurl Camera as the camera type and at the bottom of the Camera config tab, you can enter the URL to download the image.

I have not performed extensive testing and the implementation is pretty basic right now. Let me know how this works for you.

[SubBass100]

Great! I am excited to try this out. I recompiled fresh and selected pycurl.

I made sure that pyCurl Camera was selected at the top. I left INDI Camera Name blank I added my camera snapshot url to "pyCurl Camera URL" at the bottom I left INDI Camera Configuration at its defaults { "SWITCHES": {}, "PROPERTIES": {}, "TEXT": {} }

Here are a few observations: Status says Running in green Above the status where it would normally say something like indi_webcam_ccd, it is blank. no images shows on "Latest" tab My camera snapshot url ends in /cgi-bin/snapshot.cgi browsers usually just take this cgi and render it as a jpg. just in case pycurl was unhappy with this, I wrote a script to grab the cgi and copy it to a file.jpg and place that jpg into the static flash directory where I can give pycurl a valid url to download from. This also did not work. I'm heading off to bed but wanted to submit my findings asap!

[aaronwmorris]

I may need to see a few entries from your log file to track this down.

[SubBass100]

[INFO] Capture-3-2982/PycurlCamera-2 pycurl_camera.run() #59: Camera URL: http://someuser:thepassword@cameraip/cgi-bin/snapshot.cgi

Interesting, I just checked the logs with some rested eyes and can see an auth problem to the camera. The url I use contains a username and password in the url using this format "someuser:thepassword@" ](http://someuser:thepassword@cameraip/cgi-bin/snapshot.cgi) I'm not sure if my camera supports anonymous snapshots, I can check on that now.

[aaronwmorris]

Okay, looks like I need to add authentication support.

[aaronwmorris]

Merged #997 with authentication support. Just move the username and password from the URL to the form fields and let me know how it works.

[SubBass100]

Awesome! It works, I'm going to let it run for a while and keep an eye on the memory Thank you!

[SubBass100]

I'm happy to report it made it through the night, I can see the cache building and clearing. Keogram, Startrails and timelapse generated.

Thanks you!

[SubBass100]

I don't mean to post here @aaronwmorris but I tried to send you an email to your listed email address as it was the only way to PM you.

[aaronwmorris]

Closing this for now. Let me know if I can do anything else.