search

pageauc / pi-timolo

Raspberry PI-TIMOLO ( PI-TImelapse, MOtion, LOwLight ) uses RPI picamera and OpenCV for Remote Headless Security Monitoring using Motion Tracking, Rclone Auto Sync files with remote storage services. Auto Twilight Transitions and Low Light Camera Settings. Panoramic images using PanTiltHat and More. This project is featured on GitHub Awesome software.
MIT License
548 stars 101 forks source link

Time-lapse photos archiving #78

Closed ragnarjw closed 6 years ago

ragnarjw commented 6 years ago

Claude,

I cannot find a way to archive time-lapse photos which have been used by the video creation script. There is a setting to delete them, but not to save to e.g. a datestamped directory. Did you consider this?

pageauc commented 6 years ago

There is a feature in makevideo.conf to copy completed video to a network share or it could just be another directory. 

# Settings to move output video to mnt folder (share should be mounted to) 
# ------------------------------------------------------------------------
tl_share_copy_on=false         # default=false true moves video to mnt point network share per mount location below
tl_share_destination=$DIR/mnt  # A valid network share mount point to copy video to
# IMPORTANT - Make sure share is mounted or you will have files copied to the mount folder
#               This will prevent mounting of share until the files in the folder are moved/deleted.

I put this in to save room on the local drive but if you have enough room on the local storage device, you can use this to move completed videos to another local archive folder. Each video file is date/time stamped so there should not be a naming conflict, so it would not be archive to a datetime stamped folder unless you plan on having a lot. In that case there would need to be additional logic to create a new folder when a specified number of files is reached or some other trigger event. Current logic does not create this folder so you would need to make one before hand. Since makevideo.sh is just a bash script additional commands could be added by a user but I would suggest renaming the makevideo.sh script to another name so an update would not overwrite the customization.

See wiki for additional details here https://github.com/pageauc/pi-timolo/wiki/How-to-Make-a-Timelapse-Video-from-Images Claude ...

ragnarjw commented 6 years ago

Thanks Claude, this deals with the video created. I was thinking of the photos in the timelapse directory. As it is now, either you delete these photos (an action which you note in the config file could leave the user without the "raw material" for the video, or the video just becomes longer and longer every time it is created.

pageauc commented 6 years ago

OK now I understand. When a video is made there is a temporary folder created with soft links created to the actual image files to be included in the video. These links are named in a number sequence for required by avconv. I believe it would be possible to read the link meta data and move the actual image files to an archive location before deleting the symbolic links. Claude ...

pageauc commented 6 years ago

readlink -f returns the full path to the file pointed to by a symbolic link. This can be used to walk through all the temporary symbolic links and move the original image files to a specified archive location. Not sure if you want to give this a try. It would be a great learning opportunity for you. Let me know

see more detail here https://unix.stackexchange.com/questions/22128/how-to-get-full-path-of-original-file-of-a-soft-symbolic-link

Claude ...

ragnarjw commented 6 years ago

Thanks. I'll take a look at this.

pageauc commented 6 years ago

It is still a good learning exercise to try to do this. I have posted my solution on the github repo You can download the changes per

wget -O makevideo.sh https://raw.github.com/pageauc/pi-timolo/master/source/makevideo.sh
wget -O makevideo.conf https://raw.github.com/pageauc/pi-timolo/master/source/makevideo.conf

two new variables 

tl_archive_source_files=false  #default=false true= archive (mv) source image files to a destination folder
tl_archive_dest_folder="$DIR/mnt/archive"  # Destination folder to move source images after encoding complete

probably should have named first variable tl_archve_enable but already tested so it is what it is.

Note you will have to enable archiving and set a destination folder

Although I have tested I would like your feedback on testing results as well

Claude ...

ragnarjw commented 6 years ago

Ah, OK. I made my own solution which works well here at least, and must admit I haven't tested yours.

Wasn't quite sure why you wanted to use readlink on the temporary directory, those symlinks are made to keep avconv happy, I thought. The original source files are all there in the time-lapse directory, you could just move all of them instead. It doesn't really matter, either way works. One thing to keep in mind is that mv does not preserve timestamps, so I used cp with the -p switch and rm in order to keep the timestamps. The time-lapse photo file names do not contain the timestamp, so good to preserve them this way.

I called my variables, and they are exactly the same logic as yours:

tl_move_to_archive=true|false tl_archive_folder=$DIR/tl-archive

My code is a bit more elaborate than yours. If you want to take a look, they are at

wget -O newmakevideo.sh https://rawgit.com/ragnarjw/pylearn/master/newmakevideo.sh wget -O newmakevideo.conf https://rawgit.com/ragnarjw/pylearn/master/newmakevideo.conf

A couple of questions.

What are the first lines after #--- Start Script ---# meant to do? I cannot find any utils_conf, and the script output just says it is a directory.

Also, I wonder if there is a typing error in the log file description? The variable is tl_error_log_file=$DIR"/makevideo_error.log", but all the error messages go to $error_log_file

ragnarjw commented 6 years ago

After testing, I would comment

The archive folder could be created by the script if it does not exist.

I prefer having a folder per day for the source photos, but that is of course a matter of taste and could be another modification - choose a big folder to hold all, or organize in separate dated folders.

If a lot of source photos, could be a lot of lines if you echo each file.

I am a little confused by the timestamp issue, but it could be that since I have the photos and videos all mounted on a Samba share as per your docs, it is CIFS playing games. I found this to work around it: https://andrewelkins.com/2015/04/02/move-files-while-preserving-timestamps/

pageauc commented 6 years ago

The files in the temp folder are the ones that have been encoded and since it takes time to process there most likely will be more files saved in the images folder that are not part of the current video encoding job. This was a bit of a bug in the previous version of makevideo.sh. The latest version corrects this by using readlink to only encode/archive/delete the file snapshot at the time per the temp folder symbolic links.

Yes I thought about auto creating the folder but decided to error out in case of a misspelling or similar issue. btw it would be pretty easy to add option to autocreate date/time stamped archive subfolders in the archive and only error out if the main archive folder is missing. I just did a quick solution.

I agree the echo for each file can be a little much but thought it would show progress. I could

Sometimes you have to decide when something is good enough and move on. but improvements to anything are always possible.

Timestamp can be a problem when files are moved copied across a network or other storage device. Glad to see you are searching for online solutions. Took a look at your timestamp reference and it could be implemented but more error checking logic may be needed.

I have also considered changing script to python rather than bash but originally this script was simpler but grew more complicate when avconv was made default in Raspbian due to ffmeg not being maintained.

Thanks for your feedback. Claude ....

pageauc commented 6 years ago

Updated makevideo.sh to ver 4.40 now includes creating archive subfolders (note checks if main archive folder already exists (does not auto create) but does auto create subfolders in main archive folder location. Currently subfolders are year month day so only one archive subfolder is created per day. Also cleaned up some of the logic and squished a few bugs.

Also changed moving logic to preserve datetime per cp then rm per link you provided I have tested new makevideo.sh and seems to work fine but I did not test across network. Can you please test and let me know results. I appreciate your comments and suggestions. Let me know if this meets your needs.

Claude ...

You can download latest version per command below

 wget -O makevideo.sh https://raw.github.com/pageauc/pi-timolo/master/source/makevideo.sh