mlv files missing after accidentally running script

[Ars697]

Hi, I was successfully running this app for months. Today I installed the lated version and loved the LUT and new filters. During exporting some of my mlv files I checked 'post export - HDR_MOV.command'. It requested installing some packages using terminal which I did. I noticed that some folders were created in the original folder where my mlv files resided. In these new folders tiff files were generated. When all finished all my mlv files vanished. Do you have an idea where they might be. Is there a way to recover them. Regards, Artur.

[Ars697]

HDRMOV_LOG.txt I just wanted to add that in that folder where my mlv files used to be a log file has been created.

[bouncyball-git]

@dannephoto usually moves the original MLVs to other folder (that's how his Switch is working) so I think, this script written by him does the same and they are nearby, in some newly created folder.

[Ars697]

I just copied from the above mentioned log text file some of the names of files which are missing and put them in search with no results. It seems they are completely removed from the computer. This might be a glitch that other users should be ware about.

[dannephoto]

Thanks for getting back. I can't check into this atm but I see some dependencies missing. Maybe related to that. Will check into this when I can. You can use photorec I think to recover files. It might be related to spaces too. Not sure. This is your folder: RAW FOOTAGE/WAKACJE W POLSCE 2018 It should not delete your mlv though but something must have not been tested so if you want test a folder with and without spaces. Use test files of course. Photorec: https://www.cgsecurity.org/wiki/File_Formats_Recovered_By_PhotoRec

EDIT 1: trying to recreate the issue but can´t. Did you by any chance move anything before all was done?

EDIT 2: Even tried the exact same folder tree set up but still couldn´t get it to remove the MLV file itself.

EDIT 3: How did you export? tiff file or some other format?

[Ars697]

Thank you so much for helpful attitude. I really do appreciate it. I will try to describe the process more accurately. After importing all the mlv files into the app I just wanted to try this HDR just to see what it does. When I did, it said that the process couldn't be completed because some components were missing. Terminal opened straight away and prompted to import some missing packages which I did. I had the folder open and could see that something was going. Like I mentioned 2 new folders were created and they started to fill in with tiff files which were created out of 2 previously processed .mov files which were also in that same folder as the mlv files. In fact after the process was finished no only the mlv files went missing but also the 2 previously processed and exported .mov files as well. It was so shocking to me to actually see how all the files were wiped out while I watched. The whole process seemed to be running in the background even after I closed the app itself. So after the files were gone the log file appeared and I couldn't do anything. The bin didn't contain anything. never seen anything like this before. Now I suspect that the system was doing it not the app and probably initiated somehow by the app and the components/scripts.

[dannephoto]

Ale kurwa :) Hopefully you can recover those files with photorec. I used the program to recover 20gb of mlv files this summer. Thanks to your report I will rework removal routines. Probably gonna keep all files after processing since something could get wrong but I am not sure how. Very sorry for this. Will post again when fixed.

[Ars697]

I used PhotoRec which is brilliant. Indeed I recovered lots of mlv files but none of them are the files I lost. So there is hope they still must be somewhere. But for the life of me, I can not find them.

[Ars697]

I went through a finder window and used 'All My Files' scroll and couldn't find them. Is it possible that the files were hidden from view.

[masc4ii]

Do you know the tool EasyFind? That finds also hidden files in hidden folders and much more... Way better than Finder Search.

[dannephoto]

Maybe they are renamed somehow? Could you check by size, date?

[Ars697]

I used the tool EasyFind which really impressive. I looked for all .MLV videos. No luck. Seem to be gone permanently with no trace.

[Ars697]

Which is even more shocking that no trace of the files was found on the computer. Using photorec I was able to find all deleted mlv files on the CF card that was used during capture. Unfortunately, the newer files replaced the ones I lost.

[dannephoto]

@Ars697 Very sorry to hear that you seem to have permanently lost your files. I have now removed the rm -r in the scripts so it can´t remove folders anymore. Note that if using the workflow again the only way to close it is from the small bash menu that opens up in the right corner. Press C and it will cancel and kill processing... Latest commit: https://github.com/ilia3101/MLV-App/commit/4b87ca2ad81b07106683c19a77a3cfc30194be2b

[masc4ii]

Has someone tested? Can we close?

[dannephoto]

Did a short test but nothing extensive. Will have a go tomorrow. Would be great to test some bigger files too.

[dannephoto]

@masc4ii I tested the script again against a bunch of mlv files. Folders are kept and HDR mov files are created. Nothing is deleted. However due to a bug with Hugin/enfuse which sometimes fails to complete a file or two it will break ffmpeg output. This can however also be fixed with a safeguard checking for completed files or else restart processing of that particular troubling file. I can put this into the script when I find some time. You can check for safeguarding in this post: https://www.magiclantern.fm/forum/index.php?topic=21454.msg196815#msg196815

Anyway. Regarding the issues in this thread I consider this issue solved.

Something similar to this will work:

#safeguards
   if ! [ -f 1.tiff ] || ! [ -f 2.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 1' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 1') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 2') && mv aligned.tif0000.tif 1.tiff && mv aligned.tif0001.tif 2.tiff
ffmpeg -i %01d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 1' | cut -d "." -f2 | tr -d "/").tiff
   fi

   if ! [ -f 01.tiff ] || ! [ -f 02.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 2' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned2.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 2') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 3') && mv aligned2.tif0000.tif 01.tiff && mv aligned2.tif0001.tif 02.tiff 
ffmpeg -i %02d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 2' | cut -d "." -f2 | tr -d "/").tiff
   fi 

   if ! [ -f 001.tiff ] || ! [ -f 002.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 3' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned3.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 3') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 4') && mv aligned3.tif0000.tif 001.tiff && mv aligned3.tif0001.tif 002.tiff 
ffmpeg -i %03d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 3' | cut -d "." -f2 | tr -d "/").tiff
   fi 

   if ! [ -f 0001.tiff ] || ! [ -f 0002.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 4' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned4.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 4') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 5') && mv aligned4.tif0000.tif 0001.tiff && mv aligned4.tif0001.tif 0002.tiff  
ffmpeg -i %04d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 4' | cut -d "." -f2 | tr -d "/").tiff
   fi 

   if ! [ -f 00001.tiff ] || ! [ -f 00002.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 5' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned5.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 5') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 6') && mv aligned5.tif0000.tif 00001.tiff && mv aligned5.tif0001.tif 00002.tiff  
ffmpeg -i %05d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 5' | cut -d "." -f2 | tr -d "/").tiff
   fi 

   if ! [ -f 0000001.tiff ] || ! [ -f 0000002.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 6' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned6.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 6') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 7') && mv aligned6.tif0000.tif 0000001.tiff && mv aligned6.tif0001.tif 0000002.tiff   
ffmpeg -i %06d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 6' | cut -d "." -f2 | tr -d "/").tiff
   fi 

   if ! [ -f 00000001.tiff ] || ! [ -f 00000002.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 7' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned7.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 7') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 8') && mv aligned7.tif0000.tif 00000001.tiff && mv aligned7.tif0001.tif 00000002.tiff   
ffmpeg -i %07d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 7' | cut -d "." -f2 | tr -d "/").tiff
   fi

   if ! [ -f 000000001.tiff ] || ! [ -f 000000002.tiff ]
   then
   sleep 2
rm $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 8' | cut -d "." -f2 | tr -d "/").tiff
/Applications/Hugin/Hugin.app/Contents/MacOS/align_image_stack -a aligned8.tif $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 8') $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 9') && mv aligned8.tif0000.tif 000000001.tiff && mv aligned8.tif0001.tif 000000002.tiff    
ffmpeg -i %08d.tiff -pix_fmt rgb24 -vf tblend=all_mode=average,$crp_fix $(find -s . -maxdepth 1 -iname '*.tif' | awk 'FNR == 8' | cut -d "." -f2 | tr -d "/").tiff
   fi

[masc4ii]

Good... so we let it opened for you as reminder? Or do we close and you create another enhancement issue?

[dannephoto]

Nah, keep it opened, I'm on it...

[dannephoto]

@masc4ii Committed the safeguard scripts: https://github.com/ilia3101/MLV-App/commit/a26da11626bc5b0886a15a3bddc966d5f42027d3

Let´s keep this issue open and I will do some more tests. Probably on monday when back at my workstation at work.

[dannephoto]

@masc4ii Tested against the former problematic file and now working. Issue could be closed.

[masc4ii]

Great! Thx!