How to check frame md5s

[ablwr]

Create frame md5s

• We should also indicate in an additional button how to check if the MD5 are correct or not. (c/o Reto)

[retokromer]

The same Shell script should work on Linux, OS X and Windows, right?

[ablwr]

Oh, hmm! Ideally. But a system-specific one with a note is better than none at all.

[retokromer]

OK, I'll try.

[dericed]

@retokromer A framemd5 tool would have a lot of value beyond ffmprovisor. I suggest we could start developing it here with the goal of sending the patch to ffmpeg-devel as a tool for inclusion in https://github.com/FFmpeg/FFmpeg/tree/master/tools.

[retokromer]

@dericed Thank you for the hint! I already have a tool for Linux/OS X, but I must check what I have to change for Windows (my last was 3.11).

[dericed]

@retokromer source code or it didn't happen

[retokromer]

@dericed The solution we use in my company is a bash script calling ffmpeg.

[dericed]

When I try to run that I just get: -bash: @dericed: command not found

[kieranjol]

I doubt ye wanna go the python route, but this works in windows, osx and Ubuntu https://github.com/kieranjol/FFv1-MD5/blob/master/pyffv1md5.py#L64 On 21 Dec 2015 16:02, "Dave Rice" notifications@github.com wrote:

When I try to run that I just get: -bash: @dericed: command not found

— Reply to this email directly or view it on GitHub https://github.com/amiaopensource/ffmprovisr/issues/4#issuecomment-166341668 .

[kieranjol]

P.s I need to redo a lot of the rest on that script cos I wrote it when just learning python so don't judge me too harshly. On 21 Dec 2015 16:11, "Kieran O Leary" kieran.o.leary@gmail.com wrote:

I doubt ye wanna go the python route, but this works in windows, osx and Ubuntu https://github.com/kieranjol/FFv1-MD5/blob/master/pyffv1md5.py#L64 On 21 Dec 2015 16:02, "Dave Rice" notifications@github.com wrote:

When I try to run that I just get: -bash: @dericed: command not found

— Reply to this email directly or view it on GitHub https://github.com/amiaopensource/ffmprovisr/issues/4#issuecomment-166341668 .

[retokromer]

Hmm… I’m still with Perl (I did some tests with version 6 last weekend), and never had a crush on Python. Yet thank you @kieranjol I will read this!

[retokromer]

Does anyone out there know which flavour of shell comes as the default on current Windows installations?

[kieranjol]

You've two options as far as I know. The basic cmd.exe which is probably the most compatible across all windows versions https://en.wikipedia.org/wiki/Cmd.exe and the more powerful powershell https://en.wikipedia.org/wiki/Windows_PowerShell which differs by default depending on which version of windows you have. The wiki has the main differences.

On Sat, Dec 26, 2015 at 10:29 AM, Reto Kromer notifications@github.com wrote:

Does anyone out there know which flavour of shell comes as the default on current Windows installations?

— Reply to this email directly or view it on GitHub https://github.com/amiaopensource/ffmprovisr/issues/4#issuecomment-167311335 .

[retokromer]

Thank you, @kieranjol! So no one of the «classic» sh, ksh, tcsh, bash… is available under Windows by default? Then I’ll check how much I should change the bash (that works fine under various Linuxes and OS X) for «translating» it into cmd.exe.

[kieranjol]

Those shells aren't available without using something like cygwin, unfortunately!

[retokromer]

I’ll try to remember how worked the good old DOS 3.1… That doesn’t make me any younger!

[retokromer]

This is the core of the Bash script that is currently in production (and would need a lifting ;-)

RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m'

av_file="$1"
md5_file="$2"
md5_tmp="$2.tmp"
$(ffmpeg -y -i $av_file -loglevel 0 -f framemd5 $md5_tmp)
file=$(cat $md5_file)
tmp=$(cat $md5_tmp)
if [ "$file" = "$tmp" ]; then
    echo -e "${GREEN}No errors were found!${NC}"
else
    echo -e "${RED}ERROR: The following differences were detected:${NC}"
    diff "$md5_file" "$md5_tmp"
fi
rm "$md5_tmp"

[ablwr]

Wow, including setting color variables!

[dericed]

This would fail if the version of the framemd5 changes but the contents are the same. Instead of cat the md5's should be grepped out via their regex.

[retokromer]

It's just what is running at the moment on our servers, not for direct inclusion in ffmprovisr.

Following @dericed's suggestion, I am planning to code it as an FFmpeg patch, as soon as we have closed the 2015 accounts etc. (somewhere in late January or early February, I guess).

[retokromer]

By deleting the comment lines from the framemd5 reports, id est by replacing:

file=$(grep -v '^#' $md5_file)
tmp=$(grep -v '^#' $md5_tmp)

it should work fine... as a Bash script! Yet the goal is to implement this into FFmpeg's C code.

[retokromer]

@ablwr @dericed @kieranjol If OK with you I'll include this script into the ffmpeg improvised vizier for the moment.

[ablwr]

:+1:

[retokromer]

This works on my different OS X and Linuxes and I guess it works also on Windows with Cygwin. By chance, @kieranjol can you test this http://reto.sh/check_framemd5.sh on Windows?

[kieranjol]

Hi Reto, I never used Cygwin much, but I ran into a few issues:

Firstly, when you drag and drop a file, it shows up as: /cygdrive/c/Users/kieranjol/Downloads/DLD_024-003_03.mkv which is fine until an application is calling a file from a script, in which case you need to change it to c:/Users/kieranjol/Downloads/DLD_024-003_03.mkv

Not too big of a deal, and maybe there's a way around it, but it's a little awkward (on cygwins's part, not your script!) . Another issue is that you're using diff and this doesn't seem to install by default in a vanilla cygwin install. Perhaps a warning could be included in the script if diff isn't available?

kieranjol@kieranjol-PC ~ $ /cygdrive/c/Users/kieranjol/Downloads/check_framemd5.sh c:/Users/kieranjol/Downloads/DLD_024-003_03.mkv c:/Users/kieranjol/Downloads/DLD_024-003_03.framemd5 grep: /home/kieranjol/DLD_024-003_03.framemd5.tmp: No such file or directory ERROR: The following differences were detected between 'DLD_024-003_03.mkv' and 'DLD_024-003_03.framemd5': /cygdrive/c/Users/kieranjol/Downloads/check_framemd5.sh: line 69: diff: command not found rm: cannot remove ‘/home/kieranjol/DLD_024-003_03.framemd5.tmp’: No such file or directory

Installing diffutils fixes this though.

I think cygwin gets confused by the $home reference in your script when generating the tmp file:

kieranjol@kieranjol-PC ~ $ /cygdrive/c/Users/kieranjol/Downloads/check_framemd5.sh c:/Users/kieranjol/Downloads/DLD_024-003_03.mkv c:/Users/kieranjol/Downloads/DLD_024-003_03.framemd5 grep: /home/kieranjol/DLD_024-003_03.framemd5.tmp: No such file or directory ERROR: The following differences were detected between 'DLD_024-003_03.mkv' and 'DLD_024-003_03.framemd5': diff: /home/kieranjol/DLD_024-003_03.framemd5.tmp: No such file or directory rm: cannot remove ‘/home/kieranjol/DLD_024-003_03.framemd5.tmp’: No such file or directory

On a side note, your framemd5 call doesn't include -an. There probably would be a lot of diff errors if someone only ran a framemd5 on their video and then did a fresh compare which included the audio.

[retokromer]

Thank you very much, @kieranjol! I will try to fix it in the next days (before I am leaving for a trip to Southeast Asia).

[kieranjol]

You're more than welcome, @retokromer . I'll do my best to test out the next version as well.

[kieranjol]

P.S: I made some alterations and the script works as intended, presuming I change the filenames after dragging and dropping them: https://gist.github.com/kieranjol/aaddfa51f9877d14fa53/revisions $2.tmp is crude but it's just there for testing purposes.

[retokromer]

Thank you! I used $home in order to be sure (on OS X and Linux) that the temp file can be written. If you use the path to a write protected media (e.g. a data DVD or a LTO tape), then you get an error. Does ~/ work on Windows?

[kieranjol]

Ah, good idea.. In Cygwin ~/ performs the same function as $HOME, in that it points to home/kieranjol rather than c:/Users/Kieranjol. In Cygwin, the function you require is actually $USERPROFILE

I updated the script to test for diff and included $USERPROFILE as a test. Perhaps it's possible for the script to test if it's real bash vs Cygwin, and md5tmp could be set to the appropriate variable? https://gist.github.com/kieranjol/aaddfa51f9877d14fa53/revisions

[retokromer]

Yes, it’s very easy to test on which OS it’s running, and to set a different parameter.

[kieranjol]

I used uname to check for OS based on some stack overflow posts. The script works on both windows and osx now and the md5_tmp is altered based on OS. I used Cygwin* to check as it will alter depending on which windows version is used, eg:

$ uname CYGWIN_NT-6.1 There's probably a more elegant regex that's appropriate.

https://gist.github.com/kieranjol/aaddfa51f9877d14fa53/revisions

[retokromer]

I usually use the $OSTYPE variable. Could you possibly test this?

echo $OSTYPE

on your computer. I guess this works, because usually I have a code like:

if [[ $OSTYPE == "linux-gnu" ]]; then
    echo -e "You wrote ${GREEN}$1${NC} on Linux."
elif [[ $OSTYPE == "darwin"* ]]; then
    echo -e "You wrote ${GREEN}$1${NC} on OS X."
elif [[ $OSTYPE == "cygwin" ]]; then
    echo -e "You wrote ${GREEN}$1${NC} on Windows with POSIX compatibility layer."
else
    echo -e "You wrote ${RED}$1${NC} on an unknown OS."
fi

when there are differences between different OS. Of course, it works also with:

case
    ...
esac

[kieranjol]

Yup, works perfectly. I just did

if [[ $OSTYPE == "cygwin" ]]; then
     md5_tmp=""$USERPROFILE/$(basename $2).tmp""
 else 
     md5_tmp="$HOME/$(basename $2).tmp"
fi

and it worked perfectly on Yosemite and Win7.

[retokromer]

Thank you for the useful feedback! You rock @kieranjol It works fine also on El Capitan, Ubuntu 14.04 and Slackware.

[ablwr]

@kieranjol is the best!

On Mon, Feb 22, 2016 at 9:10 AM, Reto Kromer notifications@github.com wrote:

Thank you for the useful feedback! You rock @kieranjol https://github.com/kieranjol It works fine also on El Capitan, Ubuntu 14.04 and Slackware.

— Reply to this email directly or view it on GitHub https://github.com/amiaopensource/ffmprovisr/issues/4#issuecomment-187191093 .

[kieranjol]

:]

On Mon, Feb 22, 2016 at 2:27 PM, Ashley notifications@github.com wrote:

@kieranjol is the best!

On Mon, Feb 22, 2016 at 9:10 AM, Reto Kromer notifications@github.com wrote:

Thank you for the useful feedback! You rock @kieranjol https://github.com/kieranjol It works fine also on El Capitan, Ubuntu 14.04 and Slackware.

— Reply to this email directly or view it on GitHub < https://github.com/amiaopensource/ffmprovisr/issues/4#issuecomment-187191093

.

— Reply to this email directly or view it on GitHub https://github.com/amiaopensource/ffmprovisr/issues/4#issuecomment-187202732 .

[retokromer]

@kieranjol : It runs fine also on other Windows configurations, but I think I will condider «only» Cygwin here. And, sorry, I have another question to ask you: do you think we should use here and in the generating command -an, or should we omit it on both sides? I am hesitating… @ablwr : Perhaps it’s better to include the bash script as a dedicated page of ffmprovisr, not inside the index.html. Thoughts?

[retokromer]

Please see https://github.com/amiaopensource/ffmprovisr/pull/74 and https://github.com/amiaopensource/ffmprovisr/pull/75 for PR.

[retokromer]

Closes this issue… until the implementation into FFmpeg in C.