search

MediaArea / MediaConch_SourceCode

Media conformance checker
https://MediaArea.net/MediaConch
BSD 2-Clause "Simplified" License
29 stars 18 forks source link

FTP and SFTP access #651

Closed XMegamanX closed 6 years ago

XMegamanX commented 6 years ago

Hi ! I want mediaconch to have access to a file that's on a FTP/SFTP. It is possible to do it with MediaInfo but I am not able to do it with MediaConch. I have rebuild MediaConch CLI with the parameter -with-libcurl and I have rebuild libcurl with ssl and ssh enable. Is there a special way to format the input for this case ?

Thanks in advance

JeromeMartinez commented 6 years ago

We use the same build process between MediaConch and MediaInfo, and actually MediaConch FTP/SFTP support depends on MediaInfo library. And our builds of MediaConch have FTP/SFTP support (you can check on MediaConchOnline by providing a ftp:// URL).

Which source code do you use? Which platform? Did you try the binary we provide?

XMegamanX commented 6 years ago

MediaConch has been built on an Ubuntu 16.04. I use the source code for 'MediaConch Command Line Interface 18.03.2'. Because I used to use the binaries provided by you for MediaInfo and MediaConch and MediaInfo was not able to read ftp/sftp files, I had to rebuild mediaInfo with the -with-libcurl flag to make it works. So I though that it was the same for MediaConch an rebuilt it too.

JeromeMartinez commented 6 years ago

@g-maxime checked our Ubuntu 16.04 packages and we understand that they have libcurl (so FTP/FTPS) support.

So we see 2 issues:

XMegamanX commented 6 years ago

Thank you for the support :). The MediaInfo package I got is MediaInfoLib - v18.03.1, and when I try to use a sftp file :

mediainfo sftp://xxxx:yyyyyyy@myserver.fr/file.xyz --Ssh_IgnoreSecurity E: sftp://xxxx:yyyyyyy@myserver.fr/file.xyz, The Curl library you use has no support for secure connections. If you want to use such protocols, compile libcurl with SSL/SSH support

Regarding MediaConch CLI, what do you suggest ? Maybe I forgot a flag somewhere ? The command that I use is :

mediaconch --Force --Policy=myPolicy.xml --Format=XML sftp://xxxx:yyyyyyy@myserver.fr/file.xyz

JeromeMartinez commented 6 years ago

The Curl library you use has no support for secure connections. If you want to use such protocols, compile libcurl with SSL/SSH support

Looking on the net, looks like this is a common issue of libcurl with Ubuntu. We use CentOS on our servers, looks like it is not an issue on this distro.

You said "ftp" too, what is the message with FTP?

So now I understand more that MediaInfo & MediaConch have libcurl feature, just that system libcurl has no SFTP support, and compiling MediaInfo with custom libcurl is fine, but not MediaConch, so we need to check what is different with MediaConch build script.

JeromeMartinez commented 6 years ago

Just to be clear, is it right? You build your own libcurl with SFTP support in "/libcurlpath". You build MediaInfo with " --with-libcurl=/libcurlpath" option and it works (using your own libcurl) You build MediaConch with " --with-libcurl=/libcurlpath" option and it does not work (not using your own libcurl)

XMegamanX commented 6 years ago

Yes that's exactly that :)

JeromeMartinez commented 6 years ago

For getting the same environment, please provide the exact:

XMegamanX commented 6 years ago
  1. curl-7.60.0
  2. ./configure --with-ssl --with-libssh2=/usr/local
  3. make
  4. sudo make install
  1. Download MediaInfo_CLI_18.05_GNU_FromSource.tar.xz
  2. tar xvf MediaInfo_CLI_18.05_GNU_FromSource.tar.xz
  3. ./CLI_Compile.sh -with-libcurl
  4. cd MediaInfo/Project/GNU/CLI
  5. sudo make install (put bin into /usr/local/bin)
  6. mv mediainfo /usr/bin (to replace former mediainfo installed from packet)
XMegamanX commented 6 years ago

Hourra ! It is working. I have rebuild MediaConch with LDFLAGS="-Wl,--no-as-needed -ldl" ./CLI_Compile.sh -with-libcurl=~/libcurl/curl-7.60.0 and make sure that the policy file extension is correctly written (not like my test done this morning). Sorry for your time since then.

Thank you for the help \m/

XMegamanX commented 6 years ago

Sorry, I have talk too quickly. It is still not working with sftp :(

g-maxime commented 6 years ago

MediaInfo is working with sftp ?

It's --with-libcurl (with two dashes) And ./CLI_Compile.sh --with-libcurl=... isn't working for MediaConch (I will fix that)

But for now you need to edit the CLI_Compile.sh script to add --with-libcurl=~/libcurl/curl-7.60.0 tho the MediaInfoLib configuration command line (replace the --with-libcurl=runtime)

XMegamanX commented 6 years ago

Yes MediaInfo works fine with sftp. Ok I will try this, thank you

XMegamanX commented 6 years ago

Nice, the building went well and now I am able to launch MediaConch with sftp files. The problem is that all the rule tests are failed and all the 'actual' field are empty. If MediaConch is launch with the same file locally, it goes well.

JeromeMartinez commented 6 years ago

do a "mediaconch -mi" on your file, for being sure that the file is correctly read (it should show a MediaInfo report if the file is correctly read).

g-maxime commented 6 years ago

Try with --Ssh_IgnoreSecurity Sample (working for me): LD_LIBRARY_PATH=/usr/local/lib MediaConch/Project/GNU/CLI/mediaconch --Ssh_IgnoreSecurity -mi sftp://user:pass@server/~/MediaArea/MediaInfoLib/Release/Example.ogg

XMegamanX commented 6 years ago

Hell Yeah ! I was not aware that --Ssh_IgnoreSecurity option was also compatible with MediaConch (I already use it in MediaInfo). It works like a charm :).

Thank you for the high quality support.

JeromeMartinez commented 6 years ago

We definitely have to provide more error messages, at least in MediaConch, this last issue was too cryptic. But it would be better if you don't ignore security, see MediaInfo help for how to provide more info to MediaInfo and MediaConch in order to have the right security (--Ssh_PublicKeyFileName, --Ssh_PrivateKeyFileName, --Ssh_KnownHostsFileName)

XMegamanX commented 6 years ago

Ok, thanks for the advice 👍