Download of packages fails with certain CMake versions when using redirects

[MaxAlex]

I'm attempting to install the contrib package on Windows 7 64-bit, using cmake 3.0.0. Using either build type SEQAN or ALL I get an error:

c:\Users\Max\Downloads\contrib-master\contrib-master>"C:\Program Files (x86)\CMa ke\bin\cmake.exe" -DBUILD_TYPE=SEQAN -- Building for: Visual Studio 12 2013 -- ADDRESSMODEL IS: 32 bit -- MSVC Version is: 12 -- BUILD_TYPE: SEQAN (one of: ALL;SEQAN;LIBSVM;XERCESC;BOOST;GSL;COINOR;BZIP2;ZL IB;GLPK;EIGEN;WILDMAGIC) -- FORCE_REBUILD: OFF -- NUMBER_OF_JOBS: 2 (maximal number of concurrent compile jobs) -- Downloading SEQAN .. -- Downloading SEQAN .. done -- Validating archive for SEQAN .. -- Validating archive for SEQAN .. sha1 mismatch (expected: 1aecb390311a12330c5a 1c8d91cb9ac4cd8e1718 got: d4dd8ea3692d5860e5326181f3c5066139d98ca1) -- The archive file for SEQAN seems to be damaged and will be removed. -- Please try to rebuild SEQAN to trigger a new download of the archive. -- If this fails again, please contact the OpenMS support. CMake Error at macros.cmake:51 (message): Abort! Call Stack (most recent call first): macros.cmake:86 (validate_archive) macros.cmake:100 (download_contrib_archive) libraries.cmake/seqan.cmake:15 (OPENMS_SMARTEXTRACT) CMakeLists.txt:415 (OPENMS_CONTRIB_BUILD_SEQAN)

-- Configuring incomplete, errors occurred! See also "C:/Users/Max/Downloads/contrib-master/contrib-master/CMakeFiles/CMakeO utput.log".

I've attached the log file that it references, but as far as I can see immediately it doesn't describe any error:

CMakeOutput.log.txt

[cbielow]

Looks like something went wrong when downloading the seqan tar.gz (since the SHA1 does not match).

Have you tried running the installer again (which should trigger downloading it again)? Before doing that, make sure that

/src/seqan-mini-1.4.1.tar.gz and the extracted content in the same folder are removed (the build script should take care of this, in theory). alternative: seqan-mini-1.4.1.tar.gz is obtained from https://sourceforge.net/projects/open-ms/files/contrib/, so you could try to get them manually (via browser) and put them into the contrib_build/src directory. Let us know what went wrong. maybe the build system can be made more stable. sidenote: The Cmake log file contains only basic compiler checks, which don't mean a lot since we're using CMake only as scripting engine which calls the build systems of contrib libs, not to compile code directly. Am 3/2/2016 um 12:15 AM schrieb MaxAlex: > I'm attempting to install the contrib package on Windows 7 64-bit, using > cmake 3.0.0. Using either build type SEQAN or ALL I get an error: > > ``` > c:\Users\Max\Downloads\contrib-master\contrib-master>"C:\Program > Files (x86)\CMa > ke\bin\cmake.exe" -DBUILD_TYPE=SEQAN > -- Building for: Visual Studio 12 2013 > -- ADDRESSMODEL IS: 32 bit > -- MSVC Version is: 12 > -- BUILD_TYPE: SEQAN (one of: > ALL;SEQAN;LIBSVM;XERCESC;BOOST;GSL;COINOR;BZIP2;ZL > IB;GLPK;EIGEN;WILDMAGIC) > -- FORCE_REBUILD: OFF > -- NUMBER_OF_JOBS: 2 (maximal number of concurrent compile jobs) > -- Downloading SEQAN .. > -- Downloading SEQAN .. done > -- Validating archive for SEQAN .. > -- Validating archive for SEQAN .. sha1 mismatch (expected: > 1aecb390311a12330c5a > 1c8d91cb9ac4cd8e1718 got: d4dd8ea3692d5860e5326181f3c5066139d98ca1) > -- The archive file for SEQAN seems to be damaged and will be removed. > -- Please try to rebuild SEQAN to trigger a new download of the archive. > -- If this fails again, please contact the OpenMS support. > CMake Error at macros.cmake:51 (message): > Abort! > Call Stack (most recent call first): > macros.cmake:86 (validate_archive) > macros.cmake:100 (download_contrib_archive) > libraries.cmake/seqan.cmake:15 (OPENMS_SMARTEXTRACT) > CMakeLists.txt:415 (OPENMS_CONTRIB_BUILD_SEQAN) > > -- Configuring incomplete, errors occurred! > See also > "C:/Users/Max/Downloads/contrib-master/contrib-master/CMakeFiles/CMakeO > utput.log". > ``` > > I've attached the log file that it references, but as far as I can see > immediately it doesn't describe any error: > > CMakeOutput.log.txt > https://github.com/OpenMS/contrib/files/153779/CMakeOutput.log.txt > > — > Reply to this email directly or view it on GitHub > https://github.com/OpenMS/contrib/issues/14.

[MaxAlex]

The src directory was empty, and I've attempted the install multiple times with the same result. Downloading seqan-mini-1.4.1.tar.gz directly and extracting caused the build to skip downloading SeQan again, but it still failed the SHA1 check- the hash this time was 3f86 e2c8d0d469eeafbb399dc580eebc6be48e49.

[hroest]

have you tried to download the file by hand and putting it into the "./archives" folder?

[hroest]

also, can you try a different cmake version?

[timosachsenberg]

The SHA1 1aecb390311a12330c5a1c8d91cb9ac4cd8e1718 is compared to the compressed .tar.gz file and download is skipped if the sources or the .tar.gz archive is present. I downloaded the file manually on windows and checked the checksum using the https://support.microsoft.com/en-us/kb/841290#bookmark-4 tool (command line parameter -sha1). It is identical to the one stored in the contrib so things should work. Could you try rebuild from a clean contrib checkout?

[hroest]

one issue with debugging this is that cmake removes the "faulty" download and then the directory is empty. Maybe it would make more sense to move "some-library.1.2.3.tar.gz" to "some-library.1.2.3.tar.gz.failed" to allow people to investigate what went wrong?

[jpfeuffer]

I encountered the same problem but on Linux with a self built CMake (does not seem to be the case here). For me, the problem was that when you build CMake yourself, it comes with an own libcurl without ssl support and weird behaviour following links. Since sourceforge redirects and then uses https, you will download an empty file and everything seems fine at first. Solution for me was:

• Install zlib and libcurl
• Download CMake, untar

./bootstrap --system-curl
make && make install

Maybe similar things happened there. Even if not: we could increase the verbosity by adding LOG as a parameter to the FILE(DOWNLOAD ...) command in CMake, we could do what @hroest suggested above, and we should maybe completely move away from sourceforge and host on an own server.

[timosachsenberg]

we could also put them on github

[hroest]

it seems they now support binary releases with files up to 2 GB: https://help.github.com/articles/distributing-large-binaries/ personally, less sourceforge is better in my book

[cbielow]

Also github uses redirects if you have the wrong link.

But +1 for migration since we should couple the contrib buils system to the targz again. Right now its somewhat loose and error prone.

Sent via mobile phone.

Am 20. April 2016 21:16:03 MESZ, schrieb Hannes Roest notifications@github.com:

it seems they now support binary releases with files up to 2 GB: https://help.github.com/articles/distributing-large-binaries/ personally, less sourceforge is better in my book

---

You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/OpenMS/contrib/issues/14#issuecomment-212564928

[jpfeuffer]

Hmmm.. by the way: What's with the contrib-too-large subrepo? Seems like it includes the archives?

[bgruening]

I have encountered this as well on Linux with a standard CMake.

[bjpop]

I get this same issue on Ubuntu 14.04 with Cmake 2.8.12.2, and also on RHEL 6.7 with Cmake 3.5.0.

Curiously I do not have this problem on OS X 10.10.5 with Cmake 3.5.1.

[mwalzer]

That issue occurs on several OS, and as @jpfeuffer pointed out, the culprit seems to be a

libcurl without ssl

which can of course not only occur with self built CMake, but also with incomplete distro package versions... github might work without secure connection download

[jpfeuffer]

Okay, I created a pull request that uses a new base URL for the source packages of the contrib. I hosted them on our build-archive since it is a plain HTTP Server without any redirects. I considered hosting them on Github but CMake might run into trouble with SSL again. Seems to run fine on our build machines, however it worked on them before, so somebody else might need to test it.

[mwalzer]

regarding github, afaik any link to files from open projects or releases will work in https and http form.

[jpfeuffer]

Wow. You are right. For release files you can use plain HTTP. But I could not get a file from the repo without getting redirected to https (only tested shortly).

[mwalzer]

aw snap. So would it be overkill to create a release?

[jpfeuffer]

Not necessarily. Just suboptimal (think about updates to them, which are honestly rare). Which repo would you use? A new one? We could also use git functionality in CMake. Lots of options actually...

[mwalzer]

mh, what about the thirdparty one? In related news (https://github.com/OpenMS/THIRDPARTY/issues/3) , if the install downloader (not cmake) chooses only one of the architecture folders and the all folder it would be no hindrance to have a contrib folder in thirdparty and create a contrib-thirdparty-release? Just an idea, as it is thirdparty stuff as well.

[timosachsenberg]

hmm not ideal in my view as this would mean that build time requirements are provided in the same repo as the platform (and potentially compiler) dependent applications/resources used in the deployment. cmake's https://cmake.org/cmake/help/v3.2/module/ExternalProject.html using the git functionality it provides seems like an option we could look into (I guess @jpfeuffer had this in mind?).

[jpfeuffer]

I agree. See my update to PR #20 and the new subrepo contrib-sources.

[hroest]

given that we verify the hash after download, there is no need for HTTPS and we could do plain HTTP. I am a bit worried about having all of these on a more or less private server as the whole OpenMS build could fail if that server is down.

[jpfeuffer]

Yes you are right. The problem was that we were not sure if you can download single files from a GitHub repository with a HTTP URL without getting redirected to HTTPS which would require additional setting during the CMake build (see this whole issue). But I actually created a GitHub subrepo now that holds these files via the new Git Large File Storage and it is possible to get HTTP links to individual files. Continuous integration runs through (see #20). With "private server" you meant our build-archive? I reverted this and use GitHub in my latest commit. Is this public enough?

[hroest]

yes, I meant that we should not put it on our own build server

[jpfeuffer]

Well, git lfs did not work out well, the limitations on the free versions are too high (especially traffic). For now we leave it on our servers. Downtimes are really low.

[jpfeuffer]

Maybe out IT sets up an own Git LFS server. Or we eventually have to use a paid/different solution. E.g. package managers, artifactories..

[timosachsenberg]

We store them in the contrib-sources repo as release artifacts