More automated installation problems

[colinbann]

I have similar issues to those reported in #134 Automated installation problem. Output from 'python INSTALL.py core' below. I am sure I have connection to internet, and I can do other installations OK, e.g:

curl -L http://psutil.googlecode.com/files/psutil-0.6.1.tar.gz -o ./psutil.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 135k 100 135k 0 0 153k 0 --:--:-- --:--:-- --:--:-- 157k

Any suggestions much appreciated. Colin

-- output from 'python INSTALL.py core' --

/cm/shared/apps/python/2.7.3/lib/python2.7/site-packages/setuptools-5.4.1-py2.7.egg/pkg_resources.py:1049: UserWarning: /panfs/panasas01.panfs.cluster/cczcb/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).

curl: (7) couldn't connect to host tar: amos-binaries.tar.gz: Cannot open: No such file or directory tar: Error is not recoverable: exiting now

curl: (7) couldn't connect to host tar (child): stat.tar.gz: Cannot open: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now <> Selected to install workflowName: CORE. Will automatically install: Pysam Setuptools Fastqc Matplotlib Metavelvet Lap Kronatools Cython Sra Velvet Velvet-Sc Kraken Psutil Uniprot Amos Numpy Traceback (most recent call last): File "INSTALL.py", line 419, in os.chdir("Statistics-Descriptive-3.0203") OSError: [Errno 2] No such file or directory: 'Statistics-Descriptive-3.0203'

[skoren]

This does look like a network connection issue to the UMD ftp site. I've confirmed I can successfully download the files you were having an error with. Can you run: curl -L ftp://ftp.cbcb.umd.edu/pub/data/metamos/amos-3.2-BETA-Linux-x86_64.binaries.tar.gz -o ./amos-binaries.tar.gz and curl -L ftp://cbcb.umd.edu/pub/data/metamos/Statistics-Descriptive-3.0203.tar.gz -o stat.tar.gz

To see if those commands work? If they do I would try re-running the metAMOS INSTALL.py as it could have been an intermittent connectivity issue.

[colinbann]

Hello Sergey, Really appreciate your fast response. Looks like ftp connections are blocked, but http is OK. I'm getting round this by using a separate host to do downloads, then copying files into the build directory on my target platform. Do you think this approach is feasible ? I've done this for the examples in your response, and installation has proceeded further, though I see there are more to do.

[colinbann]

Sorry pushed wrong button!

[skoren]

I think you should be OK with your approach of downloading and moving files but we've not tested it. An alternate suggestion would be to install everything on one system and then move the installation to the target system. This should be OK as long as the systems have the same libraries/compiler versions installed. A third option is to use the frozen binary, if you have not already tried it, which packages the tools into a single binary and does not require running the installation script.

[colinbann]

I have gone some way into downloading and moving files (I do not have an alternative system to use). However, I am still being foiled by missing files, and I am not clear precisely which files I need to place where in the installation heirarchy (I guess not all files should be placed in the top level metAMOS-1.5rc3 directory). It seems looking through INSTALL.py may not reveal all dependencies, as there are other scripts called from this (e.g. KronaTools/updateTaxonomy.sh). So as this is an unsupported install method, I've tried out the 'frozen binary'. However, when trying the test pipeline as in the manual I get output below. I'm using the minimal DB set for now, and have set TMPDIR to '/local', as this gives sufficient space, unlike /tmp. libc.so.6 is in the usual place on our system (/lib64/libc.so.6). Do you know what is happening here ?

./run_pipeline_test.sh Error: cannot find BLAST DB directory, expected it in /panfs/panasas01.panfs.cluster/cczcb/builds/metAMOS_v1.5rc2_binary/DB//blastdbs/. Disabling blastdb dependent programs Project dir /panfs/panasas01.panfs.cluster/cczcb/builds/metAMOS_v1.5rc2_binary/Test/test1 successfully created! Use runPipeline.py to start Pipeline Warning: Cannot determine OS, defaulting to Linux Warning: Cannot determine OS version, defaulting to 0.0 Warning: Cannot determine system type, defaulting to x86_64 sh: /local/_MEIdaoUk6/lib/libc.so.6: version GLIBC_2.11' not found (required by sh) Error: cannot find BLAST DB directory, expected it in /panfs/panasas01.panfs.cluster/cczcb/builds/metAMOS_v1.5rc2_binary/DB//blastdbs/. Disabling blastdb dependent programs sh: /local/_MEIdaoUk6/lib/libc.so.6: versionGLIBC_2.11' not found (required by sh) IOError: [Errno 2] No such file or directory: '/local/_MEIdaoUk6/phylosift/phylosiftrc.orig'

Thanks, Colin

[skoren]

Hi,

The blast message is expected. For space, the BLAST DBs are not included in the miniDB download. We've seen this error once before from the frozen binary in issue #147. The issue is the frozen binary is built on an older version of GCC/OS. It runs in its own world with the older GCC libraries it is packaged with. Unfortunately, since your OS uses a newer version, no system utilities work from within metAMOS. We can't update the frozen binary to a newer version of GCC or it would break for other systems so in your case, the installation from source is the best bet.

Since you encountered issues with dependencies by manually downloading dependencies, I would suggest building the full package on a machine with internet access (where you are downloading from) and then moving the entire metAMOS directory over to where you want to run. This will work as long as the hosts are the same system type and have the same libraries/etc installed (so, for example if a perl library is installed on the download system but not the second system, metAMOS won't download it since it would be found but you would get an error when you ran). If you encounter issues with this approach, I would see if you can request FTP access to be enabled on your system temporarily to complete the metAMOS installation and then disabled again afterwards.

Sergey

[colinbann]

Thanks Sergey. Unfortunately, the system I am downloading on is very different from the platform I need the build on (Windows against Scientific Linux). I don't have access to an alternative Linux system. I am asking our network people to look at the ftp access issue from the Linux system.

[colinbann]

I'm getting a lot further now, having amended the download scripts so they use active rather than passive mode for ftp transfers, which works better in our firewall setup. Install of the 'core' dependencies appears to complete without errors. However, when I try python INSTALL.py iMetAMOS It eventually gets stuck when trying to install boost. The line in INSTALL.py is os.system("curl -L http://sourceforge.net/projects/boost/files/boost/1.54.0/boost_1_54_0.tar.gz -o boost.tar.gz") but this downloads an HTML file, rather than a gzip file, so the installation fails. Have sourceforge changed the webpage you are using in someway, to break this ?

[skoren]

The download works on my system. It does end up at an HTML page first which gets forwarded to a download link (the -L option in curl). I'm guessing this isn't happening on your system for some reason. You could try a direct mirror link instead: http://hivelocity.dl.sourceforge.net/project/boost/boost/1.54.0/boost_1_54_0.tar.gz

[colinbann]

That's useful, thanks. Build now seems to complete, but I'm wondering if there are still things missing, as when I run the 'run_pipeline_test.sh' test, I get an error because it can't find KronaTools.pm. I'm going to go back to the install, and be more careful in examining output from install procedure.

[skoren]

We've seen this issue with Krona recently but are unable to reproduce it locally. Can you make sure your INSTALL.py (or your manual installation) is downloading the latest KronaTools version. Line 124 in INSTALL.py should be KronaTools-2.4 not 2.2 which has a syntax error on newer version of Perl.

[colinbann]

Ah - INSTALL.py does use 2.2. I will re-try with edited version.

[skoren]

I will update the FTP files for version 2.2 to be 2.4 as well to ensure that users with an older checkout don't encounter this error.

[colinbann]

The 'INSTALL.py core' has now gone through, seemingly without errors. I also ran 'INSTALL.py iMetAMOS', which did produce an error (unable to connect), but on a subsequent try seemed to work. I've now run the test 'run_pipeline_test.sh', which seems OK, screen shot of results attached - does this look OK ? I did get an error in the initPipeline step: Error: cannot find BLAST DB directory, expected it in /cm/shared/apps/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs Doing an strace, it seems to be looking for refseq_protein.00.pin under the Utilities/DB directory. I have unpacked the allDBs.tar file, but this has created a 'blastdbs' subdirectory, which does include this file, but it seems the initPipeline process in looking in the 'wrong' place - any way I can change this without copying files (e.g. setting environment variables ?) Thanks, Colin

[skoren]

Sorry for not replying earlier. The allDBs.tar file was designed to work with the frozen binary so the standard installation is not picking up it's directory structure properly. You can specify the location of the refseq_protein* files by setting the BLASTDB environment variable to the path to your blastdbs subdirectory from the extracted allDBs.tar file and metAMOS should find it then.