ratschlab / rDiff

Tests for Differential RNA Isoform Expression
GNU General Public License v3.0
5 stars 2 forks source link

rDiff fails on >2K genes during gene expression step; requires 64-bit octave? (my process to build it detailed) #2

Open warrenmcg opened 9 years ago

warrenmcg commented 9 years ago

Once I got rDiff up and running, I attempted to have it analyze my SplAdder results, and it crashed with the following error:

Getting gene expression for: /FUS_KO_splicing_results//CtrBrain2_tophatEnsemblOut/CtrBr2.bam
Finished 2203 out of 22301 geneserror: out of memory or dimension too large for Octave's index type
error: called from:
error:   /home/warren/Downloads/rDiff-master/src/tools/read_utils/clip_reads.m at line 11, column 23
error:   /home/warren/Downloads/rDiff-master/src/tools/read_utils/get_reads_for_gene.m at line 45, column 12
error:   /home/warren/Downloads/rDiff-master/src/get_reads_caller.m at line 52, column 10
error:   /home/warren/Downloads/rDiff-master/src/get_read_counts.m at line 32, column 17
error:   /home/warren/Downloads/rDiff-master/src/rdiff.m at line 38, column 5
starting Octave failed

I realized that I needed to re-build Octave as 64-bit, but as this feature is still in an experimental state, it wasn't straightforward. You first need to build the Blas, Lapack, Qrupdate, Arpack, and SuiteSparse libraries as 64-bit aware too. You'll get a reminder warning about this during the build process, whether you do it or not. If you don't do it, though, Octave will likely fail to run when you try to use it (I didn't test that; I'm just making a reasonable assumption).

Here are the webpages detailing instructions on how to install 64-bit octave: initial page: http://timoeckhard.weebly.com/tutorial-install-octave-38-under-ubuntu-1304.html detailed page with step-by-step instructions, tested with gcc 4.3.4 and Debian system: http://www.gnu.org/software/octave/doc/interpreter/Compiling-Octave-with-64_002dbit-Indexing.html

Here are the modifications I made with my set-up, which was tested with Ubuntu 14.04.3, gcc 4.8.4, and Octave 3.8.1 downloaded from ftp://ftp.gnu.org/gnu/octave/octave-3.8.1.tar.gz: Initial Set-up:

sudo mkdir /usr/lib/octave-prefix64
sudo mkdir /usr/lib/octave-prefix64/lib
sudo mkdir /usr/lib/octave-prefix64/include 

/usr/lib/octave-prefix64/ is my location for the $prefix64 directory described in the detailed instructions. All the necessary files go in the $prefix64/lib/ sub-directory, and the Include files from SuiteSparse go into the $prefix64/include/ directory.

Blas and Lapack (bundled together): Downloaded lapack-3.5.0; configured the make.inc file (which provides the configuration details to set-up the Makefile) as described; had an error where the Blaslib library was missing (“librefblas.a”), so I first did make blaslib before make; then it compiled without error

Qrupdate: Downloaded qrupdate-1.1.2; configured the Makeconf file (similarly provides configuration details) as described; you need to add -L=/usr/lib/octave-prefix64/lib flag before the -lblas and -llapack flags, because these libraries are in a non-standard location now: /usr/lib/octave-prefix64 [substitute this for whatever your $prefix64 directory is]

Arpack: Downloaded 3.1.5; added “fdefault-integer-8” to any line where FFLAGS was defined using quotations (the line with just FFLAGS=-g and empty were left alone)

SuiteSparse: Downloaded SuiteSparse 4.4.5 from here, as the URL given in the detailed instructions is dead; changed the SuiteSparse_config/SuiteSparse_config.mk file to reflect the new 64-bit aware library and include directories; could not figure out how to add the suggested code to make shared libraries, so I just did without, and SuiteSparse and Octave appeared to build without issues

Octave: There is no way, as far as I know, to get Octave to build as 64-bit using the Ubuntu package manager apt-get. I thus downloaded Octave from the FTP site (link above), and configured it as detailed in the instructions, filling in $prefix64 with my directory path.

teresouza commented 8 years ago

Hi warrencmg, I've experienced the same problems trying to use octave and therefore I'm following your steps (thank you so much for that! :D ). However, lapack is giving me a hard time and I cannot see what is missing in my make.inc file; every time it ends up in a loop on eig testing steps or the recipe for an eig target fails. So was your make.inc like that?

FORTRAN = gfortran OPTS = -O2 -frecursive -fdefault-integer-8 DRVOPTS = $(OPTS) NOOPT = -O0 -frecursive -fPIC LOADER = gfortran LOADOPTS =

thank you!

vipints commented 8 years ago

try: ... OPTS = -O2 -fdefault-integer-8 -fPIC ... NOOPT = -O0 -fdefault-integer-8 -fPIC

This will fix the problem I think.

vipints commented 8 years ago

Also while compiling the lapack module, please do the following:

cd BLAS/SRC`
make -j `grep ^processor /proc/cpuinfo | wc -l`

cd ../..
make -j `grep ^processor /proc/cpuinfo | wc -l`

cp liblapack.a libtmglib.a $DEP_PATH/lib
cp librefblas.a $DEP_PATH/lib/libblas.a
cp librefblas.a $DEP_PATH/lib/blas_LINUX.a

$DEP_PATH will be replaced with your local path where you usually keep the dependencies.

teresouza commented 8 years ago

Ok, I followed all the steps to get. and this is what I got after compiling octave: [...] checking whether the integer size is correct... no configure: error: BLAS doesn't seem to support 64-bit integers. This is incompatible with --enable-64.

Where did I go wrong? One thing that I did differently was during SuitSparse compilation. There was an error saying that BLAS lib "-lopenblas" could not be found. So I went on suitsparses' config.mk and changed it to BLAS = -lblas And worked fine. So now I really don know what went wrong. Any advice?

warrenmcg commented 8 years ago

This was my configuration for the make.inc file:

FORTRAN  = gfortran
OPTS     = -O2 -frecursive -fdefault-integer-8 -fPIC
DRVOPTS  = $(OPTS)
NOOPT    = -O0 -frecursive -fdefault-integer-8 -fPIC
LOADER   = gfortran
LOADOPTS =

This was my configuration for the SuiteSparse SuiteSparse_config/SuiteSparse_config.mk file:

...
Line 98: INSTALL_LIB = /usr/lib/octave-prefix64/lib
Line 99: INSTALL_INCLUDE = /usr/lib/octave-prefix64/include
...
Line 122:  BLAS = -L=/usr/lib/octave-prefix64/lib -lblas -lgfortran
Line 123:  LAPACK = -L=/usr/lib/octave-prefix64/lib -llapack
...
Line 133: # OpenBLAS
Line 134: # BLAS = -lopenblas ## **THIS IS IMPORTANT because the original configuration file has this uncommented**
Line 135: # LAPACK =

I noticed that you forgot the -frecursive flag for NOOPTS, so maybe that was the cause of your problem? Try rebuilding/recompiling BLAS/LAPACK and SuiteSparse using this set-up and see if it works for you.

Edit: be sure that lines 98-99 refer to your $prefix64/lib and $prefix64/include directories, respectively.

teresouza commented 8 years ago

Hi warren, It does not work.

make[2]: Entering directory '/home/terezinha/Downloads/SuiteSparse/KLU/Demo' cc '-DLONGBLAS=long' -O3 -fexceptions -fPIC -I../../SuiteSparse_config -I../../AMD/Include -I../../COLAMD/Include -I../../BTF/Include -I../Include -I../../CHOLMOD/Include kludemo.c -o kludemo ../Lib/libklu.a ../../BTF/Lib/libbtf.a ../../AMD/Lib/libamd.a ../../COLAMD/Lib/libcolamd.a ../../SuiteSparse_config/libsuitesparseconfig.a ../../CHOLMOD/Lib/libcholmod.a -L/usr/lib/octave-prefix64/lib -llapack -L/usr/lib/octave-prefix64/lib -lblas -lgfortran-lblas -lgfortran -lm -lrt /home/terezinha/bin/ld: cannot find -lgfortran-lblas collect2: error: ld returned 1 exit status Makefile:68: recipe for target 'kludemo' failed make[2]: *\ [kludemo] Error 1

Is that the path? This is what I have in my octave lib:

/usr/lib/octave-prefix64/lib$ ls blas_LINUX.a libarpack.la libarpack.so.2 libblas.a libqrupdate.a libqrupdate.so.1 libtmglib.a libarpack.a libarpack.so libarpack.so.2.0.0 liblapack.a libqrupdate.so libqrupdate.so.1.1

warrenmcg commented 8 years ago

I don't have the -lgfotran-lblas flag anywhere in my config file. If you have that in your SuiteSparse_config.mk file, remove it and try again. If you don't have that flag anywhere, can you copy your lines 122 and 123 here so we can see them in detail and help you troubleshoot?

vipints commented 8 years ago

@teresouza I think there was problem in pointing the right path of BLAS in configuring the SuiteSparse. I would suggest recompile the SuiteSparse in following way: make the changes in mk file SuiteSparse_config/SuiteSparse_config.mk

vipints commented 8 years ago

CC = gcc CFLAGS = -fPIC -O -DLP64 -DLONGBLAS='long int' -DLONG='long int' F77 = gfortran F77FLAGS = -fPIC -O -fdefault-integer-8 BLAS = -L$(DEP_PATH)/lib -lblas -lgfortran LAPACK = -L$(DEP_PATH)/lib -llapack

vipints commented 8 years ago

Make sure that you copy the lib files from BLAS SRC folder and placed in $DEP_PATH (according to the local path variable you mentioned)

teresouza commented 8 years ago

Ok people, after so many hours I tried to configure octave. I got a lot of warnings (see below). Some things: Most dependencies (cURL, GLPK, etc) are in my $PATH and also isn't that odd that it did not find the COLAMD libraries and UMFPACK?

configure: WARNING: Qhull library not found. This will result in loss of functionality of some geometry functions. configure: WARNING: HDF5 library not found. Octave will not be able to save or load HDF5 data files. configure: WARNING: GLPK library not found. The glpk function for solving linear programs will be disabled. configure: WARNING: cURL library not found. The ftp objects, urlread and urlwrite functions will be disabled. configure: WARNING: OpenGL libs (GL and GLU) not found. Native graphics will be disabled. configure: WARNING: COLAMD library not found. This will result in some lack of functionality for sparse matrices. configure: WARNING: CCOLAMD library not found. This will result in some lack of functionality for sparse matrices. configure: WARNING: CHOLMOD library not found. This will result in some lack of functionality for sparse matrices. configure: WARNING: UMFPACK not found. This will result in some lack of functionality for sparse matrices. configure: WARNING: JAVA_HOME environment variable not initialized. Auto-detection will proceed but is unreliable. configure: WARNING: No javac compiler or jar executable found. Octave will not be able to call Java methods. configure: WARNING: OpenGL libs (GL and GLU) not found -- disabling GUI configure: WARNING: configure: WARNING: You used the EXPERIMENTAL --enable-64 option. configure: WARNING: Are you sure that is what you want to do? configure: WARNING: configure: WARNING: Your Fortran compiler must have an option to generate configure: WARNING: code with 8 byte signed INTEGER values. This option configure: WARNING: should be specified in the F77_INTEGER_8_FLAG variable configure: WARNING: Make. This should work automatically for gfortran. If configure: WARNING: you use another compiler, you will need to set this configure: WARNING: variable on the configure command line. You must also configure: WARNING: compile the ARPACK, BLAS, LAPACK, QRUPDATE, and SuiteSparse configure: WARNING: libraries to use 8 byte signed integers for array indexing. configure: WARNING: configure: WARNING: configure: WARNING: I didn't find the necessary libraries to compile native configure: WARNING: graphics. It isn't necessary to have native graphics, configure: WARNING: but you will need to have gnuplot installed or you won't configure: WARNING: be able to use any of Octave's plotting commands configure: WARNING: configure: configure: NOTE: Libraries or auxiliary programs may be skipped if they are configure: NOTE: not found OR if they are missing required features on your configure: NOTE: system.

EDIT: this is what I used to configure

./configure \ LD_LIBRARY_PATH="/usr/lib/octave-prefix64/lib" \ CPPFLAGS="-I=/usr/lib/octave-prefix64/include" LDFLAGS="-L=/usr/lib/octave-prefix64/lib" \ --enable-64

Contents

terezinha@terezinha-mini-seq:/usr/lib/octave-prefix64/lib$ ls blas_LINUX.a libarpack.so libbtf.a libcholmod.3.0.6.a libcxsparse.a libldl.a librbio.2.2.1.a libsuitesparseconfig.a libamd.2.4.1.a libarpack.so.2 libcamd.2.4.1.a libcholmod.a libklu.1.3.3.a libqrupdate.a librbio.a libtmglib.a libamd.a libarpack.so.2.0.0 libcamd.a libcolamd.2.9.1.a libklu.a libqrupdate.so libspqr.2.0.1.a libumfpack.5.7.1.a libarpack.a libblas.a libccolamd.2.9.1.a libcolamd.a liblapack.a libqrupdate.so.1 libspqr.a libumfpack.a libarpack.la libbtf.1.2.1.a libccolamd.a libcxsparse.3.1.4.a libldl.2.2.1.a libqrupdate.so.1.1 libsuitesparseconfig.4.4.5.a

terezinha@terezinha-mini-seq:/usr/lib/octave-prefix64/include$ ls amd.h cholmod_core.h cholmod_template.h SuiteSparseQR.hpp umfpack.h umfpack_report_status.h umfpack_timer.h btf.h cholmod_function.h colamd.h umfpack_col_to_triplet.h umfpack_load_numeric.h umfpack_report_symbolic.h umfpack_transpose.h camd.h cholmod_gpu.h cs.h umfpack_defaults.h umfpack_load_symbolic.h umfpack_report_triplet.h umfpack_triplet_to_col.h ccolamd.h cholmod_gpu_kernels.h klu.h umfpack_free_numeric.h umfpack_numeric.h umfpack_report_vector.h umfpack_wsolve.h cholmod_blas.h cholmod.h ldl.h umfpack_free_symbolic.h umfpack_qsymbolic.h umfpack_save_numeric.h cholmod_camd.h cholmod_io64.h RBio.h umfpack_get_determinant.h umfpack_report_control.h umfpack_save_symbolic.h cholmod_check.h cholmod_matrixops.h spqr.hpp umfpack_get_lunz.h umfpack_report_info.h umfpack_scale.h cholmod_cholesky.h cholmod_modify.h SuiteSparse_config.h umfpack_get_numeric.h umfpack_report_matrix.h umfpack_solve.h cholmod_complexity.h cholmod_partition.h SuiteSparseQR_C.h umfpack_get_symbolic.h umfpack_report_numeric.h umfpack_symbolic.h cholmod_config.h cholmod_supernodal.h SuiteSparseQR_definitions.h umfpack_global.h umfpack_report_perm.h umfpack_tictoc.h

vipints commented 8 years ago

That is little odd.

Once you check for all dependencies, now it is the time to look for the availability of those module files to the configure script. To get the recent module files which you complied please adjust your LD_LIBRARY_PATH variable. export LD_LIBRARY_PATH=$DEP_PATH/lib

Then try to run configure command with following option:

./configure CPPFLAGS="-I$DEP_PATH/include" LIBS="-L$DEP_PATH/lib/ -L/opt/local/lib" LDFLAGS="-L$DEP_PATH/lib/ -L/opt/local/lib" F77=gfortran --prefix=$DEP_PATH/octave --with-glpk-includedir=$DEP_PATH/include --with-glpk-libdir=$DEP_PATH/lib --with-qhull-includedir=$DEP_PATH/include/ --with-qhull-libdir=$DEP_PATH/lib --with-qrupdate-libdir=$DEP_PATH/lib/ --with-qrupdate-includedir=$DEP_PATH/include/ $opts --with-umfpack="-lumfpack -lsuitesparseconfig" --with-umfpack-libdir=$DEP_PATH/lib/ --with-umfpack-includedir=$DEP_PATH/include/

make sure that the above paths are valid and include the necessary dependencies.

vipints commented 8 years ago

Sorry I forgot to mention about the variable $opts

opts="--enable-64"

teresouza commented 8 years ago

Hi warren and vipints, just to let you know that octave did not compile. I even reinstalled ubuntu (!) and made everything from scratch, specifying libraries in the last step as vipints suggested, but somehow octave does not seem to find the packages (and I found a lot of threads reporting the same problem as well). Do you have another suggestion of how I can use the tool? I saw a few days ago that it was available at the galaxy server from Ratsch Lab and also Oqtans, but apparently there is a problem with the server (I could not login at the time and now both are down). Do you guys have plans to make it again available anytime soon? (Thanks for the help anyway!)

vipints commented 8 years ago

@teresouza I am sure something went wrong in installing the dependency modules on your local machine that might be the cause for the failure of octave 64bit installation. I have successfully tried this on different flavors of ubuntu and worked well.

I am not seeing any down on our public server, which has been operating normally from past days, didn't see any error in the log message. have you tried on this one https://galaxy.cbio.mskcc.org?

teresouza commented 8 years ago

The lnk works, but all jobs submmted do not run:

An error occurred while getting updates from the server. Please contact a Galaxy administrator if the problem persists.

From: Vipin [mailto:notifications@github.com] Sent: Friday, October 02, 2015 5:29 PM To: ratschlab/rDiff rDiff@noreply.github.com Cc: teresouza terezinhamsouza@gmail.com Subject: Re: [rDiff] rDiff fails on >2K genes during gene expression step; requires 64-bit octave? (my process to build it detailed) (#2)

@teresouza https://github.com/teresouza I am sure something went wrong in installing the dependency modules on your local machine that might be the cause for the failure of octave 64bit installation. I have successfully tried this on different flavors of ubuntu and worked well.

I am not seeing any down on our public server, which has been operating normally from past days, didn't see any error in the log message. have you tried on this one https://galaxy.cbio.mskcc.org?

— Reply to this email directly or view it on GitHub https://github.com/ratschlab/rDiff/issues/2#issuecomment-145062484 . https://github.com/notifications/beacon/AJG7o1a-kDHl90ZnEt0OyjOf16a2vSMYks5o3pojgaJpZM4F0h9M.gif

vipints commented 8 years ago

@teresouza Can you point me from where you are getting this error message.

I checked on our side and able to run jobs on without any issues, data upload also works nicely.

teresouza commented 8 years ago

Here a snapshot: http://snag.gy/kMwVW.jpg

From: Vipin [mailto:notifications@github.com] Sent: Friday, October 02, 2015 5:36 PM To: ratschlab/rDiff rDiff@noreply.github.com Cc: teresouza terezinhamsouza@gmail.com Subject: Re: [rDiff] rDiff fails on >2K genes during gene expression step; requires 64-bit octave? (my process to build it detailed) (#2)

@teresouza https://github.com/teresouza Can you point me from where you are getting this error message.

I checked on our side and able to run jobs on without any issues, data upload also works nicely.

— Reply to this email directly or view it on GitHub https://github.com/ratschlab/rDiff/issues/2#issuecomment-145065548 . https://github.com/notifications/beacon/AJG7o2x4nMy7fOrWWuanVUB4FOUjznDhks5o3pvKgaJpZM4F0h9M.gif

vipints commented 8 years ago

Not seeing anything related to this in the Log, I will definitely look in detail.

When did you try to upload the file to the instance? Can you logoff and login then try to upload the file?

Please let me know

teresouza commented 8 years ago

The upload was on Tuesday or Wednesday... I've just uploaded a test file now and worked; so I'm uploading a larger file through FTP to see if it works. Thank you!