search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
994 stars 354 forks source link

update error - numpy.dtype has the wrong size #1105

Closed dwaggott closed 9 years ago

dwaggott commented 9 years ago

Results after #1104 was fixed.

dwaggott@scg3-ln02: ~/ashley2/apps
$ rm -rf tmpbcbio-install
dwaggott@scg3-ln02: ~/ashley2/apps
$ bcbio_nextgen.py upgrade -u development  --genomes hg38
Traceback (most recent call last):
  File "/srv/gsfs0/projects/ashley/apps/bin/bcbio_nextgen.py", line 35, in <module>
    from bcbio.distributed import runfn, clargs
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 13, in <module>
    from bcbio.distributed import multitasks
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 3, in <module>
    from bcbio import heterogeneity, hla, structural, utils, chipseq, upload
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/__init__.py", line 10, in <module>
    from bcbio.heterogeneity import bubbletree, phylowgs, theta
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 21, in <module>
    from bcbio.heterogeneity import chromhacks, theta
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/theta.py", line 22, in <module>
    from bcbio.structural import cnvkit, convert
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/__init__.py", line 10, in <module>
    from bcbio.structural import (battenberg, cn_mops, cnvkit, delly,
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/cnvkit.py", line 23, in <module>
    from bcbio.structural import annotate, shared, regions, plot
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/plot.py", line 14, in <module>
    from bcbio.bam.coverage import plot_multiple_regions_coverage
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/bcbio/bam/coverage.py", line 12, in <module>
    import seaborn as sns
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/seaborn/__init__.py", line 4, in <module>
    from .linearmodels import *
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/seaborn/linearmodels.py", line 28, in <module>
    from .distributions import kdeplot
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/seaborn/distributions.py", line 13, in <module>
    import statsmodels.nonparametric.api as smnp
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/statsmodels/nonparametric/api.py", line 1, in <module>
    from .kde import KDEUnivariate
  File "/srv/gsfs0/projects/ashley/apps/bcbio/anaconda/lib/python2.7/site-packages/statsmodels/nonparametric/kde.py", line 26, in <module>
    from .linbin import fast_linbin
  File "__init__.pxd", line 155, in init statsmodels.nonparametric.linbin (statsmodels/nonparametric/linbin.c:3947)
ValueError: numpy.dtype has the wrong size, try recompiling
alexcoman commented 9 years ago

Hello,

I have encountered the same issue during the docker image building.

These are the last lines from build.log:

[localhost] local: /usr/local/share/bcbio-nextgen/anaconda/bin/Rscript /root/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -f /root/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -rf /root/tmp/cloudbiolinux
[localhost] local: rm -rf .cpanm
[localhost] local: rm -f /var/crash/*
[localhost] local: rm -f ~/*.dot
[localhost] local: rm -f ~/*.log
Creating manifest of installed packages in /usr/local/share/bcbio-nextgen/manifest
Third party tools upgrade complete.
Upgrade completed successfully.
Traceback (most recent call last):
  File "/usr/local/share/bcbio-nextgen/anaconda/bin/bcbio_nextgen.py", line 35, in <module>
    from bcbio.distributed import runfn, clargs
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 13, in <module>
    from bcbio.distributed import multitasks
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 3, in <module>
    from bcbio import heterogeneity, hla, structural, utils, chipseq, upload
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/__init__.py", line 10, in <module>
    from bcbio.heterogeneity import bubbletree, phylowgs, theta
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 21, in <module>
    from bcbio.heterogeneity import chromhacks, theta
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/theta.py", line 22, in <module>
    from bcbio.structural import cnvkit, convert
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/structural/__init__.py", line 10, in <module>
    from bcbio.structural import (battenberg, cn_mops, cnvkit, delly,
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/structural/cnvkit.py", line 23, in <module>
    from bcbio.structural import annotate, shared, regions, plot
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/structural/plot.py", line 14, in <module>
    from bcbio.bam.coverage import plot_multiple_regions_coverage
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/bam/coverage.py", line 12, in <module>
    import seaborn as sns
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seaborn/__init__.py", line 4, in <module>
    from .linearmodels import *
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seaborn/linearmodels.py", line 28, in <module>
    from .distributions import kdeplot
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seaborn/distributions.py", line 13, in <module>
    import statsmodels.nonparametric.api as smnp
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/statsmodels/nonparametric/api.py", line 1, in <module>
    from .kde import KDEUnivariate
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/statsmodels/nonparametric/kde.py", line 26, in <module>
    from .linbin import fast_linbin
  File "__init__.pxd", line 155, in init statsmodels.nonparametric.linbin (statsmodels/nonparametric/linbin.c:3947)
ValueError: numpy.dtype has the wrong size, try recompiling
The command '/bin/sh -c apt-get update && apt-get install -y build-essential zlib1g-dev wget curl python-setuptools git &&     apt-get install -y openjdk-7-jdk openjdk-7-jre ruby libncurses5-dev libcurl4-openssl-dev libbz2-dev     unzip pigz bsdmainutils &&     mkdir -p /tmp/fuse-hack && cd /tmp/fuse-hack &&     apt-get install libfuse2 &&     apt-get download fuse &&     dpkg-deb -x fuse_* . &&     dpkg-deb -e fuse_* &&     rm fuse_*.deb &&     echo -en '#!/bin/bash\nexit 0\n' > DEBIAN/postinst &&     dpkg-deb -b . /fuse.deb &&     dpkg -i /fuse.deb &&     rm -rf /tmp/fuse-hack &&     mkdir -p /tmp/bcbio-nextgen-install && cd /tmp/bcbio-nextgen-install &&     wget --no-check-certificate       https://raw.github.com/chapmanb/bcbio-nextgen/master/scripts/bcbio_nextgen_install.py &&     python bcbio_nextgen_install.py /usr/local/share/bcbio-nextgen       --nodata -u development &&     git config --global url.https://github.com/.insteadOf git://github.com/ &&     /usr/local/share/bcbio-nextgen/anaconda/bin/bcbio_nextgen.py upgrade --sudo --tooldir=/usr/local --tools &&     /usr/local/share/bcbio-nextgen/anaconda/bin/bcbio_nextgen.py upgrade --isolate -u development --tools --toolplus data  &&     echo 'export PATH=/usr/local/bin:$PATH' >> /etc/profile.d/bcbio.sh &&     wget --no-check-certificate -O createsetuser       https://raw.github.com/chapmanb/bcbio-nextgen-vm/master/scripts/createsetuser &&     chmod a+x createsetuser && mv createsetuser /sbin &&     apt-get clean &&     rm -rf /var/lib/apt/lists/* /var/tmp/* &&     /usr/local/share/bcbio-nextgen/anaconda/bin/conda remove --yes qt &&     /usr/local/share/bcbio-nextgen/anaconda/bin/conda clean --yes --tarballs &&     rm -rf /usr/local/share/bcbio-nextgen/anaconda/pkgs/qt* &&     rm -rf $(brew --cache) &&     rm -rf /usr/local/.git &&     rm -rf /.cpanm &&     rm -rf /tmp/bcbio-nextgen-install &&     mkdir -p /mnt/biodata &&     mkdir -p /tmp/bcbio-nextgen &&     mv /usr/local/share/bcbio-nextgen/galaxy/bcbio_system.yaml /usr/local/share/bcbio-nextgen/config &&     rmdir /usr/local/share/bcbio-nextgen/galaxy &&     ln -s /mnt/biodata/galaxy /usr/local/share/bcbio-nextgen/galaxy &&     ln -s /mnt/biodata/gemini_data /usr/local/share/bcbio-nextgen/gemini_data &&     ln -s /mnt/biodata/genomes /usr/local/share/bcbio-nextgen/genomes &&     ln -s /mnt/biodata/liftOver /usr/local/share/bcbio-nextgen/liftOver &&     chmod a+rwx /usr/local/share/bcbio-nextgen &&     chmod a+rwx /usr/local/share/bcbio-nextgen/config &&     chmod a+rwx /usr/local/share/bcbio-nextgen/config/*.yaml &&     chmod a+rwx /usr/local/share/bcbio-nextgen/gemini-config.yaml &&     find /usr/local -perm /u+x -execdir chmod a+x {} \; &&     find /usr/local -perm /u+w -execdir chmod a+w {} \;' returned a non-zero code: 1

Thanks, Alex

chapmanb commented 9 years ago

Daryl and Alex; Sorry about the issues. It looks like a lot of the defaults for numpy packages switched from numpy 1.9 to numpy 1.10, causing this issue. I'll have to do some digging into this to build new packages for numpy 1.10 for those that are incompatible. I couldn't replicate this locally so am not sure what is different on your setups causing the problem. Could you report the results of:

$ bcbio_conda list | grep np1
biopython                 1.65                np110py27_0
[...]
$ bcbio_conda list | grep nump
numpy                     1.10.1                   py27_0

and I can try to prioritize obvious packages that are out of date. If you spot any standard packages like statsmodels, seaborn or numpy that don't have np110 you could try updating them manually with:

bcbio_conda install problem_package

it's possible you caught them in the middle of the upgrade. I'll try to re-sync all of our packages to match the new default as well. Thanks again for the heads up.

lpantano commented 9 years ago

Hi,

I had the same problem in another computer. I am not sure exactly why, but I did this:

/groups/bcbio/b cbio/anaconda/bin/conda install statsmodels and it fixed the problem. I think is some kind of de-sync between packages.

Sorry couldn't figure out what was the problem. But that is what I did.

cheers

dwaggott commented 9 years ago 
dwaggott@scg3-ln02: ~/ashley2/apps
$ bcbio_conda list | grep np1
biopython                 1.65                 np19py27_0  
crossmap                  0.1.8                np19py27_0  
ete2                      2.2.1072             np19py27_1  
hmmlearn                  20150818             np19py27_0  
htseq                     0.6.1                np19py27_0  
matplotlib                1.4.3                np19py27_2  
numexpr                   2.4.4                np19py27_0  
pandas                    0.16.2               np19py27_0  
patsy                     0.4.0               np110py27_0  
phylowgs                  20150714             np19py27_0  
pyloh                     1.4.1                np19py27_0  
scikit-learn              0.16.1               np19py27_0  
scipy                     0.16.0               np19py27_1  
seaborn                   0.6.0               np110py27_0  
statsmodels               0.6.1               np110py27_0  
dwaggott@scg3-ln02: ~/ashley2/apps
$ bcbio_conda list | grep nump
numpy                     1.9.3                    py27_0
dwaggott commented 9 years ago

That seems to get it over the hump. Thanks!

Feel free to close. 

$ bcbio/anaconda/bin/conda install statsmodels                                                                                                                                                                                                                                                                                
Fetching package metadata: ....
Solving package specifications: ......................
Package plan for installation in environment /srv/gsfs0/projects/ashley/apps/bcbio/anaconda:

The following packages will be UPDATED:

    numpy:  1.9.3-py27_0      --> 1.10.1-py27_0     
    pandas: 0.16.2-np19py27_0 --> 0.17.0-np110py27_0
    scipy:  0.16.0-np19py27_1 --> 0.16.0-np110py27_1

Proceed ([y]/n)? y

Unlinking packages ...
[      COMPLETE      ]|#################################################################################################################################################################################################################################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|#################################################################################################################################################################################################################################################################################################| 100%

$ bcbio_nextgen.py upgrade -u development  --genomes hg38
Fetching package metadata: ........
Solving package specifications: ....................................................................................................
Package plan for installation in environment /srv/gsfs0/projects/ashley/apps/bcbio/anaconda:

The following packages will be UPDATED:

    biopython:    1.65-np19py27_0   --> 1.65-np110py27_0 
    matplotlib:   1.4.3-np19py27_2  --> 1.4.3-np110py27_2
    scikit-learn: 0.16.1-np19py27_0 --> 0.17-np110py27_1 

...

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    glpk-4.56                  |                0         3.0 MB
    java-jdk-8.0.45            |                0        42.7 MB
    pbgzip-2015.10.28          |                0          80 KB
    seqtk-r82                  |                0         103 KB
    fastqc-0.11.4              |                1         9.5 MB
    hdf5-1.8.15.1              |                2         1.9 MB
    bowtie2-2.2.6              |           py27_0         8.8 MB
    cufflinks-2.2.1            |           py27_0        47.9 MB
    ply-3.8                    |           py27_0          71 KB
    tophat-2.1.0               |           py27_0        37.9 MB
    pyutilib-5.1.3556          |           py27_0         280 KB
    pyomo-4.1.10527            |           py27_0         1.5 MB
    pytables-3.2.2             |       np19py27_0         3.4 MB
    optitype-2015.10.20        |           py27_0         4.5 MB
    ------------------------------------------------------------
                                           Total:       161.6 MB

The following NEW packages will be INSTALLED:

    bowtie2:      2.2.6-py27_0     
    cufflinks:    2.2.1-py27_0     
    curl:         7.45.0-0         
    fastqc:       0.11.4-1         
    glpk:         4.56-0           
    hdf5:         1.8.15.1-2       
    java-jdk:     8.0.45-0         
    optitype:     2015.10.20-py27_0
    pbgzip:       2015.10.28-0     
    ply:          3.8-py27_0       
    pyomo:        4.1.10527-py27_0 
    pytables:     3.2.2-np19py27_0 
    pyutilib:     5.1.3556-py27_0  
    seqtk:        r82-0            
    tophat:       2.1.0-py27_0     

The following packages will be UPDATED:

    biopython:    1.65-np110py27_0   --> 1.65-np19py27_0  
    matplotlib:   1.4.3-np110py27_2  --> 1.4.3-np19py27_2 
    scipy:        0.16.0-np110py27_1 --> 0.16.0-np19py27_1

The following packages will be DOWNGRADED:

    numpy:        1.10.1-py27_0      --> 1.9.3-py27_0     
    pandas:       0.17.0-np110py27_0 --> 0.16.2-np19py27_0
    scikit-learn: 0.17-np110py27_1   --> 0.16.1-np19py27_0
...
chapmanb commented 9 years ago

Daryl and Lorena; Thanks for the additional debugging. It looks like there was an in-progress rollout of numpy 110 compatible packages, so depending on update times you could end up with a mix of numpy 1.9 and 1.10 packages, causing issues.

I updated all the conda packages we use to 1.10 to be consistent which will hopefully avoid this going forward. To fix a broken installation a command like:

/path/to/bcbio/anaconda/bin/conda install seaborn

should pull in the latest pandas, numpy and other dependent packages. Fingers crossed this will clean up the problem going forward. Thanks again for the report.

sxv commented 8 years ago

Received this error after updating to latest stable version today. The above suggestions did not work, but I was able to fix with:

pip install --upgrade numpy