search

compomics / moFF

A modest Feature Finder (moFF) to extract MS1 intensities from Thermo raw file
Apache License 2.0
33 stars 11 forks source link

ValueError: Columns must be same length as key #43

Closed yafeng closed 5 years ago

yafeng commented 5 years ago

Hi, I got this error when running the command in the instruction. python moff_all.py --config_file absence_peak_data/configuration_iRT.ini

No module named 'brainpy._c.composition'
Matching between run module (mbr)
MBR Output folder in : /hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/absence_peak_data/mbr_output
B002419_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol_inYeast.txt
B002413_Ap_22cm_Yeast_171215184201.txt
B002417_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol.txt
B002421_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol.txt
Reading file: absence_peak_data/B002419_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol_inYeast.txt
Reading file: absence_peak_data/B002413_Ap_22cm_Yeast_171215184201.txt
Reading file: absence_peak_data/B002417_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol.txt
Reading file: absence_peak_data/B002421_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol.txt
Read input --> done
matched features   2449  MS2 features  5815
matched features   1272  MS2 features  7084
matched features   8321  MS2 features  106
matched features   8296  MS2 features  128
Apex module...
Starting Apex for absence_peak_data/mbr_output/B002419_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol_inYeast_match.txt ...
moff Input file: absence_peak_data/mbr_output/B002419_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol_inYeast_match.txt  XIC_tol 5.0 XIC_win 4.0000 moff_rtWin_peak 1.0000
RAW file from folder :  absence_peak_data/raw_repo/
Output file in :  absence_peak_data/output
Apex module has detected mbr peptides
starting estimation of quality measures..
quality measures estimation  using 582  MS2 ident. peptides randomly sampled
MAD retention time along all isotope count    479.000000
mean       0.847468
std        1.503114
min        0.000000
25%        0.000000
50%        0.674053
75%        0.832173
max       11.003640
Name: RT_drift, dtype: float64
Estimated distribition ratio exp. int. left isotope vs. monoisotopic isotope count    151.000000
mean       0.813199
std        0.113688
min        0.578109
25%        0.732179
50%        0.811201
75%        0.886644
max        1.056126
Name: delta_log_int, dtype: float64
quality threhsold estimated : MAD_retetion_time 1.2549080000000081  Ratio Int. FakeIsotope/1estIsotope: 0.9336942595876153
starting apex quantification of MS2 peptides..
end  apex quantification of MS2 peptides..
starting quantification with matched peaks using the quality filtering...
initial # matched peaks: (2449, 15)
end apex quantification matched peptide
Computational time (sec):  1011.8213
after filtering matched peak #805
Starting Apex for absence_peak_data/mbr_output/B002413_Ap_22cm_Yeast_171215184201_match.txt ...
moff Input file: absence_peak_data/mbr_output/B002413_Ap_22cm_Yeast_171215184201_match.txt  XIC_tol 5.0 XIC_win 4.0000 moff_rtWin_peak 1.0000
RAW file from folder :  absence_peak_data/raw_repo/
Output file in :  absence_peak_data/output
Apex module has detected mbr peptides
starting estimation of quality measures..
quality measures estimation  using 708  MS2 ident. peptides randomly sampled
MAD retention time along all isotope count    581.000000
mean       0.932257
std        1.708456
min        0.000000
25%        0.000000
50%        0.548507
75%        0.827493
max       10.349933
Name: RT_drift, dtype: float64
Estimated distribition ratio exp. int. left isotope vs. monoisotopic isotope count    230.000000
mean       0.817862
std        0.125736
min        0.497536
25%        0.728463
50%        0.823870
75%        0.904802
max        1.147108
Name: delta_log_int, dtype: float64
quality threhsold estimated : MAD_retetion_time 1.2892266666667638  Ratio Int. FakeIsotope/1estIsotope: 0.9483079412122462
starting apex quantification of MS2 peptides..
end  apex quantification of MS2 peptides..
starting quantification with matched peaks using the quality filtering...
initial # matched peaks: (1272, 15)
end apex quantification matched peptide
Computational time (sec):  1183.7997
after filtering matched peak #450
Starting Apex for absence_peak_data/mbr_output/B002417_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol_match.txt ...
moff Input file: absence_peak_data/mbr_output/B002417_Ap_22cm_iRT_PRC-Hans_equimolar_100fmol_match.txt  XIC_tol 5.0 XIC_win 4.0000 moff_rtWin_peak 1.0000
RAW file from folder :  absence_peak_data/raw_repo/
Output file in :  absence_peak_data/output
Apex module has detected mbr peptides
starting estimation of quality measures..
quality measures estimation  using 11  MS2 ident. peptides randomly sampled
Traceback (most recent call last):
  File "/hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/moff.py", line 863, in apex_multithr
    h_rt_w, s_w, s_w_match, offset_index), axis=1)
  File "/hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3/lib/python3.6/site-packages/pandas/core/frame.py", line 3116, in __setitem__
    self._setitem_array(key, value)
  File "/hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3/lib/python3.6/site-packages/pandas/core/frame.py", line 3138, in _setitem_array
    raise ValueError('Columns must be same length as key')
Traceback (most recent call last):
ValueError: Columns must be same length as key

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/moff.py", line 133, in save_moff_apex_result
    if result[df_index].get()[1] == -1:
  File "/hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
ValueError: Columns must be same length as key
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/moff.py", line 888, in apex_multithr
    raise e
  File "/hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/moff.py", line 863, in apex_multithr
    h_rt_w, s_w, s_w_match, offset_index), axis=1)
  File "/hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3/lib/python3.6/site-packages/pandas/core/frame.py", line 3116, in __setitem__
    self._setitem_array(key, value)
  File "/hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3/lib/python3.6/site-packages/pandas/core/frame.py", line 3138, in _setitem_array
    raise ValueError('Columns must be same length as key')
ValueError: Columns must be same length as key
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "moff_all.py", line 374, in <module>
    ptm_map, args.sample_size, args.quantile_thr_filtering, args.match_filter, log_file, num_CPU)
  File "/hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/moff.py", line 539, in estimate_parameter
    ms2_data = save_moff_apex_result(result)
  File "/hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/moff.py", line 144, in save_moff_apex_result
    raise e
  File "/hwfssz5/ST_CANCER/CGR/USER/zhuyafeng/moFF/moFF/moff.py", line 133, in save_moff_apex_result
    if result[df_index].get()[1] == -1:
  File "/hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
ValueError: Columns must be same length as key
Maux82 commented 5 years ago

Hi, I am not able to reprodruce the error. I will try to fix it and came to you in the next days

Are your sure that you have all the dependencie in your python enviroments ?

module named 'brainpy._c.composition'

this sounds something wrong with the brain-isotopic-distribution library.

How many core do you have in your machine ? is it a server machine machine (more than 8) or laptop ?

yafeng commented 5 years ago

Hi, Thanks for your quick reply (especially during Christmas break!) Yes, i ran it on a multi-core server. And here it is my conda environment

 conda list
# packages in environment at /hwfssz1/ST_CANCER/POL/SHARE/tools/miniconda3:
#
# Name                    Version                   Build  Channel
anaconda                  custom           py36hbbc8b67_0
asn1crypto                0.24.0                py36_1003    conda-forge
attrs                     18.2.0                     py_0    conda-forge
biopython                 1.72                     py36_0    conda-forge
blas                      1.0                         mkl
blast                     2.5.0                hc0b0e79_3    bioconda
blat                      36                            0    bioconda
boost                     1.68.0           py36h3e44d54_1    conda-forge
boost-cpp                 1.68.0               h3a22d5f_0    conda-forge
brain-isotopic-distribution 1.4.0                 py36_1000    conda-forge
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
cairo                     1.14.12              he56eebe_3    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
cffi                      1.11.5           py36h5e8e0c9_1    conda-forge
cftime                    1.0.2.1          py36h7eb728f_0    conda-forge
chardet                   3.0.4                 py36_1003    conda-forge
coinmp                    1.8.3                         0    conda-forge
conda                     4.5.12                py36_1000    conda-forge
conda-env                 2.6.0                         1    conda-forge
coreutils                 8.30                 h470a237_0    conda-forge
cryptography              2.3.1            py36hdffb7b8_0    conda-forge
cryptography-vectors      2.3.1                 py36_1000    conda-forge
cufflinks                 2.2.1                    py36_2    bioconda
curl                      7.61.0               h93b3f91_2    conda-forge
cycler                    0.10.0                     py_1    conda-forge
dbus                      1.13.0               h3a4f0e9_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
eigen                     3.3.5                h2d50403_1    conda-forge
expat                     2.2.5                hfc679d8_2    conda-forge
fontconfig                2.13.0               hd36ec8e_5    conda-forge
freetype                  2.8.1                hfa320df_1    conda-forge
gettext                   0.19.8.1             h5e8e0c9_1    conda-forge
glib                      2.55.0               h464dc38_2    conda-forge
glpk                      4.65                 h16a7912_1    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
gnutls                    3.5.19               h2a4e5f8_1    conda-forge
graphite2                 1.3.12               hfc679d8_1    conda-forge
gsl                       2.2.1                h0c605f7_3
gst-plugins-base          1.12.5               hde13a9d_0    conda-forge
gstreamer                 1.12.5               h61a6719_0    conda-forge
harfbuzz                  1.7.6                         0    conda-forge
hdf4                      4.2.13               h951d187_2    conda-forge
hdf5                      1.10.3               hc401514_2    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
idna                      2.7                   py36_1002    conda-forge
intel-openmp              2019.0                      118
ipython_genutils          0.2.0                      py_1    conda-forge
java-jdk                  8.0.92                        1    bioconda
jpeg                      9c                   h470a237_1    conda-forge
jsonschema                3.0.0a3               py36_1000    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
kiwisolver                1.0.1            py36h2d50403_2    conda-forge
krb5                      1.14.6                        0    conda-forge
libboost                  1.67.0               h46d08c1_4
libcurl                   7.61.1               heec0ca6_0
libedit                   3.1.20170329         haf1bffa_1    conda-forge
libevent                  2.0.22               hdffb7b8_2    conda-forge
libffi                    3.2.1                hfc679d8_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libnetcdf                 4.6.1               h9cd6fdc_11    conda-forge
libopenblas               0.2.20               h9ac9557_7
libpng                    1.6.34               ha92aebf_2    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
libsvm                    323                           0    conda-forge
libtiff                   4.0.9                he6b73bb_2    conda-forge
libuuid                   2.32.1               h470a237_2    conda-forge
libxcb                    1.13                 h470a237_2    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
libxslt                   1.1.32               h88dbc4e_2    conda-forge
lxml                      4.2.5            py36hc9114bc_0    conda-forge
matplotlib                2.2.2                    py36_1    conda-forge
mkl                       2018.0.3                      1
mkl_fft                   1.0.10                   py36_0    conda-forge
mkl_random                1.0.2                    py36_0    conda-forge
moff                      2.0.2                    py36_0    bioconda
mono                      5.14.0.177           hfc679d8_0    conda-forge
msgf_plus                 2016.10.26               py36_2    bioconda
msstitch                  2.5                      py36_0    bioconda
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1                  hfc679d8_1    conda-forge
netcdf4                   1.4.2            py36hac939d9_0    conda-forge
nettle                    3.3                           0    conda-forge
nextflow                  18.10.1              ha4d7672_2    bioconda
numpy                     1.15.4           py36h1d66e8a_0
numpy-base                1.15.4           py36h81de0dd_0
openblas                  0.3.3                ha44fe06_1    conda-forge
openjdk                   8.0.152              h46b5887_1
openssl                   1.0.2p               h470a237_1    conda-forge
pandas                    0.23.4           py36hf8a1672_0    conda-forge
pango                     1.40.14              hd50be51_1    conda-forge
pcre                      8.39                          0    conda-forge
percolator                3.1                 boost_1.624    bioconda
perl                      5.26.2               h470a237_0    conda-forge
perl-archive-tar          2.18                    pl526_3    bioconda
perl-carp                 1.38                    pl526_1    bioconda
perl-compress-raw-bzip2   2.074           pl526hfc679d8_0    bioconda
perl-compress-raw-zlib    2.081           pl526h2d50403_0    bioconda
perl-data-dumper          2.161                   pl526_2    bioconda
perl-encode               2.88                    pl526_1    bioconda
perl-exporter             5.72                    pl526_1    bioconda
perl-exporter-tiny        1.000000                pl526_0    bioconda
perl-extutils-makemaker   7.34                    pl526_2    bioconda
perl-io-compress          2.069           pl526hfc679d8_5    bioconda
perl-io-zlib              1.10                    pl526_2    bioconda
perl-list-moreutils       0.428                   pl526_1    bioconda
perl-list-moreutils-xs    0.428                   pl526_0    bioconda
perl-parent               0.236                   pl526_1    bioconda
perl-pathtools            3.73                 h470a237_2    bioconda
perl-scalar-list-utils    1.45            pl526h470a237_3    bioconda
perl-test-more            1.001002                pl526_1    bioconda
perl-xsloader             0.24                    pl526_0    bioconda
pip                       18.1                  py36_1000    conda-forge
pixman                    0.34.0               h470a237_3    conda-forge
plotly                    3.4.2                      py_0    conda-forge
pthread-stubs             0.4                  h470a237_1    conda-forge
py-boost                  1.67.0           py36h04863e7_4
pybigwig                  0.3.12           py36hdfb72b2_2    bioconda
pycosat                   0.6.3            py36h470a237_1    conda-forge
pycparser                 2.19                       py_0    conda-forge
pymzml                    2.0.6                      py_0    bioconda
pynumpress                0.0.3            py36h24bf2e0_0    conda-forge
pyopenssl                 18.0.0                py36_1000    conda-forge
pyparsing                 2.3.0                      py_0    conda-forge
pyqt                      5.6.0            py36h8210e8a_8    conda-forge
pyrsistent                0.14.8           py36h470a237_0    conda-forge
pysocks                   1.6.8                 py36_1002    conda-forge
pyteomics                 3.5.1                      py_2    bioconda
python                    3.6.6                h5001a0f_3    conda-forge
python-dateutil           2.7.5                      py_0    conda-forge
pytz                      2018.7                     py_0    conda-forge
pyyaml                    3.13             py36h470a237_1    conda-forge
qt                        5.6.2                h50c60fd_8    conda-forge
r                         3.5.1                    r351_0    r
r-base                    3.5.1                h4fe35fd_0    conda-forge
r-boot                    1.3_20           r351hf348343_0    r
r-class                   7.3_14           r351hd10c6a6_4    r
r-cluster                 2.0.7_1          r351hac1494b_0    r
r-codetools               0.2_15           r351h6115d3f_0    r
r-foreign                 0.8_71           r351h96ca727_0    r
r-kernsmooth              2.23_15          r351hac1494b_4    r
r-lattice                 0.20_35          r351h96ca727_0    r
r-mass                    7.3_50           r351h96ca727_0    r
r-matrix                  1.2_14           r351h96ca727_0    r
r-mgcv                    1.8_24           r351h96ca727_0    r
r-nlme                    3.1_137          r351ha65eedd_0    r
r-nnet                    7.3_12           r351h96ca727_0    r
r-recommended             3.5.1                    r351_0    r
r-rpart                   4.1_13           r351hd10c6a6_0    r
r-spatial                 7.3_11           r351hd10c6a6_4    r
r-survival                2.42_6           r351h96ca727_0    r
readline                  7.0                  haf1bffa_1    conda-forge
requests                  2.20.0                py36_1000    conda-forge
retrying                  1.3.3                      py_2    conda-forge
ruamel_yaml               0.15.71          py36h470a237_0    conda-forge
scikit-learn              0.19.1           py36hedc7406_0
scipy                     1.1.0            py36hc49cb51_0
setuptools                40.5.0                   py36_0    conda-forge
simplejson                3.16.1           py36h470a237_0    conda-forge
sip                       4.18.1           py36hfc679d8_0    conda-forge
six                       1.11.0                py36_1001    conda-forge
sqlalchemy                1.2.15           py36h470a237_0    conda-forge
sqlite                    3.25.2               hb1c47c0_0    conda-forge
tk                        8.6.8                ha92aebf_0    conda-forge
tmux                      2.7                  hc78d2af_3    conda-forge
tokyocabinet              1.4.48               h96824bc_3    conda-forge
tornado                   5.1.1            py36h470a237_0    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
urllib3                   1.23                  py36_1001    conda-forge
wheel                     0.32.2                   py36_0    conda-forge
xerces-c                  3.1.2                         0
xorg-kbproto              1.0.7                h470a237_2    conda-forge
xorg-libice               1.0.9                h470a237_4    conda-forge
xorg-libsm                1.2.3                h8c8a85c_0    conda-forge
xorg-libx11               1.6.6                h470a237_0    conda-forge
xorg-libxau               1.0.8                h470a237_6    conda-forge
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge
xorg-libxext              1.3.3                h470a237_4    conda-forge
xorg-libxrender           0.9.10               h470a237_2    conda-forge
xorg-libxt                1.1.5                h470a237_2    conda-forge
xorg-renderproto          0.11.1               h470a237_2    conda-forge
xorg-xextproto            7.3.0                h470a237_2    conda-forge
xorg-xproto               7.0.31               h470a237_7    conda-forge
xsd                       4.0.0_dep                     0    bioconda
xz                        5.2.4                h470a237_1    conda-forge
yaml                      0.1.7                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge
Maux82 commented 5 years ago

Hi, the conda env seems good. Try to run it, modifying the 'CPU' parameter in the conf file from 0 to 4. In the example data, one of the run has really few identifications, if the number of CPU is higher (> 4) it could brake. Let me know if this solve the issue.

yafeng commented 5 years ago

I did the changes you said. It works now! Thanks!