search

julie-forman-kay-lab / IDPConformerGenerator

Build conformational representations of Intrinsically Disordered Proteins and Regions by a guided sampling of the protein torsion space
https://idpconformergenerator.readthedocs.io/
Apache License 2.0
19 stars 6 forks source link

idpconfgen build stuck on macOS python 3.11.7 #262

Closed benjbuch closed 10 months ago

benjbuch commented 10 months ago

Running the example files (small peptide as well as drksh3) step by step (system specs below), the build process doesn't show any activity (memory or CPU usage) for hours which I believe is wrong. Taking a sample, it all threads appear to remain uncollected in __psynch_cvwait (in libsystem_kernel.dylib).

This is the keyboard interrupt:

idpconfgen build -db idpconfgen_database.json -seq EGAAGAASS -nc 10 --dloop-off --dany -n
[2024-01-04 13:17:18,000]Setting Up Working Directory:
[2024-01-04 13:17:18,000]    Using current working directory: /Users/benjamin/Executables/IDPConformerGenerator/example/small_peptide.
[2024-01-04 13:17:18,000]    Previous data will be overwritten, if file names overlap.
[2024-01-04 13:17:18,001]Starting The Building Process:
[2024-01-04 13:17:18,001]    input sequence: EGAAGAASS
[2024-01-04 13:17:18,001]running in 7 cores with 3 remaining confs
[2024-01-04 13:17:18,001]Fragment Size And Probabilities Selected:
[2024-01-04 13:17:18,001]    sizes of: (1, 2, 3, 4, 5)
[2024-01-04 13:17:18,001]    probs of: (0.1, 0.1, 0.3, 0.3, 0.2)
[2024-01-04 13:17:18,018]preparing regex xmers
[2024-01-04 13:17:18,018]    searching database with DSSP regexes: ([HGETS BIPL]+)
 100.0% 5/5 on 7 cpus |██████████████████████████████████████████████████████████████████████|
[2024-01-04 13:17:18,710]Corrected Fragment Selection Probabilities:
[2024-01-04 13:17:18,710]    sizes of: (1, 2, 3, 4)
[2024-01-04 13:17:18,710]    probs of: (0.125, 0.125, 0.37499999999999994, 0.37499999999999994)
[2024-01-04 13:17:26,005]    Lennard-Jones type configured to: 'pairs'
[2024-01-04 13:17:26,006]prepared lj
[2024-01-04 13:17:26,006]done preparing energy func
[2024-01-04 13:17:26,025]    Lennard-Jones type configured to: 'pairs'
[2024-01-04 13:17:26,025]prepared lj
[2024-01-04 13:17:26,025]done preparing energy func
^CProcess SpawnPoolWorker-9:
Process SpawnPoolWorker-13:
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-12:
Process SpawnPoolWorker-11:
Process SpawnPoolWorker-14:
Process SpawnPoolWorker-10:
Traceback (most recent call last):
  File "/usr/local/Miniconda/envs/idpconfgen/lib/python3.11/multiprocessing/pool.py", line 856, in next
    item = self._items.popleft()
           ^^^^^^^^^^^^^^^^^^^^^
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Miniconda/envs/idpconfgen/bin/idpconfgen", line 33, in <module>
    sys.exit(load_entry_point('idpconfgen', 'console_scripts', 'idpconfgen')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/IDPConformerGenerator/src/idpconfgen/cli.py", line 142, in maincli
    cmd.func(**vars(cmd))
  File "~/IDPConformerGenerator/src/idpconfgen/cli_build.py", line 809, in main
    for _ in imap:
  File "/usr/local/Miniconda/envs/idpconfgen/lib/python3.11/multiprocessing/pool.py", line 861, in next
    self._cond.wait(timeout)
  File "/usr/local/Miniconda/envs/idpconfgen/lib/python3.11/threading.py", line 327, in wait
    waiter.acquire()
KeyboardInterrupt

System:

RAM and CPU usage have not been exhausted.

Not sure what to change.

menoliu commented 10 months ago

Hi @benjbuch thanks for submitting this issue! Let's try to de-bug this together. I know that MacOS and DSSP have some compatibility issues so my first thought is a database error. You can download the database I'm currently using via DropBox here. Also just out of curiosity if you want to run with -n 5 as it's a multiple of -nc 10.

If you could attach the error file generated by IDPConformerGenerator (should be a hidden file in your output/working directory) that would help a lot as well.

Edit: If you're ever considering running IDPConformerGenerator on Linux or Windows Subsystem for Linux (preferred), please note that we only support DSSP up to version 3 as indicated in the documentation.

Edit 2: Another idea came to mind is disabling the side-chain packing stage if FASPR didn't install correctly since that's written in C. You can do so with the -dsd flag.

joaomcteixeira commented 10 months ago

Hi @benjbuch

Thanks for reaching out to us and for using IDPConfGen. On top of what @menoliu said, I have run the small_peptide example with the latest IDPCG version, 0.7.14, and it works properly. Below is the output log so you know what to expect.

The first thing we ask you is to update IDPCG to the latest version to ensure we are on the same page. From the general log, I cannot understand where the error is. Why are the random seed numbers not assigned? As @menoliu said, could you please send us the detailed logs? Those are the .idpconfgen_* files stored in the run folder.

Finally, can you test the -nc 10 -n 5 option @menoliu told you? This should not affect, but if it affects we have found something :wink:

EDIT: You can also send us the database created from the cull100 file so we can check if there's a mistake there related to DSSP as @menoliu suggested.

Cheers,

· idpconfgen build -db idpconfgen_database.json -seq EGAAGAASS -nc 10 --dloop-off --dany -n
[2024-01-04 22:23:22,490]Setting Up Working Directory:
[2024-01-04 22:23:22,490]    Using current working directory: /home/joao/github/IDPConformerGenerator/example/small_peptide.
[2024-01-04 22:23:22,490]    Previous data will be overwritten, if file names overlap.
[2024-01-04 22:23:22,490]Starting The Building Process:
[2024-01-04 22:23:22,491]    input sequence: EGAAGAASS
[2024-01-04 22:23:22,491]running in 7 cores with 3 remaining confs
[2024-01-04 22:23:22,491]Fragment Size And Probabilities Selected:
[2024-01-04 22:23:22,491]    sizes of: (1, 2, 3, 4, 5)
[2024-01-04 22:23:22,491]    probs of: (0.1, 0.1, 0.3, 0.3, 0.2)
[2024-01-04 22:23:22,507]preparing regex xmers
[2024-01-04 22:23:22,507]    searching database with DSSP regexes: ([HGETS BIPL]+)
 100.0% 5/5 on 7 cpus |██████████████████████████████████████████████████████████████████████████████████████|
[2024-01-04 22:23:22,561]Corrected Fragment Selection Probabilities:
[2024-01-04 22:23:22,561]    sizes of: (1, 2, 3, 4)
[2024-01-04 22:23:22,561]    probs of: (0.125, 0.125, 0.37499999999999994, 0.37499999999999994)
[2024-01-04 22:23:31,241]    Lennard-Jones type configured to: 'pairs'
[2024-01-04 22:23:31,242]prepared lj
[2024-01-04 22:23:31,242]done preparing energy func
[2024-01-04 22:23:31,260]    Lennard-Jones type configured to: 'pairs'
[2024-01-04 22:23:31,260]prepared lj
[2024-01-04 22:23:31,260]done preparing energy func
[2024-01-04 22:23:31,369]random seed: 0
[2024-01-04 22:23:31,369]random seed: 1
[2024-01-04 22:23:31,369]random seed: 3
[2024-01-04 22:23:31,369]random seed: 2
[2024-01-04 22:23:31,369]random seed: 4
[2024-01-04 22:23:31,369]random seed: 5
[2024-01-04 22:23:31,369]random seed: 6
[2024-01-04 22:23:31,370]    configuring sidechain method: faspr
[2024-01-04 22:23:31,370]    configuring sidechain method: faspr
[2024-01-04 22:23:31,370]    configuring sidechain method: faspr
[2024-01-04 22:23:31,370]    configuring sidechain method: faspr
[2024-01-04 22:23:31,370]    configuring sidechain method: faspr
[2024-01-04 22:23:31,370]    configuring sidechain method: faspr
[2024-01-04 22:23:31,370]    configuring sidechain method: faspr
[2024-01-04 22:24:00,935]seed 1 - finished conf: 1 with energy -30.72579174466174
[2024-01-04 22:24:02,368]seed 2 - finished conf: 1 with energy -22.602166585271966
[2024-01-04 22:24:03,160]seed 6 - finished conf: 1 with energy -47.32615496751219
[2024-01-04 22:24:03,601]seed 4 - finished conf: 1 with energy -18.264621420747247
[2024-01-04 22:24:03,659]seed 5 - finished conf: 1 with energy -51.652707725920735
[2024-01-04 22:24:03,959]seed 0 - finished conf: 1 with energy -26.93900483233467
[2024-01-04 22:24:04,002]seed 3 - finished conf: 1 with energy -31.617709795974918
[2024-01-04 22:24:04,132]random seed: 7
[2024-01-04 22:24:04,132]    configuring sidechain method: faspr
[2024-01-04 22:24:20,246]seed 7 - finished conf: 1 with energy -39.069678774751935
[2024-01-04 22:24:20,265]seed 7 - finished conf: 2 with energy -84.08267392453207
[2024-01-04 22:24:20,287]seed 7 - finished conf: 3 with energy -47.52029127357153
[2024-01-04 22:24:20,289]10 conformers built in 49.028 seconds
[2024-01-04 22:24:20,289]    finished properly
benjbuch commented 10 months ago

Hi @joaomcteixeira and @menoliu,

Thank you for your help and all your work that has fueled in this project!

I've reinstalled conda (mamba v1.4.2, conda 23.3.1) running python (v3.11.7) and reinstalled idpconfgen (v0.7.14), mkdssp (v2.2.1), and the latest MC-SCE. [Not sure if all of this is necessary, but I'd rather err on this side after the recent macOS updates.]

Test 1: Maybe it's the dssp version (@menoliu indicated that only v3 is supported; the current documentation reads v2 or v3 https://idpconformergenerator.readthedocs.io/en/latest/installation.html); I'll try to upgrade dssp.

However, using idpconfgen_database_2023.json did not solve the issue. So maybe it's not the dssp version.

Test 2: Same with specifying the number of cores -n 5. So, you were right, this is not reason. :-)

Test 3: Same with disabling FASPR -dsd. I think, I'm not even reaching that stage.

Question 4: Is there a way to specify the allocated memory? When I look into common issues with multiprocessing in python, explicitly specifying the memory to use seems to be a solution. (It's just a guess.)

This is an example of what I get with my database:

idpconfgen build -db idpconfgen_database.json -seq EGAAGAASS -nc 10 --dloop-off --dany -n
[2024-01-04 20:09:20,336]Setting Up Working Directory:
[2024-01-04 20:09:20,336]    Using current working directory: ~/IDPConformerGenerator/example/small_peptide.
[2024-01-04 20:09:20,336]    Previous data will be overwritten, if file names overlap.
[2024-01-04 20:09:20,337]Starting The Building Process:
[2024-01-04 20:09:20,337]    input sequence: EGAAGAASS
[2024-01-04 20:09:20,337]running in 7 cores with 3 remaining confs
[2024-01-04 20:09:20,337]Fragment Size And Probabilities Selected:
[2024-01-04 20:09:20,337]    sizes of: (1, 2, 3, 4, 5)
[2024-01-04 20:09:20,338]    probs of: (0.1, 0.1, 0.3, 0.3, 0.2)
[2024-01-04 20:09:20,352]preparing regex xmers
[2024-01-04 20:09:20,353]    searching database with DSSP regexes: ([HGETS BIPL]+)
 100.0% 5/5 on 7 cpus |█████████████████████████████████████████████████████████████|
[2024-01-04 20:09:21,089]Corrected Fragment Selection Probabilities:
[2024-01-04 20:09:21,089]    sizes of: (1, 2, 3, 4)
[2024-01-04 20:09:21,089]    probs of: (0.125, 0.125, 0.37499999999999994, 0.37499999999999994)
[2024-01-04 20:09:28,500]    Lennard-Jones type configured to: 'pairs'
[2024-01-04 20:09:28,502]prepared lj
[2024-01-04 20:09:28,502]done preparing energy func
[2024-01-04 20:09:28,520]    Lennard-Jones type configured to: 'pairs'
[2024-01-04 20:09:28,521]prepared lj
[2024-01-04 20:09:28,521]done preparing energy func

From your sample output (which was very helpful, thank you!) I expected to get the next prompt (random seed 0) several seconds thereafter. However, in my case, nothing is happening (for an hour) and, as being said, there is seemingly no activity. (I'll leave it overnight, but I'm skeptical.)

I'm attaching the database file and the (debug) logs. The error logs were empty.

Many thanks!

idpconfgen_database.json

idpconfgen_build.debug.txt idpconfgen_pdbdl.debug.txt idpconfgen_sscalc.debug.txt idpconfgen_torsion.debug.txt

benjbuch commented 10 months ago

In case it is helpful this is the conda environment:

# packages in environment at /usr/local/Conda/miniforge3/envs/idpconfgen:
#
# Name                    Version                   Build  Channel
absl-py                   2.0.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
boost-mac                 1.57.0                        0    salilab
brotli                    1.1.0                h0dc2134_1    conda-forge
brotli-bin                1.1.0                h0dc2134_1    conda-forge
bzip2                     1.0.8                h10d778d_5    conda-forge
ca-certificates           2023.11.17           h8857fd0_0    conda-forge
cachetools                5.3.2                    pypi_0    pypi
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2                    pypi_0    pypi
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.0           py311h7bea37d_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dill                      0.3.7                    pypi_0    pypi
distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
dssp                      2.2.1                         0    salilab
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
flatbuffers               23.5.26                  pypi_0    pypi
fonttools                 4.47.0          py311he705e18_0    conda-forge
freetype                  2.12.1               h60636b9_2    conda-forge
gast                      0.5.4                    pypi_0    pypi
google-auth               2.26.1                   pypi_0    pypi
google-auth-oauthlib      1.2.0                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.60.0                   pypi_0    pypi
h5py                      3.10.0                   pypi_0    pypi
idna                      3.6                      pypi_0    pypi
idpconfgen                0.7.14                    dev_0    <develop>
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
keras                     2.15.0                   pypi_0    pypi
kiwisolver                1.4.5           py311h5fe6e05_1    conda-forge
lcms2                     2.16                 ha2f27b4_0    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libblas                   3.9.0           20_osx64_openblas    conda-forge
libbrotlicommon           1.1.0                h0dc2134_1    conda-forge
libbrotlidec              1.1.0                h0dc2134_1    conda-forge
libbrotlienc              1.1.0                h0dc2134_1    conda-forge
libcblas                  3.9.0           20_osx64_openblas    conda-forge
libclang                  16.0.6                   pypi_0    pypi
libcxx                    16.0.6               hd57cbcb_0    conda-forge
libdeflate                1.19                 ha4e1b8e_0    conda-forge
libexpat                  2.5.0                hf0c8a7f_1    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           13_2_0_h97931a8_1    conda-forge
libgfortran5              13.2.0               h2873a65_1    conda-forge
libjpeg-turbo             3.0.0                h0dc2134_1    conda-forge
liblapack                 3.9.0           20_osx64_openblas    conda-forge
libllvm14                 14.0.6               hc8e404f_4    conda-forge
libopenblas               0.3.25          openmp_hfef2a42_0    conda-forge
libpng                    1.6.39               ha978bb4_0    conda-forge
libsqlite                 3.44.2               h92b6c6a_0    conda-forge
libtiff                   4.6.0                h684deea_2    conda-forge
libwebp-base              1.3.2                h0dc2134_0    conda-forge
libxcb                    1.15                 hb7f2c08_0    conda-forge
libzlib                   1.2.13               h8a1eda9_5    conda-forge
llvm-openmp               17.0.6               hb6ac08f_0    conda-forge
llvmlite                  0.41.1          py311hb5c2e0a_0    conda-forge
markdown                  3.5.1                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.8.2           py311h6eed73b_0    conda-forge
matplotlib-base           3.8.2           py311hd316c10_0    conda-forge
ml-dtypes                 0.2.0                    pypi_0    pypi
multiprocess              0.70.15                  pypi_0    pypi
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.4                  h93d8f39_2    conda-forge
numba                     0.58.1          py311h32f2313_0    conda-forge
numpy                     1.26.3          py311hc43a94b_0    conda-forge
oauthlib                  3.2.2                    pypi_0    pypi
openjpeg                  2.5.0                ha4da562_3    conda-forge
openssl                   3.2.0                hd75f5a5_1    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pathos                    0.3.1                    pypi_0    pypi
pillow                    10.2.0          py311hea5c87a_0    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.1.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
pox                       0.3.3                    pypi_0    pypi
ppft                      1.7.6.7                  pypi_0    pypi
protobuf                  4.23.4                   pypi_0    pypi
pthread-stubs             0.4               hc929b4f_1001    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pybind11                  2.11.1          py311h5fe6e05_2    conda-forge
pybind11-global           2.11.1          py311h5fe6e05_2    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
python                    3.11.7          h9f0c242_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.11                    4_cp311    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scipy                     1.11.4          py311he0bea55_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
tensorboard               2.15.1                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
tensorflow                2.15.0                   pypi_0    pypi
tensorflow-estimator      2.15.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.35.0                   pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
tk                        8.6.13               h1abcd95_1    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tornado                   6.3.3           py311h2725bcf_1    conda-forge
tox                       3.27.1             pyhd8ed1ab_0    conda-forge
tqdm                      4.66.1                   pypi_0    pypi
typing-extensions         4.9.0                    pypi_0    pypi
tzdata                    2023d                h0c530f3_0    conda-forge
urllib3                   2.1.0                    pypi_0    pypi
virtualenv                20.25.0            pyhd8ed1ab_0    conda-forge
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1                   pypi_0    pypi
xorg-libxau               1.0.11               h0dc2134_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h8a1eda9_5    conda-forge
zstd                      1.5.5                h829000d_0    conda-forge
menoliu commented 10 months ago

Thanks @benjbuch , that is very odd, I don't see anything out of the ordinary from your debug files and yes I should have been more clear that we support DSSP up to version 3 (so v2 is included).

This may be a MacOS exclusive issue however... we are currently looking into building a docker container to avoid installation and cross-OS compatibility issues!

benjbuch commented 10 months ago

Yeah, maybe that's the best way forward. It may be related to the specific python build. Thanks for you help!