search

markovmodel / PyEMMA

🚂 Python API for Emma's Markov Model Algorithms 🚂
http://pyemma.org
GNU Lesser General Public License v3.0
311 stars 119 forks source link

logfile error #750

Closed franknoe closed 8 years ago

franknoe commented 8 years ago

This error occurs when running multiple concurrent PyEMMA runs (which do something completely harmless have no I/O, like solving a generalized eigenvalue problem). Looks like a failure to concurrently access the same logfile or directory for writing, although I don't know what is being logged here. I guess multiple processes importing PyEMMA at the same time?

This occurs with version 2.0.4. It it fixed in the 2.1rc?

Generally, I think we should not try to write any files unless the user explicitly wants to or if it's necessary for functionality. What is the functionality of this one? 

Traceback (most recent call last):

  File "/home/cclementi/oASIS_TrypsinBenzamidine/ManyRuns/CLEAN_oASIS_sparse_tica_tryp-ben-FULLDATASET-ContactMapALLATOMS_and_BENZ.py", line 8, in <module>

    from pyemma.util.linalg import eig_corr

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.0.4-py2.7-linux-x86_64.egg/pyemma/__init__.py", line 27, in <module>

    from . import coordinates

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.0.4-py2.7-linux-x86_64.egg/pyemma/coordinates/__init__.py", line 107, in <module>

    from .api import *

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.0.4-py2.7-linux-x86_64.egg/pyemma/coordinates/api.py", line 24, in <module>

    from pyemma.util.log import getLogger as _getLogger

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.0.4-py2.7-linux-x86_64.egg/pyemma/util/log.py", line 110, in <module>

    setupLogging()

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.0.4-py2.7-linux-x86_64.egg/pyemma/util/log.py", line 49, in setupLogging

    cfg_dir = config.create_cfg_dir(def_conf_file)

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.0.4-py2.7-linux-x86_64.egg/pyemma/util/config.py", line 134, in create_cfg_dir

    raise RuntimeError("%s is not writeable" % pyemma_cfg_dir)

RuntimeError: /home/cclementi/.pyemma is not writeable
marscher commented 8 years ago

So you only start multiple Python interpreters, which use pyemma for this to happen? Please test the RC, because the config initialisation has been slightly changed.

This code branch is responsible for setting up the logging system, runtime configuration (like switching progress bars) etc. So far no problem occurred with that (we use it in fact since 1.2). What kind of file system is that? The check if this directory is writable, does in fact not touch any files, but just checks the permissions.

@fabian-paul has mentioned something like that in conjunction with MPI. But I guess this is not the case here.

franknoe commented 8 years ago

OK, we can try with the RC. Yes, it's multiple Python interpreters started through a queuing system, and I guess it's a concurrency issues. Similar issues have occurred in an MPI environment.

In any case we should discuss this configuration handling. I really believe that PyEMMA should not be writing any files unless you ask it to, and that configuration information should not be stored between sessions.

Am 30/03/16 um 11:36 schrieb Martin K. Scherer:

So you only start multiple Python interpreters, which use pyemma for this to happen? Please test the RC, because the config initialisation has been slightly changed.

This code branch is responsible for setting up the logging system, runtime configuration (like switching progress bars) etc. So far no problem occurred with that (we use it in fact since 1.2). What kind of file system is that? The check if this directory is writable, does in fact not touch any files, but just checks the permissions.

@fabian-paul https://github.com/fabian-paul has mentioned something like that in conjunction with MPI. But I guess this is not the case here.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-203346191

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

marscher commented 8 years ago

We can disable this automatic saving by introducing an environment variable like PYEMMA_NO_CONFIG to disable usage of the config system at all (we would then need to hardcode some default values in the source code). Having the same configuration for multiple runs is in my opinion a nice thing to have.

marscher commented 8 years ago

Anyway I would like to reproduce this somehow. So please provide some information.

franknoe commented 8 years ago

Let's discuss it later in person. I don't understand the usecase.

Am 30/03/16 um 11:44 schrieb Martin K. Scherer:

We can disable this automatic saving by introducing an environment variable like PYEMMA_NO_CONFIG to disable usage of the config system at all (we would then need to hardcode some default values in the source code). Having the same configuration for multiple runs is in my opinion a nice thing to have.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-203350443

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

franknoe commented 8 years ago

@cclementi ran into this error, she'll be around later for discussion and show you her setup. But I think it will be hard to reproduce, because it occurs randomly - our guess is that multiple processes try to import PyEMMA simultaneously on the same machine (I think that import pyemma is sufficient to trigger it).

Am 30/03/16 um 11:44 schrieb Martin K. Scherer:

Anyway I would like to reproduce this somehow. So please provide some information.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-203350539

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

cclementi commented 8 years ago

Hi Martin,

I am on my way to FU, and available to tell you about this if you like. It’s a bit annoying that when I run multiple jobs that require to import pyemma they can randomly die because they can’t access this file at the same time. Actually, I would prefer not to have a log file automatically (or at least an option to switch it off).

Thanks, -Cecilia

On Mar 30, 2016, at 11:49 AM, Frank Noe notifications@github.com wrote:

@cclementi ran into this error, she'll be around later for discussion and show you her setup. But I think it will be hard to reproduce, because it occurs randomly - our guess is that multiple processes try to import PyEMMA simultaneously on the same machine (I think that import pyemma is sufficient to trigger it).

Am 30/03/16 um 11:44 schrieb Martin K. Scherer:

Anyway I would like to reproduce this somehow. So please provide some information.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-203350539

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-203352126

marscher commented 8 years ago

In the meantime you could also do something like this to give each job its own directory:

directory=$(mktemp -d)
export PYEMMA_CFG_DIR=$directory
python myscript.py
marscher commented 8 years ago

in 2.1 all config initialization errors are non-fatal (due to the request of @fabian-paul before christmas). The user will only see some warnings in this case. The config variables are then taken from the shipped default files

@cclementi I suggest you use 2.1rc3 or latest devel (pip install https://github.com/markovmodel/pyemma@devel)

franknoe commented 8 years ago

is this issue solved - @cclementi?

Am 30/03/16 um 15:38 schrieb Martin K. Scherer:

Closed #750 https://github.com/markovmodel/PyEMMA/issues/750.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#event-608272692

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

franknoe commented 8 years ago

Issue is not solved in 2.1. Please don't close issues unless they are actually solved.

Traceback (most recent call last):

  File "/home/cclementi/oASIS_TrypsinBenzamidine/ManyRuns/CoarseRes_oASIS_sparse_tica_tryp-ben-FULLDATASET-ContactMapALLATOMS_and_BENZ.py", line 8, in <module>

    from pyemma.util.linalg import eig_corr

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.1rc3-py2.7-linux-x86_64.egg/pyemma/__init__.py", line 27, in <module>

    from . import coordinates

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.1rc3-py2.7-linux-x86_64.egg/pyemma/coordinates/__init__.py", line 107, in <module>

    from .api import *

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.1rc3-py2.7-linux-x86_64.egg/pyemma/coordinates/api.py", line 24, in <module>

    from pyemma.util.log import getLogger as _getLogger

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.1rc3-py2.7-linux-x86_64.egg/pyemma/util/log.py", line 123, in <module>

    setupLogging()

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.1rc3-py2.7-linux-x86_64.egg/pyemma/util/log.py", line 49, in setupLogging

    cfg_dir = config.create_cfg_dir(def_conf_file)

  File "/home/cclementi/anaconda2/lib/python2.7/site-packages/pyEMMA-2.1rc3-py2.7-linux-x86_64.egg/pyemma/util/config.py", line 134, in create_cfg_dir

    raise RuntimeError("%s is not writeable" % pyemma_cfg_dir)

RuntimeError: /home/cclementi/.pyemma is not writeable
marscher commented 8 years ago

You're right, this is not released, but is contained on devel. If you have a look here: https://github.com/markovmodel/PyEMMA/blob/devel/pyemma/util/config.py#L125 You see that exceptions during the initialisation are non-fatal.

marscher commented 8 years ago

ping @cclementi Can you please confirm, that this issue does not appear any more in 2.1?

cclementi commented 8 years ago

Hi Martin,

I haven't experienced this issue again since using the devel version. I haven't thoroughly checked it, though. I will do it as soon as I have some spare time.

Best, -Cecilia

On Thu, Apr 7, 2016 at 4:05 PM, Martin K. Scherer notifications@github.com wrote:

ping @cclementi https://github.com/cclementi Can you please confirm, that this issue does not appear any more in 2.1?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-206922703

marscher commented 8 years ago

Thank you!

franknoe commented 8 years ago

I think we can close this for now, as we have follow-up issues dealing with writing files.

Am 07/04/16 um 17:52 schrieb Cecilia Clementi:

Hi Martin,

I haven't experienced this issue again since using the devel version. I haven't thoroughly checked it, though. I will do it as soon as I have some spare time.

Best, -Cecilia

On Thu, Apr 7, 2016 at 4:05 PM, Martin K. Scherer notifications@github.com wrote:

ping @cclementi https://github.com/cclementi Can you please confirm, that this issue does not appear any more in 2.1?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-206922703

— You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub https://github.com/markovmodel/PyEMMA/issues/750#issuecomment-206966424

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany