Getting more errors running `pipeline.py` on trajectories saved in `munged3`

[sonyahanson]

Not quite sure how these are all related, but so far running pipeline.py on three different datasets gets three different errors (the first is in #12).

I did not get any of these problems running on a small number of trajectories (python pipeline.py '/cbio/jclab/projects/fah/fah-data/munged3/no-solvent/11400/run0-clone*0.h5').

The new errors are (for 11400):

    self._model = self._estimate(X)
  File "/cbio/jclab/home/hansons/opt/anaconda/lib/python2.7/site-packages/pyEMMA-2.2.2-py2.7-linux-x86_64.egg/pyemma/msm/estimators/maximum_likelihood_msm.py", line 220, in _estimate
    dtrajstats = _DiscreteTrajectoryStats(dtrajs)
  File "/cbio/jclab/home/hansons/opt/anaconda/lib/python2.7/site-packages/pyEMMA-2.2.2-py2.7-linux-x86_64.egg/pyemma/msm/estimators/_dtraj_stats.py", line 47, in __init__
    self._hist = msmest.count_states(self._dtrajs)
  File "/cbio/jclab/home/hansons/opt/anaconda/lib/python2.7/site-packages/msmtools-1.1-py2.7-linux-x86_64.egg/msmtools/estimation/api.py", line 104, in count_states
    return _count_states(dtrajs)
  File "/cbio/jclab/home/hansons/opt/anaconda/lib/python2.7/site-packages/msmtools-1.1-py2.7-linux-x86_64.egg/msmtools/dtraj/api.py", line 330, in count_states
    bc = np.bincount(dtraj)
ValueError: The first argument of bincount must be non-negative

(for 11406):

    run_pipeline(fnames, project_name = project_name)
  File "pipeline.py", line 130, in run_pipeline
    write_pdbs_of_clusters(source_full, msm, project_name)
  File "pipeline.py", line 170, in write_pdbs_of_clusters
    pyemma.coordinates.save_traj(source, samples[i], '{0}_state_{1}.pdb'.format(project_name, i))
  File "/cbio/jclab/home/hansons/opt/anaconda/lib/python2.7/site-packages/pyEMMA-2.2.2-py2.7-linux-x86_64.egg/pyemma/coordinates/api.py", line 658, in save_traj
    traj = frames_from_files(trajfiles, top, indexes, chunksize, stride, reader=reader)
  File "/cbio/jclab/home/hansons/opt/anaconda/lib/python2.7/site-packages/pyEMMA-2.2.2-py2.7-linux-x86_64.egg/pyemma/coordinates/data/util/frames_from_file.py", line 111, in frames_from_files
    length))
ValueError: largest specified index (487 * stride=487 * 1=487) is larger than trajectory length '/cbio/jclab/projects/fah/fah-data/munged3/no-solvent/11406/run8-clone35.h5' = 320

[maxentile]

Thanks for pointing these out! Will try to see what's causing these...

For the bincount error, I guess that means that some entries in dtrajs are -1? Not sure where that would happen.

For the frames_from_files error, my best guess is that source and source_full are seeing different length trajectories, since the munging pipeline is running at the same time in that directory. How do you think we should tackle this? Maybe just moving the definition of source_full from line 128 to be immediately after the definition of source on line 98 would solve that issue...

[jchodera]

Maybe just moving the definition of source_full from line 128 to be immediately after the definition of source on line 98 would solve that issue...

Worth a try, but maybe post an issue on the pyemma issue tracker? It would be useful for the pyemma tools to be robust to the number of frames increasing in such a situation as the calculation proceeds.

As a temporary workaround, you could probably rsync the (no-solvent) h5 and pdb files to a staging trajectory, like /cbio/jclab/projects/fah/fah-msm/staging immediately before analyzing them. Each project is generally << 1 TB.

[sonyahanson]

Hmm... Indeed when I come across this error again, and I look at dtrajs.npy:

 for sublist in dtrajs:
    if all(i >=0 for i in sublist) == False:
       print sublist
   ....:         
[149 415 415 ..., 149 149  -1]

[maxentile]

I think that's good evidence this error is because the trajectory files are being written to while the script is running

[jchodera]

Are you waiting for me to report this to the pyemma issue tracker, or can one of you do that?

[sonyahanson]

we are not waiting for you

[maxentile]

I'll report it now!