'StanModel' object has no attribute 'fit_class'

[saketkc]

Hi, I am having trobule getting the rp-bp pipeline running. It seems to be arising from pystan, however, I have the correct version installed. I can't seem to get to the root of the error and hence decided to open this issue.

My environment file is attached. Here is the relevant step and failure:

(1)> cat failed_command.sh 
estimate-metagene-profile-bayes-factors /staging/as/skchoudh/rpbp_test_hg38/metagene-profiles/RIBO_CCDS_lengths.alignments-only-unique.metagene-profile.csv.gz /staging/as/skchoudh/rpbp_test_hg38/metagene-profiles/RIBO_CCDS_lengths.alignments-only-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 6 --periodic-models /home/rcf-40/skchoudh/.local/share/rpbp/rpbp_models/periodic/start-high-low-low.pkl --nonperiodic-models /home/rcf-40/skchoudh/.local/share/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl /home/rcf-40/skchoudh/.local/share/rpbp/rpbp_models/nonperiodic/start-high-high-low.pkl /home/rcf-40/skchoudh/.local/share/rpbp/rpbp_models/nonperiodic/start-high-low-high.pkl --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --stderr-logging-level NOTSET --log-file log.alignments-only_hg38.txt --logging-level INFO --file-logging-level NOTSET --stdout-logging-level NOTSET

(0)> bash failed_command.sh 
INFO     riboutils.estimate_metagene_profile_bayes_factors 2018-12-09 21:29:22,780 : Estimating Bayes factors for lengths: 28,30
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
    r = call_item()
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/externals/loky/process_executor.py", line 272, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/_parallel_backends.py", line 567, in __call__
    return self.func(*args, **kwargs)
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 114, in estimate_profile_bayes_factors
    iterations=args.iterations,chains=args.chains,seed=args.seed)
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 63, in estimate_marginal_likelihoods
    for pm in periodic_models]
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 63, in <listcomp>
    for pm in periodic_models]
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/pystan/model.py", line 721, in sampling
    fit = self.fit_class(data, seed)
AttributeError: 'StanModel' object has no attribute 'fit_class'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/bin/estimate-metagene-profile-bayes-factors", line 11, in <module>
    sys.exit(main())
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 222, in main
    progress_bar=True
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/misc/parallel.py", line 101, in apply_parallel_groups
    for name,group in tqdm.tqdm(groups, total=len(groups), leave=True, file=sys.stdout))
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/parallel.py", line 930, in __call__
    self.retrieve()
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/rpbp/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception

rpbp_env.txt

I really appreciate your efforts on rp-bp!

[eboileau]

I need to fix versions, I think this may help solving some issues. Meanwhile, can you make sure you have these versions installed:

biopython==1.70
Cython==0.28
dask==0.17.1
distributed==1.21.3
fastparquet==0.1.4
joblib==0.11
llvmlite==0.22.0
numpy==1.14.2
pandas==0.22.0
patsy==0.5.0
pysam==0.14
scikit-learn==0.19.1
scipy==1.0.0
tornado==5.0
tqdm==4.19.7
sklearn==0.0
statsmodels==0.8.0
psutil==5.4.3

In particular, joblib has changed it's default backend from multiprocessing to loky since version 0.12, and I am not sure yet of the implications, but I suspect this may be the issue. Can you try fixing versions of the packages in a fresh virtual environment?

[saketkc]

Thanks for your suggestion @eboileau. However that doesn't help either. Attached are the error and the new environment.

Thanks! stderr.txt rpbp_env_v2.txt

[eboileau]

Hi, when re-installing, the models under .local/share/rpbp/rpbp_models would not recompile by default, it may be that they did not compile correctly initially. Did you keep an install log? You could always delete the models and/or force recompilation. Honestly, I am not sure, as I cannot reproduce this error. Could you just check if pystan is working correctly:

import pystan
model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
model = pystan.StanModel(model_code=model_code) 
y = model.sampling(n_jobs=1).extract()['y']
y.mean()  # ~ 0

[eboileau]

@saketkc, the latest dev branch upgraded most dependencies, and fixes the joblib backend problem. You can also reinstall and force the recompilation of the Stan models using the specified version of Pystan.