Model location not detected in OSX

[bmmalone]

Originally from @tmargus in Issue #34:

create_orf_profiles.py can't pick up location of models. Where in configuration files these can be specified or program should figure out itself? Anyway, by hard coding models into create_orf_profiles.py periodic_models_str = "/Users/tmargus/git/rp-bp/rpbp_models/periodic/start-high-low-low.stan" non_periodic_models_str = "/Users/tmargus/git/rp-bp/rpbp_models/nonperiodic/no-periodicity.stan"

This is related to appdirs. It is not immediately clear why they are not found correctly.

[tmargus]

pipe install -e . created appropriate folders for models under ~/Library/Application\ Support/rpbp/rpbp_models/ but all subfolders like periodic/, nonperiodic/, trnslated/, and untranslated/ were empty.

I copied models to their destination

@:rpbp_models$ cp nonperiodic/*  ~/Library/Application\ Support/rpbp/rpbp_models/nonperiodic/
@:rpbp_models$ cp periodic/*     ~/Library/Application\ Support/rpbp/rpbp_models/periodic/
@:rpbp_models$ cp translated/*   ~/Library/Application\ Support/rpbp/rpbp_models/translated/
@:rpbp_models$ cp untranslated/* ~/Library/Application\ Support/rpbp/rpbp_models/untranslated/

but it didn't help. It still don't find models. estimate-metagene-profile-bayes-factors: error: argument --periodic-models: expected at least one argument Maybe it remembers what it have copied. Tonu

[bmmalone]

Hi Tonu,

Internally, the .stan files (which are just plain text files) are (supposed to be) compiled and saved as pickle (.pkl) files in the respective folders. The pipeline scans those folders for .pkl during execution, so it wouldn't detect the original stan model files.

It seems as though the pickling and saving isn't working as expected.

Could you please try completely removing the ~/Library/Application\ Support/rpbp/ folder (and everything underneath it), and then re-installing with pip3 install --verbose --log log.rpbp-install.txt . ?

In particular, towards the end of that, there should be some lines like:

INFO : pickle-stan rpbp_models/nonperiodic/no-periodicity.stan /home/bmalone/.local/share/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl

The second argument to pickle-stan is exactly where the pickled model should be saved. Also, in case there are any error messages during compilation, they will show up there.

Could you please let me know (upload the file, etc.) if there is anything interesting going on there? Thanks.

Have a good day, Brandon

[tmargus]

Hi Brnadon, installation script cant cover space between Application and Supprot splits it into two should write like ../Application\ Supprot/.. but miss the backslash.

INFO     : pickle-stan rpbp_models/nonperiodic/no-periodicity.stan /Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl
    usage: pickle-stan [-h] stan out
    pickle-stan: error: unrecognized arguments: Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl

Tonu

[bmmalone]

Ah, okay. could you please try manually this command (from the main rpbp folder):

pickle-stan "rpbp_models/nonperiodic/no-periodicity.stan" "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl"

If that works, then I can quickly put together a fix.

Thanks, Brandon

[tmargus]

Hi,

works fine!

pickle-stan "rpbp_models/nonperiodic/no-periodicity.stan" "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl"
DIAGNOSTIC(S) FROM PARSER:
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.
Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead.

INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_1c0d8baec383282006d5731a3770192a NOW.

Tonu

On 1. veebr 2017, at 20:42, bmmalone notifications@github.com wrote:

Ah, okay. could you please try manually this command (from the main rpbp folder):

pickle-stan "rpbp_models/nonperiodic/no-periodicity.stan" "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl"

If that works, then I can quickly put together a fix.

Thanks, Brandon

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dieterich-lab/rp-bp/issues/36#issuecomment-276759765, or mute the thread https://github.com/notifications/unsubscribe-auth/AHXfyp59WFyqCKvwrjcYiTYiwg16uyEdks5rYOAIgaJpZM4L0F9r.

[bmmalone]

Great! I have pushed a new version which includes a small fix to handle the space. It is working as expected locally. Could you please give it a shot and let me know how it goes? Thanks for your persistence.

[tmargus]

it still not clean. I compiled these .stan models to .pkl manually and now program finds them. However, estimate-metagene-profile-bayes-factors is running to errors.

FileNotFoundError: [Errno 2] No such file or directory: '/Users/tmargus/Library/Application' Is it caused how pkl's are compiled?

Error message:

INFO     root     2017-02-01 20:59:25,406 : estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models /Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl --nonperiodic-models /Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --log-file log_1.txt --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET
INFO     root     2017-02-01 20:59:25,406 : calling
INFO     riboutils.estimate_metagene_profile_bayes_factors 2017-02-01 20:59:26,079 : Estimating Bayes factors for lengths: 16,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34
  0%|                                                                                                                                                                                                                                                                                                                                            | 0/18 [00:00<?, ?it/s]multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/joblib/_parallel_backends.py", line 344, in __call__
    return self.func(*args, **kwargs)
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/joblib/parallel.py", line 131, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/joblib/parallel.py", line 131, in <listcomp>
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 72, in estimate_profile_bayes_factors
    periodic_models = [pickle.load(open(pm, 'rb')) for pm in args.periodic_models]
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 72, in <listcomp>
    periodic_models = [pickle.load(open(pm, 'rb')) for pm in args.periodic_models]
FileNotFoundError: [Errno 2] No such file or directory: '/Users/tmargus/Library/Application'

During handling of the above exception, another exception occurred:
...................

[bmmalone]

Hi Tonu,

Sorry, this is essentially the same problem, just in another place. The call to estimate-metagene-profile-bayes-factors also needs the spaces escaped or quoted.

If you'd like to run it by hand to get a feel for the output, you can use:

estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl" --nonperiodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl" --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --log-file log_1.txt --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET

The same problem will crop up later during a call to estimate-orf-bayes-factors, so the entire pipeline still won't work. I will (attempt to) track down all of the places this is occurring and try to have a patch in the next hour or so.

Have a good day, Brandon

[tmargus]

aa... ok! Now this is command is working! it creates file c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz I will be back tomorrow Tonu

On 1. veebr 2017, at 21:40, bmmalone notifications@github.com wrote:

estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl" --nonperiodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl" --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --log-file log_1.txt --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET

[bmmalone]

Hi Tonu,

Okay, good :) the space is definitely the problem, then.

At least for the models, it turns out there is a single function in a pre-requisite project, riboseq-utils, which is always used to do the scanning for pickle files I mentioned earlier. I updated the function in this commit: dieterich-lab/riboseq-utils@4d32fc4b193fc74659ea423e342ab2688510ca6d to always shell-escape the model filenames with quotes.

Locally, it appears to work; however, you will need updated versions of both riboseq-utils and misc. I'm not really sure of the easiest way to do that... maybe either uninstall them with pip3 uninstall riboutils misc, then re-run pip3 install . for rpbp, or directly clone the repos and install both of them with pip. Unfortunately, the integration with them is still fragile since we have not added things to pypi, yet.

Please take a look and let me know how it turns out. In particular, in the logs of the commands (like the one you posted above), the paths to the models should be quoted.

Have a good day, Brandon

[tmargus]

Hi Brandon,

I uninstalled misc and riboseq-utils wit pip, download new git repos fro misc and riboseq-utils and installed them with pip3 install . Then rerun install in rp-bp/ folder pip3 install -e . It should do the thing.

I did not change *.pkl models in ~/Library/Application\ Support/rpbp/rpbp_models/. running command process-all-samples c-elegans-test.yaml --overwrite --num-cpus 4 --logging-level INFO --log-file log_2.txt still ends with error but I don't see estimate-metagene-profile-bayes-factors in report. It still complains about pkl files Some input files ["'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl'", "'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl'"] are missing. but now the names are double qouted ["'pkl'"]. Could it be the reason of error?

Part of Error message.

INFO     root     2017-02-02 11:40:02,017 : extract-metagene-profiles /Users/tmargus/projects/rpbp/data/without-rrna-mapping/c-elegans-rep-1.test-unique.bam /Users/tmargus/projects/rpbp/data/WBcel235.79.chrI/WBcel235.79.chrI.bed.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz --num-cpus 4 --logging-level INFO --stdout-logging-level NOTSET --stderr-logging-level NOTSET --log-file log_2.txt --file-logging-level NOTSET  --start-upstream 50 --start-downstream 20 --end-upstream 50 --end-downstream 20
INFO     root     2017-02-02 11:40:02,017 : calling
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,495 : [extract-metagene-profiles]: /Users/tmargus/anaconda/bin/extract-metagene-profiles /Users/tmargus/projects/rpbp/data/without-rrna-mapping/c-elegans-rep-1.test-unique.bam /Users/tmargus/projects/rpbp/data/WBcel235.79.chrI/WBcel235.79.chrI.bed.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz --num-cpus 4 --logging-level INFO --stdout-logging-level NOTSET --stderr-logging-level NOTSET --log-file log_2.txt --file-logging-level NOTSET --start-upstream 50 --start-downstream 20 --end-upstream 50 --end-downstream 20
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10841/10841 [00:00<00:00, 337473.09it/s]
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,627 : Reading annotations
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,655 : Constructing canonical translation initiation ORF data frames
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,683 : Constructing canonical translation termination ORF data frames
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,691 : Finding translation initiation site matches
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,736 : Finding translation termination site matches
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,763 : Extracting metagene profiles
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,764 : Profiles will be created for lengths: 12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,40,41,42,46,48
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,824 : Writing metagene profiles to disk
INFO     root     2017-02-02 11:40:02,918 : gunzip -t /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz
INFO     root     2017-02-02 11:40:02,918 : calling
WARNING  misc.utils 2017-02-02 11:40:02,925 : [utils.call_if_not_exists]: This function is deprecated. Please use the version in misc.shell_utils instead.
WARNING  misc.utils 2017-02-02 11:40:02,925 : Some input files ["'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl'", "'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl'"] are missing. Skipping call: 
estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models '/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl' --nonperiodic-models '/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl' --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --stdout-logging-level NOTSET --stderr-logging-level NOTSET --log-file log_2.txt --file-logging-level NOTSET

Cheers, Tonu

[bmmalone]

Hi Tonu,

Sorry, I forgot to push the updates to "misc" last night. Could you please re-pull and install it. Sorry about that.

Also, you are right that the reason it can't find the file is because it is looking for a filename which begins with a literal quote; the update to "misc" fixes that behavior.

Have a good day, Brandon

[tmargus]

It worked now! Great! Thanks a lot. Tonu

[bmmalone]

Hi Tonu,

Great! Thanks so much for iterating on this. I'm going to close this issue now, but please don't hesitate to re-open it if necessary, or file any others if anything else comes up. We consider it very important that our software is both usable and based on solid theoretical principles.

That said, the fix for this issue will not help for other paths with spaces. However, all of the other paths are easily controlled, so hopefully that is not much of an issue. Still, I have preemptively opened a new issue (#41) that at least documents this behavior.

Thanks, again.

Have a good day, Brandon