Beginner manual

[dakl]

Hi,

Sorry is this is an ignorant comment: Is there a beginners manual describing how to get this running as a pipeline on UPPMAX? I can't find any info on how a regular user can actually use this.

cheerio Daniel

[brainstorm]

Hello dakl,

Did you have a look at the README there on?:

https://github.com/SciLifeLab/bcbb/tree/master/nextgen

If you did, we're actively working on getting the pipeline deployable in a much simpler fashion, @guillermo-carrasco is working full time on this at the moment.

Other than the official documentation, you can have a look at:

https://github.com/guillermo-carrasco/bcbb_chapmanb/blob/master/.travis.yml

Were you have a simplified recipe on deploying it in a regular machine if you intend to use it outside HPC systems.

Hope that helps!

[dakl]

HI Roman,

I did read through that README and it did help a little bit. It is however very similar to the one in the original bcbb/nextgen repo and understanding what I have to change to run it on UPPMAX was not so easy.

I guess I'm after a few things specifically:

• Tutorial on how to get the pipeline running with example fastq files, including where the data ends up (locally, galaxy etc)
• Tutorial on moditying the pipeline, adding modules etc

Very much looking forward to the easy-deploy version of this. Any ETA on that? Should I expect a month or 6?

Thanks again for a great job!

//D

[vals]

Hi @dakl, I think you are getting at the point that it's not particularly easy to understand exactly how to run it once it is installed.

Some things that might clarify things a bit

• Once nextgen has been installed with python setup.py install, you can run automated_initial_analysis.py (without arguements), and it will print some usage information.
• There are som example configuration files in the config directory: https://github.com/SciLifeLab/bcbb/tree/master/nextgen/config , you can look at these to get a feeling of what configuration and software needed.
• In the directory tests there is a file test_automated_analysis.py, this contains a few analysis examples and it might give some idea of how to different analysises.

To actually run the tests, you need to configure a testing post_process.yaml file and either put it in nextgen/tests/data/automated or link to it from that directory. You could base this test config on nextgen/tests/data/automated/post_process-sample.yaml: https://github.com/SciLifeLab/bcbb/blob/master/nextgen/tests/data/automated/post_process-sample.yaml.

[guillermo-carrasco]

Hi @dakl ,

As Roman said, I'm working now on a recipe to automatically install and configure the pipeline on UPPMAX. With this recipe you'll be able to get a running and testable pipeline in UPPMAX in a much more simpler way (in fact, the idea is to just run the recipe to get everything ready).

About the ETA, I guess that it will be ready in about two-three weeks as much. Hope this helps you :-)

[dakl]

Sweet, looking forward to that. Keep up the great work :)

Daniel Klevebring

On Thursday 6 September 2012 at 14:11, Guillermo Carrasco wrote:

Hi @dakl (https://github.com/dakl) , As Roman said, I'm working now on a recipe to automatically install and configure the pipeline on UPPMAX. With this recipe you'll be able to get a running and testable pipeline in UPPMAX in a much more simpler way (in fact, the idea is to just run the recipe to get everything ready). About the ETA, I guess that it will be ready in about two-three weeks as much. Hope this helps you :-)

— Reply to this email directly or view it on GitHub (https://github.com/SciLifeLab/bcbb/issues/202#issuecomment-8330191).

[guillermo-carrasco]

Hi @dakl ,

I've finished the first version of the recipe to automate the installation of the pipeline in UPPMAX. You can find the recipe here, and its use its quite straightforward. You just need to:

1. Clone the repository into your UPPMAX account (in kalkyl):

   git clone https://github.com/guillermo-carrasco/bcbio-nextgen-deploy.git

2. Change dir to that repositoy:

   cd bcbio-nextgen-deploy

3. If you want (preferably), clean any previous installation of the pipeline:

   python setup_uppmax.py purge

WARNING: This will completely remove any previous installation of the pipeline, including configuration files, logs, etc.

1. Let the script install and configure the pipeline, and run the testsuite:

   python setup_uppmax.py install

When the script finishes installing the pipeline it will run the testsuite with a reduced dataset. It shouldn't take more than 10 minutes to finish, and will enter directly in "running" because the job is called with the "-p devel" option. You can see the results in ~/opt/bcbb/nextgen/tests/nosetests.xml and the output in ~/opt/bcbb/nextgen/tests/tests_results.out.

Please don't hesitate to tell me (comment here or directly by email) if you have any problem. Any feedback is really appreciated!

Thanks and hope that helps!

[dakl]

Nice work!

However :)

$ python setup_uppmax.py install 2012-09-21 16:18:38,293 - INFO - SETTING UP VIRTUALENVWRAPPER 2012-09-21 16:18:38,293 - INFO - Editing .bashrc... 2012-09-21 16:18:38,296 - INFO - Installing virtualenvwrapper and creating a virtual environment "master" for the production pipeline... Traceback (most recent call last): File "setup_uppmax.py", line 216, in function(env, config_lines) File "setup_uppmax.py", line 47, in install os.makedirs(pjoin(home, 'opt/mypython/lib/python2.6/site-packages')) File "/usr/lib64/python2.6/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 17] File exists: '/bubo/home/h14/kleve/opt/mypython/lib/python2.6/site-packages'

Any idea why?

cheers D

On Thu, Sep 20, 2012 at 4:17 PM, Guillermo Carrasco < notifications@github.com> wrote:

Hi @dakl https://github.com/dakl ,

I've finished the first version of the recipe to automate the installation of the pipeline in UPPMAX. You can find the recipe herehttps://github.com/guillermo-carrasco/bcbio-nextgen-deploy, and its use its quite straightforward. You just need to:

1.

Clone the repository into your UPPMAX account (in kalkyl):

git clone https://github.com/guillermo-carrasco/bcbio-nextgen-deploy.git

2.

Change dir to that repositoy:

cd bcbio-nextgen-deploy

3.

If you want (preferably), clean any previous installation of the pipeline:

python setup_uppmax.py purge

WARNING: This will completely remove any previous installation of the pipeline, including configuration files, logs, etc.

1.

Let the script install and configure the pipeline, and run the testsuite:

python setup_uppmax.py install

When the script finishes installing the pipeline it will run the testsuite with a reduced dataset. It shouldn't take more than 10 minutes to finish, and will enter directly in "running" because the job is called with the "-p devel" option. You can see the results in ~/opt/bcbb/nextgen/tests/nosetests.xml and the output in ~/opt/bcbb/nextgen/tests/tests_results.out.

Please don't hesitate to tell me (comment here or directly by email) if you have any problem. Any feedback is really appreciated!

Thanks and hope that helps!

— Reply to this email directly or view it on GitHubhttps://github.com/SciLifeLab/bcbb/issues/202#issuecomment-8729431.

[guillermo-carrasco]

Hy,

Yes, it means that probably you didn't run the purge function, and it's definitely my fault for not checking that, I'm sorry! I've made the necessary checking. Can you please pull the changes and tell me if it works now? Execute the pull inside the bcbio-nextgen-deploy folder:

    git pull origin master

And repeat the procedure. Maybe, to perform a cleaner installation would be better to execute first the purge function.

    python setup_uppmax.py purge

Anyway thank you very much for your feedback, and please tell me if you continue with the problem or have others.

Thank you!

[dakl]

Hi,

I actually did purge the first time, but that threw an error (perhaps I should have included that :)

Anyway, after the pull, it starting doing stuff but then:

virtualenvwrapper.user_scripts creating /bubo/home/h14/kleve/.virtualenvs/master/bin/predeactivate virtualenvwrapper.user_scripts creating /bubo/home/h14/kleve/.virtualenvs/master/bin/postdeactivate virtualenvwrapper.user_scripts creating /bubo/home/h14/kleve/.virtualenvs/master/bin/preactivate virtualenvwrapper.user_scripts creating /bubo/home/h14/kleve/.virtualenvs/master/bin/postactivate virtualenvwrapper.user_scripts creating /bubo/home/h14/kleve/.virtualenvs/master/bin/get_env_details /bubo/home/h14/kleve/.virtualenvs/postactivate: line 8: /bubo/home/h14/kleve/opt/config/modules: No such file or directory Traceback (most recent call last): File "setup_uppmax.py", line 218, in function(env, config_lines) File "setup_uppmax.py", line 50, in install check_call(install_and_create_virtualenv, shell=True, env=env) File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command ' easy_install --prefix=~/opt/mypython pip && pip install virtualenvwrapper --install-option="--prefix=~/opt/mypython" && . ~/.bashrc && module unload python && mkvirtualenv --python=/sw/comp/python/2.7_kalkyl/bin/python master ' returned non-zero exit status 1

BR Daniel

On Fri, Sep 21, 2012 at 4:44 PM, Guillermo Carrasco < notifications@github.com> wrote:

Hy,

Yes, it means that probably you didn't run the purge function, and it's definitely my fault for not checking that, I'm sorry! I've made the necessary checking. Can you please pull the changes and tell me if it works now? Execute the pull inside the bcbio-nextgen-deploy folder:

git pull origin master

And repeat the procedure. Maybe, to perform a cleaner installation would be better to execute first the purge function.

python setup_uppmax.py purge

Anyway thank you very much for your feedback, and please tell me if you continue with the problem or have others.

Thank you!

— Reply to this email directly or view it on GitHubhttps://github.com/SciLifeLab/bcbb/issues/202#issuecomment-8766866.

[guillermo-carrasco]

It looks like I have a problem in the purge function which I haven't realized. I have to fix it, but you can get a quick fix if you want to install the pipeline now, execute:

    echo "" > ~/.virtualenvs/postactivate

And re-run

    python setup_uppmax.py install

Thank you again and sorry, I'll let you know when I fix the bug.

[dakl]

That did it, but:

2012-09-24 08:34:53,213 - INFO - Cloning config repository... Initialized empty Git repository in /bubo/home/h14/kleve/opt/config/.git/ The authenticity of host 'code.scilifelab.se (130.237.80.12)' can't be established. RSA key fingerprint is 24:3f:ce:e6:84:69:05:f7:9e:9c:00:78:2c:96:67:1a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'code.scilifelab.se,130.237.80.12' (RSA) to the list of known hosts. git@code.scilifelab.se's password:

:)

//D

On Fri, Sep 21, 2012 at 5:18 PM, Guillermo Carrasco < notifications@github.com> wrote:

It looks like I have a problem in the purge function which I haven't realized. I have to fix it, but you can get a quick fix if you want to install the pipeline now, execute:

echo "" > ~/.virtualenvs/postactivate

And re-run

python setup_uppmax.py install

Thank you again and sorry, I'll let you know when I fix the bug.

— Reply to this email directly or view it on GitHubhttps://github.com/SciLifeLab/bcbb/issues/202#issuecomment-8767987.

[guillermo-carrasco]

Hi @dakl ,

Sorry about that, now I've fixed this and also the problem in the purge function. I need you to do a few steps:

• Pull the changes I've made in the recipe (within bcbio-nextgen-deployment directroy):

  git pull origin master

• Clean the previous broken installation:

  echo "" > ~/.virtualenvs/postactivate && python setup_uppmax.py purge

• And install the pipeline:

  python setup_uppmax.py install

Sorry for the inconveniences, please keep me informed and tell me if you have any other problems.

Thank you very much!