search

kbaseattic / assembly

An extensible framework for genome assembly.
MIT License
12 stars 14 forks source link

Test the robustness #49

Closed sebhtml closed 10 years ago

sebhtml commented 10 years ago

test on sal using the client from the master branch

sebhtml commented 10 years ago

sal is down. also, needs API spec to test it.

levinas commented 10 years ago

@sebhtml Could you help test the client API using some recent real datasets, Ideally with a small genome such as Buchnera aphidicola and a medium-sized genome such as E. coli K12 or Mycobacterium tuberculosis? Perhaps one run with reference and the other without. It would be good to gather some output for us to put into the tutorial.

The test control server is 140.221.84.203 and the queue is wasp. You only need the -q queue parameter for the run command. The latest test script is test/june_2014_test.t, but it's incomplete.

@cbun, how many threads do we have running on the edge control server? Could you add a recipe called 'fast' or that does just "tagdust -> (kmergenie + velvet) or spades" without merging? Something like that. I'll get a hackish N50-based get implemented.

Here are the basic commands. I will be updating my narrative text in the comment section of #72, and eventually committing to a markdown file in the repo, which will be converted to the HTML tutorial and the narrative. 

# auto recipe
ar-run --pair s_1.fastq s_2.fastq --reference contigs.fa
# explicit pipelines examples; Chris can suggest better parameters
ar-run --data data_id --p 'none tag_dust' 'velvet spades ray idba' 
...
# status
ar-stat -j job_id
ar-stat --list-data
# get report 
ar-get --wait -j job_id
ar-get --report -j job_id
# capture a picture from quast html 

# to be implemented today
ar-get --pick assembly_id

By the way, you can deploy a client this way. 

make -f Makefile.standalone deploy

It will not support the kbase workspace function.

sebhtml commented 10 years ago

Datasets:

In /mnt/stuff

SRS213780 - Escherichia coli str. K-12 substr. W3110 http://www.ebi.ac.uk/ena/data/view/SRS213780

SRS365561 - Mycoplasma feriruminatoris http://www.ebi.ac.uk/ena/data/view/SRS365561

Machine:

140.221.84.146 ( Mockingbird.boisvert.info ) https://cloud.mcs.anl.gov/nova/instances_and_volumes/instances/f3db91da-b310-4374-b38b-2c26dc4a9a37/detail

sebhtml commented 10 years ago

http://www.centosblog.com/enable-epel-repo-on-centos-5-and-centos-6/

sebhtml commented 10 years ago

Connecting to ftp.sra.ebi.ac.uk|193.62.197.89|:21... failed: Connection refused.

sebhtml commented 10 years ago

will use DDBJ... (NCBI only has SRA files, and I dislike them). http://trace.ddbj.nig.ac.jp/DRASearch/

sebhtml commented 10 years ago

[boisvert@kbase-test-june-2014 stuff]$ ls -lh /|grep fastq -rw-rw-r--. 1 boisvert boisvert 173M Jun 26 13:16 SRS213780/SRR306102_1.fastq.bz2 -rw-rw-r--. 1 boisvert boisvert 185M Jun 26 13:22 SRS213780/SRR306102_2.fastq.bz2 -rw-rw-r--. 1 boisvert boisvert 1.6G Jun 26 13:40 SRS365561/SRR578718_1.fastq.bz2 -rw-rw-r--. 1 boisvert boisvert 1.7G Jun 26 13:43 SRS365561/SRR578718_2.fastq.bz2

sebhtml commented 10 years ago

It is not clear how I can deploy the client.

[boisvert@kbase-test-june-2014 assembly]$ find .|grep ar-run ./client/ar-run.pl [boisvert@kbase-test-june-2014 assembly]$ make Makefile:9: ../../tools/Makefile.common: No such file or directory Makefile:146: ../../tools/Makefile.common.rules: No such file or directory make: *\ No rule to make target `../../tools/Makefile.common.rules'. Stop.

sebhtml commented 10 years ago

can't login

[boisvert@kbase-test-june-2014 test]$ ar-login File "/mnt/stuff/boisvert/assembly/deployment/pybin/arast.py", line 263 arast_msg = {k:options[k] for k in ['pipeline', 'data_id', 'message', 'queue', 'version', 'recipe', 'wasp'] ^ SyntaxError: invalid syntax Died at /mnt/stuff/boisvert/assembly/deployment/plbin/ar-login.pl line 31.

sebhtml commented 10 years ago

http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/

sebhtml commented 10 years ago

It does not work with Python 2.7.6...

[boisvert@kbase-test-june-2014 assembly]$ ar-login File "/mnt/stuff/boisvert/assembly/deployment/pybin/arast.py", line 263 arast_msg = {k:options[k] for k in ['pipeline', 'data_id', 'message', 'queue', 'version', 'recipe', 'wasp'] ^ SyntaxError: invalid syntax Died at /mnt/stuff/boisvert/assembly/deployment/plbin/ar-login.pl line 31. [boisvert@kbase-test-june-2014 assembly]$ head -n1 /mnt/stuff/boisvert/assembly/deployment/pybin/arast.py

! /usr/bin/env python

[boisvert@kbase-test-june-2014 assembly]$ /usr/bin/env python --version Python 2.7.6

sebhtml commented 10 years ago

" is CentOS 6.4. and python 2.7.3 works there too"

sebhtml commented 10 years ago

can't find Text::Table in CentOS 6.4.

[root@kbase-test-june-2014 Python-3.4.1]# yum install -y "perl(JSON)" Loaded plugins: fastestmirror, security Loading mirror speeds from cached hostfile

[root@kbase-test-june-2014 Python-3.4.1]# yum install -y "perl(Text::Table)" Loaded plugins: fastestmirror, security Loading mirror speeds from cached hostfile

sebhtml commented 10 years ago

install a perl module: http://perl.about.com/od/packagesmodules/qt/perlcpan.htm

sebhtml commented 10 years ago

[boisvert@kbase-test-june-2014 test]$ ./integration_test_1.sh Please authenticate with KBase credentials

from line 199 in: [boisvert@kbase-test-june-2014 test]$ vim /mnt/stuff/boisvert/assembly/deployment/plbin/ar-upload.pl

sebhtml commented 10 years ago

arast upload and ar-upload are different things.

[boisvert@kbase-test-june-2014 test]$ ./integration_test_1.sh Traceback (most recent call last): File "/mnt/stuff/boisvert/assembly/deployment/pybin/arast.py", line 398, in main() File "/mnt/stuff/boisvert/assembly/deployment/pybin/arast.py", line 188, in main aclient = client.Client(ARAST_URL, a_user, a_token) File "/mnt/stuff/boisvert/assembly/deployment/lib/assembly/client.py", line 24, in init self.url = url + ':{}'.format(self.port) ValueError: zero length field name in format

sebhtml commented 10 years ago

Tasks generated:

sebhtml commented 10 years ago

Fangfang's patch: https://github.com/levinas/assembly/commit/b5d0b406e8658d737cafe28732c74e33ee73f726.patch

sebhtml commented 10 years ago

The 6 tasks were completed.

sebhtml commented 10 years ago

This test works.

[boisvert@kbase-test-june-2014 test]$ ./integration_test_1.sh 
arast upload -m SRS213780 --pair SRR306102_1.fastq.bz2 SRR306102_2.fastq.bz2 ARAST_URL=
Upload complete: SRR306102_1.fastq.bz2
Upload complete: SRR306102_2.fastq.bz2
Data ID: 5
sebhtml commented 10 years ago

I am closing this ticket as it is not specific enough.