search

galaxyproject / tools-iuc

Tool Shed repositories maintained by the Intergalactic Utilities Commission
https://galaxyproject.org/iuc
MIT License
161 stars 418 forks source link

IUC Contribution Fest - RADSeq Tools and Workflows #422

Closed tiagoantao closed 7 years ago

tiagoantao commented 8 years ago

We are planning a Remote Contribution Fest on 7th and 8th of March for developers to work on Galaxy RADSeq tools.

RADSeq is a cheap sequencing technology that is used by many resource-limited groups who would benefit a lot from easy-to-use galaxy tools. Indeed there has been quite some interest in analyzing RADSeq with Galaxy. Currently there is a wrapper for stacks and little more to help with RAD specific analysis (though many other galaxy tools are useful with RADs - bwa, cap3, gatk, velvet, ...).

If you are interested in participating in the hackathon but not interested in actual tool development - we will assemble a list of smaller, manageable Python and JavaScript tasks to work on and certainly documentation is a chronically lacking for collections so we could use help there and no actual coding would be required.

We encourage ideas or advice about how to organize this so please let us know. A core group will be available on IRC all day and we will have google hangouts across those days to organize, answer questions, and report progress.

We will do our best to coordinate and make this hackathon a nice and productive experience and we would like to especially focus on working reasonable hours and discourage overnighters.

All forms of contribution are welcome!

Google hangout link for the hackathon: https://hangouts.google.com/call/iiksaostt5dwfms4lph6nuj74ye

Original idea from @bgruening (galaxyproject/tools-devteam#26 (comment)). See #299 and #239 for information on the last two hackathons. This text is inspired on #299.

Things to work on

This list will be updated as comments are made on the issue. Your are strongly encouraged to leave your suggestion/opinion/criticism below.

Note that while the list is currently stacks-oriented, there is no reason for that to be the final list of stuff to do... Non-stacks based suggestions are strongly encouraged

bgruening commented 8 years ago

Just wanted to say that this is awesome! I'm in and try to help wherever I can!

abretaud commented 8 years ago

Writing wrappers for PyRAD or dDocent would be cool too

bgruening commented 8 years ago

@abretaud I added them to the list, thanks!

tiagoantao commented 8 years ago

@bgruening @abretaud I have added a link to PyRad documentation. I will probably send an email to the author to see if he wants to participate. He is a Python programmer, so he might be interested...

tiagoantao commented 8 years ago

@abretaud @bgruening @yvanlebras It is now new year and the metagenomics fest is long gone. I was thinking in resurrecting this in the next few days (start preparing, finding potential volunteers, ...)

abretaud commented 8 years ago

Hi (and happy new year!) It looks like a good idea! It's not really clear for me yet what we will be needed for the stacks web interface (i.e. potential IE). I don't have much time these days to look into it, but it would good to think about it before the hackathon

tiagoantao commented 8 years ago

@abretaud There is a stacks package as you probably know http://toolshed.genouest.org/repos/cmonjeau/stacks . This was developed by @yvanlebras and @cmonjeau .

I am making a few changes myself (which I can make public), namely:

I had a local request for supporting clone_filter, which I can work on.

With regards to this plugin, @yvanlebras, you would consider accepting patches? And maybe put it under tools-uic?

yvanlebras commented 8 years ago

Hi @tiagoantao,

For sure we will accept patches and put it under tools-iuc ! We will exchange locally in Rennes with @abretaud and @cmonjeau to prepare our participation to this contribution fest.

tiagoantao commented 8 years ago

@yvanlebras Any possibility you would put the code on github (even if not under tools-iuc)? I could submit a PR against it.

yvanlebras commented 8 years ago

@tiagoantao of course! I have to check with @cmonjeau, because I see only these 1.18 stacks tools even if we have a more recent 1.32 one..... I will keep you informed!

yvanlebras commented 8 years ago

.....@tiagoantao I forgot the hyperlink: https://github.com/genouest/galaxy-tools/tree/master/tools/stacks

tiagoantao commented 8 years ago

@yvanlebras In my version, I have upgraded all the tools. I will start submitting PRs today.

yvanlebras commented 8 years ago

@tiagoantao Thanks! OK, I will have a look.

yvanlebras commented 8 years ago

With @lecorguille, Maria Bernard and @abretaud, we begin to create things like remote hackathon dedicated Trello and HUBzero space to prepare the event.

tiagoantao commented 8 years ago

I had a request for rapture support: http://genetics.org/content/early/2015/12/22/genetics.115.183665

tiagoantao commented 8 years ago

I am going to drop an email to the authors of PyRad and dDocent about our hackthon

yvanlebras commented 8 years ago

Hi Tiago,

Good idea!

I cc Julian concerning Stacks !

Here in France, we begin to prepare the event ! A preliminary description is written on our GUGGO Galaxy working group dedicated space ( https://cesgo.genouest.org/groups/guggo/wiki/RADseqHackathon ).

Don't hesitate to come back to us if we can help !

Cheers,

Yvan

----- Mail original -----

De: "Tiago Antao" notifications@github.com À: "galaxyproject/tools-iuc" tools-iuc@noreply.github.com Cc: "yvanlebras" yvan.le_bras@irisa.fr Envoyé: Vendredi 5 Février 2016 01:26:26 Objet: Re: [tools-iuc] IUC Contribution Fest - RADSeq Tools and Workflows (#422)

I am going to drop an email to the authors of PyRad and dDocent about our hackthon

— Reply to this email directly or view it on GitHub .

Yvan Le Bras, PhD @Yvan2935 <°))))>< e-Biogenouest project http://www.e-biogenouest.org CNRS UMR 6074 IRISA-INRIA, Campus de Beaulieu, 35042 Rennes Cedex tél.: +33 (0) 2 99 84 71 79 / +33 (0) 6.10.43.96.51 yvan.le_bras@irisa.fr

jpuritz commented 8 years ago

Hi everyone,

I'm happy to help with implementing dDocent and filtering scripts!

Thanks for including dDocent.

abretaud commented 8 years ago

@bgruening I have seen some PR about a migration to conda to define tool dependencies: for this hackathon do you think we should use this new method or is it better to stick to the good-old tool_dependencies.xm method? In the first case, is there some guide/documentation/howto that we could use?

bgruening commented 8 years ago

@abretaud yes we should. I will help as much as I can. This will save us so much time and provides us with free testing.

abretaud commented 8 years ago

All right, I'm preparing a few slides to introduce this on monday

bgruening commented 8 years ago

Uh awesome - let me know if you have any questions and if I should prep some packages for monday and get it into bioconda.

abretaud commented 8 years ago

As there is a long todo list for stacks, maybe having it in bioconda before the hackathon would help, if you have some time to do it?

tiagoantao commented 8 years ago

I can do the stacks on bioconda

abretaud commented 8 years ago

Great, thanks!

tiagoantao commented 8 years ago

I have added this to bioconda as a PR, while I can merge it, I would prefer @bgruening to have a look.

here: https://github.com/bioconda/bioconda-recipes/pull/925

yvanlebras commented 8 years ago

@cmonjeau have made PR for pyrad and mawk bioconda recipes. If you can also have a look @bgruening ;) mawk bioconda recipes pyRAD bioconda recipes

bgruening commented 8 years ago

You all rock! This will be a fantastic hackathon with conda dependencies and travis testing all at once! I commented on the PR!

Eduardo-Alves commented 8 years ago

Hi, has anyone started looking into "the feasibility of implementing the stacks web interface on Galaxy". I started looking into visualization plugins using mako but I believe there must be a way to do it with JSON/Backbone. Anyone has some pointers? I am happy to help!

abretaud commented 8 years ago

I was thinking about working on an interactive environment for stacks. IIRC the web interface allow to modify the data, so using IE looks like the perfect solution for this (ie load the stacks preliminary results in the IE, make changes to it if you want, and then inject the corrected data to your history)

I'll see next week if I have time to work on this during the hackathon, but your help is very welcome anyhow!

Eduardo-Alves commented 8 years ago

Anthony,

The only update I can see in the web interface is to add an annotation, which is probably saved to the mysql database. I haven't developed IE, do you mean creating an apache IE and using the existing web interface? Can you add a mySQL database to an IE?

bgruening commented 8 years ago

@Eduardo-Alves you can add anything to an IE. For example we have a RStudio IE or a Jupyter/IPython IE.

yvanlebras commented 8 years ago

Our @abretaud original idea ;) was to use the existing Stacks Galaxy tools, who don't use the STACKS database and thus don't propose the Stacks web interface based on the database (if I'm not wrong...), to produce results and then, "reconstruct" a database who will (maybe) give the possibility to create the Stacks web interface......

One idea is to use the quite recently developped load_radtags.pl Stacks script to take Galaxy tools results (for exemple an archive from denovo_map) and create the database on a IE where we can interact with this db from the classical Stacks web interface....

For this, it seems that we need a "Stacks IE" (with database & web interface) + a way to run the load_radtags.pl from a Galaxy Stacks history archive dataset to load these data in the web interface + a way to output IE database interaction results on the Galaxy Stacks history....

Not sure I'm clear, but don't hesitate to ask questions.... Maybe this is not possible, or not really interesting..... But this way seems to be the best to facilitate manual correction on the genotypes for example....

Eduardo-Alves commented 8 years ago

Anthony,

Wouldn't implementing it as an IE restrict it to instances where the admin is capable of setting it up? For example, I only have galaxy admin access to the university galaxy instance but no command line access to the installation so I am not able to install an IE. I think a visualization plugin might be harder to implement but visualizations will be installable from toolshed in version 16.04 https://github.com/galaxyproject/galaxy/issues/1191

Eduardo-Alves commented 8 years ago

@yvanlebras do you have any screenshots examples of the manual correction of the genotypes in the web interface? all I can see is the annotate link http://gbs-cloud-tutorial.readthedocs.org/en/latest/_images/stacks_web_database.png

yvanlebras commented 8 years ago

@Eduardo-Alves there is things like that in the visualization part (at the bottom)

Eduardo-Alves commented 8 years ago

@yvanlebras yes, that's the screenshot I sent you. It is just displaying the results, not modifying. The only think you can add is an annotation.
My suggestion is to use the existing stacks tool to generate the raw data, then load it to a sqllite database which is single user and doesn't require setup like mysql and then generate the web interface using mako template. In this way it can be published as a plugin and will be installable from toolshed. https://wiki.galaxyproject.org/VisualizationsRegistry

bgruening commented 8 years ago

I just opened a google hangout https://hangouts.google.com/call/iiksaostt5dwfms4lph6nuj74ye that we can use on Monday. I will be around from 10.00 MEZ and help wherever I can.

yvanlebras commented 8 years ago

Thanks Björn !  We will begin at 10h30 Rennes time. I will try to go back to you before to give you more informations on what we plan to do. For now, it appears positions like : -Maria Bernard and @lecorguille Gildas Le Corguillé will work on the implementation of some in-house Stacks post-process scripts

-------- Message d'origine -------- De : Björn Grüning notifications@github.com Date : 05/03/2016 12:07 (GMT+01:00) À : galaxyproject/tools-iuc tools-iuc@noreply.github.com Cc : yvanlebras yvan.le_bras@irisa.fr Objet : Re: [tools-iuc] IUC Contribution Fest - RADSeq Tools and Workflows (#422)

I just opened a google hangout https://hangouts.google.com/call/iiksaostt5dwfms4lph6nuj74ye that we can use on Monday. I will be around from 10.00 MEZ and help wherever I can.

Reply to this email directly or view it on GitHub: https://github.com/galaxyproject/tools-iuc/issues/422#issuecomment-192623057

bgruening commented 8 years ago

Hi!

unfortunately, I need to attend a meeting tomorrow and only will be available from 15.00 MEZ :)

I created a few branches and we have for most of the tools already conda packages that can be reused. This hopefully means we do not need to spend time to create tool_dependencies or install a particular version - and gives us more time to improve the wrappers.

pyrad: https://github.com/galaxyproject/tools-iuc/pull/646 ddocent: https://github.com/galaxyproject/tools-iuc/pull/647 stacks: https://github.com/galaxyproject/tools-iuc/pull/648 mixed radseq: https://github.com/galaxyproject/tools-iuc/pull/649

Would be great if we can use these branches tomorrow. The IUC will merge as often as possible into these branches so we can synchronise our work in these branches and distribute the work. Does this sounds like a plan?

We can coordinate our work via a google hangout:

https://hangouts.google.com/call/iiksaostt5dwfms4lph6nuj74ye or IRC #galaxyproject on freenode

With a little bit luck we will also have some contributes that want to focus on usability aspect of our tools, including documentation. It would be nice if we can introduce them to use the github webinterface to contribute help text or similar things like an example workflow. For example this could be used to create a nice help text for PyRad: http://rst.ninjs.org

Thanks and sorry for joining so late, Bjoern

tiagoantao commented 8 years ago

I am currently in Europe (UK) and will be joining around 10 WET (that is 11 in France/Germany).

I will be working on mostly two things: i) cap3 integration with stacks and ii)rapture. I do have a few remaining patches for the stacks tool (though mostly were already integrated) - but it will not be my main focus.

yvanlebras commented 8 years ago

Thanks @tiagoantao for these details ! We thus maybe can propose to find someone today to update Stacks to the 1.37 version, support for data collection and support Galaxy_SLOTS.

yvanlebras commented 8 years ago

Hi everyone, as we will make some introductions (planemo/bioconda, Diocker, RADseq) this morning for french people, you can access the presentations slides on the GUGGO dedicated pages

Eduardo-Alves commented 8 years ago

Bon jour! Will the introductions going to be also on hangouts?

tiagoantao commented 8 years ago

@yvanlebras : 1.37 and GALAXY_SLOTS were already included, in the github version. they are ready!

tiagoantao commented 8 years ago

There was nobody on the hangouts. when somebody logs in, just write here and I will be joining also. I am working on cap3 for now.

mvdbeek commented 8 years ago

@tiagoantao there is already a cap3 wrapper in the toolshed: https://github.com/ARTbio/tools-artbio/tree/master/tools/msp_cap3 we can include it in the iuc, if that's preferable.

Not sure what would be needed to make the integration with stacks though.

tiagoantao commented 8 years ago

@mvdbeek I know but it does not work well with stacks

mvdbeek commented 8 years ago

what does not work well? can it be improved ? just trying to avoid duplicate efforts.

tiagoantao commented 8 years ago

There is a script called exec_velvet.pl in stacks that uses velvet. We would like it to use cap3. There is probably no need to make a new tool (evaluating that), just integrating them.

On 7 March 2016 at 03:21, Marius van den Beek notifications@github.com wrote:

what does not work well? can it be improved ? just trying to avoid duplicate efforts.

— Reply to this email directly or view it on GitHub https://github.com/galaxyproject/tools-iuc/issues/422#issuecomment-193194650 .

"While I may be sending this email outside my normal office hours, I have no expectation to receive a reply outside yours" - @tomstafford