search

galaxyproject / tools-iuc

Tool Shed repositories maintained by the Intergalactic Utilities Commission
https://galaxyproject.org/iuc
MIT License
160 stars 417 forks source link

IUC Contribution Fest - Metagenomics Tools and Workflows #299

Closed jmchilton closed 8 years ago

jmchilton commented 8 years ago

Remote Contribution Fest happens on 30 Nov and 1 Dec

We are planning a Remote Contribution Fest on 30th of November and 1st of December for developers to work mainly on Galaxy metagenomic tools.

The remote nature of this gives people who don't have the opportunity to come to GCC hackathons (which have always been productive and a lot of fun) a chance to participate in a Galaxy hackathon. Having a well defined topic will allow us to accomplish a lot and let people who don't have particular tasks in mind find something to work on very quickly.

We are collecting ideas to work on, but we expect to attract the most participation by simply getting tool developers interested in getting help adding collection support to their existing tools and workflows to show up and participate.

If you are interested in participating in the hackathon but not interested in actual tool development - we will assemble a list of smaller, manageable Python and JavaScript tasks to work on and certainly documentation is a chronically lacking for collections so we could use help there and no actual coding would be required.

We encourage ideas or advice about how to organize this so please let us know. A core group will be available on IRC all day and we will have 4 google hangouts across those days to organize, answer questions, and report progress.

We will do our best to coordinate and make this hackathon a nice and productive experience and we would like to especially focus on working reasonable hours and discourage overnighters.

All forms of contribution are welcome!

Original idea from @bgruening (https://github.com/galaxyproject/tools-devteam/issues/26#issuecomment-126957869). See #239 for information on the last hackathon.

Things to work on

https://www.google.com/maps/d/edit?mid=zQipez5uCQ8I.khoUIhze9Fek&usp

Google Hangout

https://hangouts.google.com/call/iems2ffkpd4s7cyjoq4mi3hgwma

IRC

https://wiki.galaxyproject.org/News/IRCPublicLogProposal

hexylena commented 8 years ago

:+1: to biom, I'd be happy to do a phinch viz plugin/tool if a biom datatype was available

jmchilton commented 8 years ago

@erasche - add to the TODO list and sign yourself up :smile:

bgruening commented 8 years ago

It would be lovely if we could get @jennaj on-board and maybe can create a workflow that we stick into the Tool Shed and myexperiments.org.

jennaj commented 8 years ago

@bgruening Funny this should come up here - it was also brought up last Fri at a data hack meeting. Metagenomics is definitely on the table for the next cycle. Maybe we can combine forces :) I'll email the data hacks with a link to this.

bgruening commented 8 years ago

Thanks @jennaj!

nekrut commented 8 years ago

Perhaps we need a bit of a big picture. Improving individual tools is necessary, but we should focus on entire types of MG workflows (e.g., what are potential complete workflows for doing human microbiome studies). This is best accomplished using actual biological projects. Can you guys make a list of analyses you are currently supporting. I'll describe mine

lparsons commented 8 years ago

@nekrut Great point and a very good first step. I'm looking to analyze 16S data to generate OTU percentages per sample and PCA plot for diversity analysis. Following a workflow similar to that outlined in http://nbviewer.ipython.org/github/biocore/qiime/blob/1.9.1/examples/ipynb/illumina_overview_tutorial.ipynb

robldavidson commented 8 years ago

I like the idea of linking up the GalaxyScientists (@JennaJ's data hackers) with the IUC for this - it would be good to have a project to foster some dialogue between these two groups.

tiagoantao commented 8 years ago

I am not a metagenomics person, but I have seen people struggling with the mothur tool (if you install it, you will see what I mean). My impression is that a few workflows would help a lot. If you look at the mothur documentation, you could easily get inspiration. For example here: http://www.mothur.org/wiki/Analysis_examples

PS - You could consider doing a RAD-seq hackathon in the future. I would like to help on that one...

bebatut commented 8 years ago

I'm currently developing a Galaxy instance dedicated to gut microbiota data processing (metagenomics and metatranscriptomics). I made some wrappers for tools such as HUMAnN, MetaPhlAn, SortMeRNA, Reago, PRINSEQ, for which I did not find wrappers fitting my requirements. The aim is to construct a workflow to process data from intestinal microbiota. I would like to join you on this hackathon

bgruening commented 8 years ago

@bebatut this is awesome. If you like please join this hackathon and we could review your wrappers if you like. Btw.: SortMeRNA and PRINSEQ wrappers are already here

@tiagoantao excellent suggestion!

nsoranzo commented 8 years ago

@bebatut MetaPhlAn and SortMeRNA are also on the Tool Shed, please do not reinvent the wheel, if something is missing please contribute to the already available tools!

https://toolshed.g2.bx.psu.edu/view/rnateam/sortmerna/ https://toolshed.g2.bx.psu.edu/view/dannon/metaphlan/

@bgruening Should PRINSEQ also be published on the MTS or it's not ready yet? @dannon What about adding metaphlan wrapper code to the IUC github repo?

bgruening commented 8 years ago

@nsoranzo PRINSEQ It needs some more love but it was on a good way. Happy to work with @bebatut on this during the hackathon.

bebatut commented 8 years ago

@bgruening @nsoranzo I saw the wrappers for Prinseq and SortMeRNA, but they do not correspond to what I wanted. For example, not all the parameters are accessible with SortMeRNA wrapper. I developed a wrapper for MetaPhlAn 2 not for MetaPhlAn...

bgruening commented 8 years ago

@bebatut let's merge our wrappers during the hackthon and join (maintenance) forces!

tiagoantao commented 8 years ago

I think that there is a bug in the kraken data manager. Try the following admin>local data>Run data manager tools>Kraken

Select partial library to download> Human

Depending on my galaxy installation I am getting different errors. The most common is not being able to download human chromosomes. But works well with the other options

nekrut commented 8 years ago

https://test.galaxyproject.org/u/sjcc/w/16s

Here is the king of stuff we are currently doing:

Input > Quality Trim (FASTQ Quality Trimmer) > Join R1 + R2 (fastq-join) > remove reads with ambiguous bases (Manipulate FASTQ) > remove chimeras (cd-hit-dup) > assign reads to taxon (Kraken) > filter Kraken by confidence level (Kraken-filter) > translate taxon id number into taxon name (Kraken-translate) > report Kraken results (Kraken-report)

Filter Kraken report to see what data you want (Filter) <-- I use this to isolate a taxonomic level (i.e. P, C, O, F, G, S)

Rarefaction analysis (Vegan Rarefaction) Diversity (Vegan Diversity)

bgruening commented 8 years ago

@tiagoantao can you move this bug report to a proper issue, so it get not lost? @nekrut thanks very useful. We could work on materializing this into a TS workflow with documentation. Anything on you wishlist, preferably a "easy" tool without *.loc files?

jmchilton commented 8 years ago

I think we as Galaxy community should also consider a Brew related hackathon (https://github.com/Homebrew/homebrew-science/issues/2876) and I would like if we could schedule one or the other for November. Any particular thoughts on whether we should do this one in November and the Homebrew one in January or vice versa?

bgruening commented 8 years ago

November for the metagenomic Codefest and January for the next. Until then we can make the brew idea more concrete.

bgruening commented 8 years ago

What about the 5th and 6th of November? To early?

jmchilton commented 8 years ago

@bgruening I'll be at a conference on the 5th and traveling on the 6th. I don't know if Anton intends to participate directly with the hackathon but he will be at the same conference. Obviously I have much less to contribute to a metagenomics hackathon than a collections one though, so feel free to continue without me. I could do the 19th and 20th or the 9th and 10th.

jennaj commented 8 years ago

2nd Nov 9th and 10th. Some of us might or could be in PA that are not otherwise, which could be nice if the primary reason for being there does not conflict (occurs later in the week).

blankenberg commented 8 years ago

BIOM xref: https://github.com/galaxyproject/galaxy/issues/941

lparsons commented 8 years ago

I could do either of: 5th and 6th, 9th and 10th, or 19th and 20th. That said, the 9th and 10th are looking rather nice.

jmchilton commented 8 years ago

@bgruening Can you do the 9th and 10th?

bgruening commented 8 years ago

If so I will sitting next to you and will have a hard time to motivate peoples from Freiburg remotely. But yes, I'm ok with this.

jmchilton commented 8 years ago

Oh - yeah I suppose that might not be the best use of your limited time here. Forgot about that.

bgruening commented 8 years ago

30th of November and 1st of December?

jmchilton commented 8 years ago

This works for me.

lparsons commented 8 years ago

Nov 30th-Dec 1st works for me.

bgruening commented 8 years ago

It seems that we have a schedule :) If no one else speaks up I will write a mail with the announcement tomorrow.

martenson commented 8 years ago

rewrote the leading post to be more like #239

martenson commented 8 years ago

An inspiration how Jenkins community does virtual hackathons: http://www.meetup.com/jenkinsmeetup/events/226735213/

hexylena commented 8 years ago

Neat! I like the presentation aspect for newcomers.

On 11/22/2015 07:23 PM, Martin Cech wrote:

An inspiration how Jenkins community does virtual hackathons: http://www.meetup.com/jenkinsmeetup/events/226735213/

— Reply to this email directly or view it on GitHub https://github.com/galaxyproject/tools-iuc/issues/299#issuecomment-158825346.

shiltemann commented 8 years ago

+1 to Phinch! (that was next on my to-do list)

I have also been working on creating mothur pipelines lately, many of the tools have variable number of inputs and outputs, making them hard to use in workflows. I've been rewriting the tools I use in my pipeline to produce and consume collections instead, would be nice to do this for the entire toolsuite.

yvanlebras commented 8 years ago

Hi everyone,

Thanks for this very interesting proposal!

A litlle message to inform you that we will be ~10 people from France interested by this Hackathon. Thanks to western France GUGGO Galaxy group, INRIA, the French Institut of Bioinformatics and the Environmental Genomics CNRS group, we will centralized some actions at INRIA, Rennes.

Anybody can consult our draft of program (https://cesgo.genouest.org/groups/guggo/wiki/MEtagenomicsHackathon) but briefly, we planned to work on:

We will probably focus primarily on the toolshedization / dockerization / IFB cloudification of FROGS and ASaiM. Then, if we have time and/or if people particularly interested by these tasks, we propose to work on the modification of "home-made" Mothur and Qiime Galaxy tools.

I don't really know how a remote hackathon can work, So don't hesitate to give me some advices / informations.

We canhave opportunities for visioconfernece through Renater Rendez-vous

bgruening commented 8 years ago

@shiltemann So I count you in for next week? :) @yvanlebras this France community is awesome! I looking forward to this Codefest a lot. We usually have a google hangout open for the two days to coordinate us and to answer questions. In addition we are usually all on IRC. A few IUC members will be there and help to answer questions and reviewing code if this is desired. We can also help with Docker and any other questions. I would propose to create an IUC branch for Mothur and Qiime so we can all contribute and merge our wrappers. Maybe we can create two teams one for Mothur one for Qiime ... we will see. The aim is to have a lot of fun to hack on tools and to improve Galaxy for Metagenomic!

shiltemann commented 8 years ago

Yup sure :) I don't have a whole bunch of time, but I'd be happy to contribute where I can :)

IyadKandalaft commented 8 years ago

@bgruening I think the idea for having a mothur team and a qiime team makes sense. The mothur tool suite needs to be refactored in some way to make it a little more manageable. When we did the upgrade from 1.27 to 1.33, I had a student develop a tool that parses out the mothur documentation to list all the changes since a version that you specify. This should allow us to do another update of the Galaxy wrappers to 1.34+.

oxyko commented 8 years ago

Hey,

We are two developers from AAFC-MBB group (Canada), who are interested in contributing to Mothur and/or Quiime. Let us know if there is a kick-off meeting and/or Mothur/Quiime team that we can join.

Our Galaxy repo: https://github.com/AAFC-MBB/Galaxy

bgruening commented 8 years ago

@IyadKandalaft sounds great! Let's start with your latest version. If you send it to me I will create a branch and the Mothur team can work on this one.

IyadKandalaft commented 8 years ago

@bgruening Latest mothur 1.33 wrappers & tool dependencies (same as the central tool shed version): https://github.com/AAFC-MBB/Galaxy/tree/master/wrappers/Mothur_Toolsuite

bgruening commented 8 years ago

@oxyko do you have time to join this hackathon on: 30th of November and 1st of December @IyadKandalaft can I take this and put it in an IUC branch?

IyadKandalaft commented 8 years ago

@bgruening of course you can - we are proponents of FOSS.

oxyko commented 8 years ago

@bgruening yep. Just wondering about the format/schedule

bgruening commented 8 years ago

@oxyko it the top post not clear enough? Do you have concrete questions? This is very informal we try to get people together and organise our activities over IRC / Google-Hangouts. Code will be managed via git and we have a few expert that can help reviewing and can answer tricky questions related to the Galaxy universe. Last time we also fixed a few bug in Galaxy to make our wrappers even more awesome!

bgruening commented 8 years ago

@IyadKandalaft what do you think? https://github.com/galaxyproject/tools-iuc/issues/419

oxyko commented 8 years ago

@bgruening My concrete questions are:

  1. Is there going to be a kick-off meeting over the hangouts?
  2. How to contact the 4 mentioned google hangouts on the day of the hackathon?
bgruening commented 8 years ago
  1. Kick-off meeting is hard because we have so many timezone. If you like I will get an introduction and show a little bit planemo. Just let me know.
  2. There will be only one hangout, I will post a link over on Friday or over the weekend and everyone can connect (let see how many people can join such a hangout :))