jmchilton
commented
8 years ago @nekrut Wanted me to summarize n Slack conversation that ran away with itself on this issue. If I mis-characterize anyone - feel free to just edit my comment on Github.
- @jxtx "We scale or we die."
- Broad agreement that mulled and Conda are still good and should continue to be developed and improved. Small disagreements perhaps in how good and how rocky things have been - and whether things have been oversold.
- Broad agreement that CWL should be implemented within Galaxy - and broad concern about whether it can really replace Galaxy tools. People think it should be implemented regardless and that the spec needs to continue or re-focus on how to build great interfaces from these abstract tool and workflow descriptions.
- Concern with CWL is that it cannot be used to make "great" GUIs.
- @jxtx has the concern with CWL that templating broadly speaking is better than input binding (on this point @jmchilton disagrees).
- Agreement that @jmchilton should prepare a cwltool + conda + mulled demo for Biological Data Science poster in October. @jmchilton should also present this work to GA4GH working group on containers.
- With respect to Dockstore - the push in Galaxy should be to consume things from Dockstore less than using it to store Galaxy tools.
- Some discussion on "what is Galaxy" and "what is an idealized tool" - probably too abstract to be actionable.
- There is concern from @jxtx that Galaxy shouldn't do workflow scheduling - we do too much already and we need to find places to collaborate. @jmchilton :heart:s collaboration but worries the process of adapting to something like Mesos/Pegasus/Condor/toil would mean losing features and doing things sub-optimally. @jmchilton in short - "enhancing the core things we do is easier than replacing the core of what we do".
- @jmchilton believe scaling to 10,000 datasets in a collection across large workflows is very do-able if we focus on it - going higher is possible too if we ignore the GUI concerns.
- Some in the community have the perception that Galaxy can't scale - this worries everyone and @jmchilton made the case that it can and has scaled. No real discussion of the public relations problem - focus is more on continuing to build a technically great product that actually can scale.
- We need to work on dealing with cluster and infrastructure failures if we aren't going to pursue another backend, we should find collaborators to help implement these things if it will help save time for our limited devs and admins.
- @jxtx in short - GA4GH is building failure detection into the spec and learning from our mistakes in this regard.
- Predicting resource usage is hard, people want to work on it though.
jxtx
nsoranzo
Main Galaxy Roadmap
as described in issues.rst
The next major review of this roadmap will be done by the Galaxy Project after the 18.01 release. The discussion under this issue is moderated to keep the roadmap clear, but please feel free to comment on any part of it.
Priorities
Main/usegalaxy.org
UI rewrite
Dataset collections
Interactive environments
[Move to new Roadmap]
Workflows and Job Running
Data Libraries
Tool Shed / tool administration
Galaxy Hub and Training Materials
Metrics collection
Uploading large numbers of datasets
Cloud
OAuth
Guided tours
Histories
[New Roadmap]
main / performance
Tools
Main Maintenance/Operations
New / enhanced tools suites
Workflows and Job Running
TS/dependencies
Workflow hosting service