Monocle

[zouter]

Hello @Xiaojieqiu and @ctrapnell

This issue is for discussing the wrapper for your trajectory inference method, Monocle, which we wrapped for our benchmarking study (10.1101/276907). In our dynmethods framework, we collected some meta information about your method, and created a docker wrapper so that all methods can be easily run and compared. The code for this wrapper is located in docker containers[1][2]. The way this container is structured is described in this vignette.

We created 2 separate wrappers:

• Monocle DDRTree: Monocle (v2) using DDRTree
• Monocle ICA: Monocle (v1) using ICA

We are creating this issue to ensure your method is being evaluated in the way it was designed for. The checklist below contains some important questions for you to have a look at.

• [ ] Parameters, defined in definition.yml (more info)
  • Are all important parameters described in this file?
  • For each parameter, is the proposed default value reasonable?
  • For each parameter, is the proposed parameter space reasonable (e.g. lower and upper boundaries)?
  • Is the description of the parameters correct and up-to-date?
• [ ] Input, defined in definition.yml and loaded in run.R (more info)
  • Is the correct type of expression requested (raw counts or normalised expression)?
  • Is all prior information (required or optional) requested?
  • Would some other type of prior information help the method?
• [ ] Output, defined in definition.yml and saved in run.R (more info)
  • Is the output correctly processed towards the common trajectory model? Would some other postprocessing method make more sense?
  • Is all relevant output saved (dimensionality reduction, clustering/grouping, pseudotime, ...)
• [ ] Wrapper script, see run.R (more info)
  • This is a script that is executed upon starting the docker container. It will receive several input files as defined by definition.yml, and is expected to produce certain output files, also as defined by definition.yml.
  • Is the script a correct representation of the general workflow a user is expected to follow when they want to apply your method to their data?
• [ ] Quality control, see the qc worksheet
  • We also evaluated the implementation of a method based on a large check list of good software software development practices.
  • Are the answers we wrote down for your method correct and up to date? Do you disagree with certain answers? (Feel free to leave a comment in the worksheet)
  • You can improve the QC score of your method by implementing the required changes and letting us know. Do not gloss over this, as it is the easiest way to improve the overall ranking of your TI method in our study!

The most convenient way for you to test and adapt the wrapper is to install dyno, download and modify these files, and run your method on a dataset of interest or one of our synthetic toy datasets. This is further described in this vignette. Once finished, we prefer that you fork the dynmethods repository, make the necessary changes, and send us a pull request. Alternatively, you can also send us the files and we will make the necessary changes.

If you have any further questions or remarks, feel free to reply to this issue.

Kind regards, @rcannood and @zouter

[rcannood]

I recently saw on twitter that monocle 3 is being released. Should monocle 3 be included in dynmethods?

[zouter]

It is still in alpha release, but it would be great to include it, especially as it competes with PAGA on several fronts (disconnected graphs, scalability, ...). Perhaps @Xiaojieqiu or @ctrapnell can chime in on how we can best handle this?

[ctrapnell]

I would suggest that you not include it until Monocle 3 is declared finished. Monocle 3 isn't even feature complete, and we are likely to change much about how it works both internally and in terms of the interface.

More generally, comparing software that hasn't even been described in a preprint form against published, polished code in some sort of competitive bakeoff just doesn't seem fair. Doing so risks discouraging our group and others from releasing alpha code in the future.

[rcannood]

Hello @ctrapnell,

With the features that you listed in your recent tweets (disjoint trajectories, cycles, high scalability, finding trajectory dependent genes, tests written by Jun Cao), I think it'd be hard for anyone not to get excited by the announcement of Monocle 3. Please don't misinterpret my enthousiasm as an attempt to stifle your group in any form or fashion.

I understand that Monocle 3 is still in an alpha state, and that you would prefer us not to include it as part of this package just yet; that's fine ─ that is why we are asking for your opinion. Would you agree waiting for a bioRxiv / Bioconductor release of Monocle 3 before including it in dynmethods would seem a reasonable approach to take?

On a related note, would you have time to take a look at the dynmethods wrappers for Monocle ICA/DDRTree in the near future? Do you have any suggestions or improvements? Do recent changes in the Monocle code result in a better QC score?

Kind regards, Robrecht

[ctrapnell]

Hi @rcannood -

I'm glad you are excited about it! I didn't mean to snap, either. I do think it's important to have quantitative evaluations of tools. My point was that until there's a paper to go along with the software, it will be hard for the wider community of methods developers to learn what Monocle 3 is doing well vs. the places where other tools' designs are the right way to go.

So I definitely think that waiting until there's at least a pre-print would be good, but perhaps Monocle 3 could be included once the preprints are up but before the papers are actually published. As we did with Monocle 2, Monocle 3 will probably be described in several papers as we bring different chunks of functionality online. We're not entirely sure what the preprint schedule will be exactly, but hopefully the main trajectory code will be described at least on bioRxiv within a couple of months.

Re: the DDRTree wrapper - we haven't had a chance to look at this yet, but will try to do so soon. Thanks for including Monocle 2 in dynmethods, and for keeping an eye on Monocle 3!

C

[rcannood]

I'm closing this issue. If you have any further remarks on the Monocle 2 wrapper (which was updated recently), or if you would like us to add a wrapper for Monocle 3 at some point, feel free to open up a new issue!

Kind regards, Robrecht