CellTrails

[rcannood]

Hello @dcellwanger

This issue is for discussing the wrapper for your trajectory inference method, CellTrails, which we wrapped for our benchmarking study (10.1101/276907). In our dynmethods framework, we collected some meta information about your method, and created a docker wrapper so that all methods can be easily run and compared. The code for this wrapper is located in a docker container. The way this container is structured is described in this vignette.

We are creating this issue to ensure your method is being evaluated in the way it was designed for. The checklist below contains some important questions for you to have a look at.

• [x] Parameters, defined in definition.yml (more info)
  • Are all important parameters described in this file?
  • For each parameter, is the proposed default value reasonable?
  • For each parameter, is the proposed parameter space reasonable (e.g. lower and upper boundaries)?
  • Is the description of the parameters correct and up-to-date?
• [x] Input, defined in definition.yml and loaded in run.R (more info)
  • Is the correct type of expression requested (raw counts or normalised expression)?
  • Is all prior information (required or optional) requested?
  • Would some other type of prior information help the method?
• [x] Output, defined in definition.yml and saved in run.R (more info)
  • Is the output correctly processed towards the common trajectory model? Would some other postprocessing method make more sense?
  • Is all relevant output saved (dimensionality reduction, clustering/grouping, pseudotime, ...)
• [x] Wrapper script, see run.R (more info)
  • This is a script that is executed upon starting the docker container. It will receive several input files as defined by definition.yml, and is expected to produce certain output files, also as defined by definition.yml.
  • Is the script a correct representation of the general workflow a user is expected to follow when they want to apply your method to their data?
• [x] Quality control, see the qc worksheet
  • We also evaluated the implementation of a method based on a large check list of good software software development practices.
  • Are the answers we wrote down for your method correct and up to date? Do you disagree with certain answers? (Feel free to leave a comment in the worksheet)
  • You can improve the QC score of your method by implementing the required changes and letting us know. Do not gloss over this, as it is the easiest way to improve the overall ranking of your TI method in our study!

The most convenient way for you to test and adapt the wrapper is to install dyno, download and modify these files, and run your method on a dataset of interest or one of our synthetic toy datasets. This is further described in this vignette. Once finished, we prefer that you fork the dynmethods repository, make the necessary changes, and send us a pull request. Alternatively, you can also send us the files and we will make the necessary changes.

If you have any further questions or remarks, feel free to reply to this issue.

Kind regards, @zouter and @rcannood

[dcellwanger]

Hello @rcannood and @zouter,

Thanks for considering CellTrails in your work. I am very sorry for my late response - I was out of office. We will have a look into your requests. What is your anticipated project timeline?

Best wishes, Daniel

[zouter]

Hi @dcellwanger

Thank you for your response!

Our current timeline is that we would generate the final benchmarking results from the 6th of august onwards, but hope to obtain the final versions of each method around the end of july.

Thanks! Wouter

[dcellwanger]

Hi @zouter,

I browsed through your spreadsheet 'trajectory inference QC' and have the following comments on the CellTrails sheet:

The method uses continuous integration, for example on Travis CI

We have added Travis CI integration. Please note that the most recent release (CellTrails 0.99.15) resolved of a minor bug which was arising due to the latest major update of one of its dependencies (ggplot2).

The tutorial showcases the method on several datasets (1=0, 2=0.5, >2=1)

1. The whole workflow is demonstrated in detail using one real biological dataset (data from Ellwanger et al., 2018)
2. Also the whole protocol for analyzing another real biological dataset is provided in the Appendix section (Section 9.1; data from Burns et al., 2015)
3. Section 3 uses simulated data
4. Section 4.3 uses Th2 differentiation data (from Mahata et al., 2014)

The paper shows the method's usefulness on several (1), one (0.25) or no real datasets.

The paper demonstrates that the CellTrails framework is suitable for a wide range of problems using

1. RT-qPCR data from the inner ear (Figure 1-6; data from Ellwanger et al., 2018),
2. RNA-Seq data from the inner ear (Figure 7; data from Burns et al., 2015), as well as
3. single-cell measurements from hematopoiesis (Figure S7F-H; data from Moignard et al., 2013), and
4. from early development of the mouse embryo (Figure 7I-K; data from Guo et al., 2010).

Package contains dummy proofing, i.e. testing whether the parameters and data supplied by the user make sense and are useful

To a certain degree, CellTrails checks, indeed, wrong parameter settings which occurred during the pre-release package usage by the internal and external userbase; an exception is thrown or a default value is set as indicated by a warning message.

Cheers, Daniel

[rcannood]

Hello Daniel,

Thanks for getting back in touch with us. I have a had a look at your comments and modified the QC scores accordingly.

Kind regards, Robrecht

QC23

QC23 (continuous integration): 0 → 1 If you like, you can add code coverage quite easily as well, by adding the following code to your .travis.yml. This would increase the QC24 from 0 to 1 as well.

r_packages:
 - covr
after_success:
 - R CMD BiocCheck .
 - R -e 'covr::codecov()'

QC34 and QC49

QC34 (tutorial datasets): 0 → 1 QC49 (paper datasets): 0.25 → 1 I see! This was an oversight on our part.

QC20

QC20 (dummy proofing): 0 → 1 I had a closer look through the code and indeed saw dummy proofing for various exposed functions

[dcellwanger]

Hello Robrecht,

Q24,25 Current code coverage = 81.63%

Q28,29 Version numbering and separation in devel/stable branches is somewhat tied to Bioconductor's life cycle (see Version Numbering), and therefore, meeting this criteria disadvantages novel Bioconductor packages.

Since its recent acceptance at Bioconductor, CellTrails was only listed at Bioconductor's devel branch. I would argue that the code at GitHub represents the developmental version, while the code at Bioconductor is the stable version. Please note that CellTrails had, indeed, several minor releases (see NEWS).

Biannually the Bioconductor devel branch becomes the Bioconductor release branch (in mid-April and mid-October). This will automatically generate a developmental branch of CellTrails at Bioconductor (mirrored at GitHub), where new features are introduced, and a stable branch of CellTrails at Bioconductor, respectively. Package version numbering is performed accordingly (see link above).

-Daniel

[rcannood]

QC24: Nice! I updated it in the QC worksheet.

QC28,29: Fair enough. I will discuss this with @zouter. Perhaps all the bioconductor packages should get a score of 1 for this question. I

[rcannood]

Hi Daniel,

QC28,29: We've changed the values of these scores to '1' for all bioconductor packages. This results in CellTrails having a near-perfect QC score.

I'm closing this issue for now. If you have any further comments on the CellTrails wrapper, feel free to create a new issue or pull request.

Robrecht