Error with Calculate Metrics: no applicable method for 'rename_' applied to an object of class "NULL"

[cohenp05]

Hello,

Thanks for a great package and publication! I am trying to use dyneval to compare some trajectories that I've built using infer_trajectories, and I looped over it to run the same set of methods n=10 times on the same dataset. However, when I run the dyneval::calculate_metrics call I get the error

Error in UseMethod("rename") : no applicable method for 'rename' applied to an object of class "NULL"

Here is the call I ran. The models are stored in a list object titled Trial 1 containing a list object that houses the output of infer_trajectories (per model run).

dyneval::calculate_metrics(dataset = monocyte_dyn, model = trial1$Method_1$model[[1]], metrics = metric_ids, expression_source = monocyte_dyn$expression)

I checked the class of each input. Which is shown below:

> class(monocyte_dyn)
[1] "dynwrap::with_expression" "dynwrap::data_wrapper"    "list"                    
> class(trial1$Method_1$model[[1]])
[1] "dynwrap::with_cell_waypoints" "dynwrap::with_cell_waypoints" "dynwrap::with_dimred"         "dynwrap::with_trajectory"    
[5] "dynwrap::data_wrapper"        "list"                        
> class(metrics)
[1] "list"
> class(monocyte_dyn$expression)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"

Here is my sessionInfo():

R version 3.5.3 Patched (2019-03-11 r76644)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.5

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
 [1] splines   stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] rlist_0.4.6.1       gridExtra_2.3       uwot_0.1.3          monocle_2.10.1      DDRTree_0.1.5       irlba_2.3.3         VGAM_1.1-1         
 [8] Biobase_2.42.0      BiocGenerics_0.28.0 Matrix_1.2-17       Seurat_3.0.0        forcats_0.3.0       stringr_1.4.0       dplyr_0.8.1        
[15] purrr_0.3.2         readr_1.3.1         tidyr_0.8.3         tibble_2.1.2        ggplot2_3.1.1       tidyverse_1.2.1     dyntoy_0.9.9       
[22] dyneval_0.9.9       dyno_0.1.1          dynwrap_1.1.2       dynplot_1.0.1       dynmethods_1.0.1    dynguidelines_1.0.0 dynfeature_1.0.0   
[29] rootSolve_1.7      

loaded via a namespace (and not attached):
  [1] R.methodsS3_1.7.1     GA_3.2                acepack_1.4.1         knitr_1.23            R.utils_2.8.0         data.table_1.12.2    
  [7] rpart_4.1-13          generics_0.0.2        metap_1.0             callr_3.2.0           cowplot_0.9.4         usethis_1.4.0        
 [13] RANN_2.6.1            combinat_0.0-8        carrier_0.1.0         future_1.11.1.1       xml2_1.2.0            lubridate_1.7.4      
 [19] httpuv_1.5.1          assertthat_0.2.1      viridis_0.5.1         xfun_0.7              hms_0.4.2             promises_1.0.1       
 [25] fansi_0.4.0           caTools_1.17.1.1      readxl_1.2.0          igraph_1.2.4.1        htmlwidgets_1.3       sparsesvd_0.1-4      
 [31] dyndimred_1.0.1       backports_1.1.4       gbRd_0.4-11           RcppParallel_4.4.3    vctrs_0.1.0           remotes_2.0.4        
 [37] ROCR_1.0-7            withr_2.1.2           ggforce_0.2.2         checkmate_1.9.3       sctransform_0.2.0     prettyunits_1.0.2    
 [43] cluster_2.0.7-1       ape_5.2               lazyeval_0.2.2        crayon_1.3.4          babelwhale_0.0.0.9000 pkgconfig_2.0.2      
 [49] slam_0.1-44           tweenr_1.0.1          nlme_3.1-137          pkgload_1.0.2         nnet_7.3-12           devtools_2.0.1       
 [55] rlang_0.3.4           globals_0.12.4        modelr_0.1.2          rsvd_1.0.0            cellranger_1.1.0      rprojroot_1.3-2      
 [61] polyclip_1.10-0       matrixStats_0.54.0    lmtest_0.9-36         zoo_1.8-4             base64enc_0.1-3       ggridges_0.5.1       
 [67] processx_3.3.1        pheatmap_1.0.12       png_0.1-7             viridisLite_0.3.0     bitops_1.0-6          R.oo_1.22.0          
 [73] KernSmooth_2.23-15    scales_1.0.0          memoise_1.1.0         magrittr_1.5          plyr_1.8.4            ica_1.0-2            
 [79] gplots_3.0.1.1        bibtex_0.4.2          gdata_2.18.0          compiler_3.5.3        HSMMSingleCell_1.2.0  lsei_1.2-0           
 [85] RColorBrewer_1.1-2    fitdistrplus_1.0-14   cli_1.1.0             listenv_0.7.0         patchwork_0.0.1       pbapply_1.3-4        
 [91] ps_1.3.0              htmlTable_1.13.1      Formula_1.2-3         MASS_7.3-51.1         tidyselect_0.2.5      stringi_1.4.3        
 [97] densityClust_0.3      yaml_2.2.0            latticeExtra_0.6-28   ggrepel_0.8.1         grid_3.5.3            tools_3.5.3          
[103] future.apply_1.1.0    rstudioapi_0.10       foreach_1.4.4         foreign_0.8-71        rje_1.9               farver_1.1.0         
[109] Rtsne_0.15            ggraph_1.0.2          proxyC_0.1.3          digest_0.6.19         FNN_1.1.3             shiny_1.3.2          
[115] dynparam_1.0.0        qlcMatrix_0.9.7       Rcpp_1.0.1            broom_0.5.1           SDMTools_1.1-221      later_0.8.0          
[121] httr_1.4.0            npsurv_0.4-0          Rdpack_0.10-1         colorspace_1.4-1      rvest_0.3.2           fs_1.2.6             
[127] pdist_1.2             ranger_0.11.2         reticulate_1.10       plotly_4.8.0          sessioninfo_1.1.1     xtable_1.8-4         
[133] jsonlite_1.6          tidygraph_1.1.2       zeallot_0.1.0         testthat_2.1.1        R6_2.4.0              Hmisc_4.2-0          
[139] pillar_1.4.1          htmltools_0.3.6       mime_0.6              glue_1.3.1            codetools_0.2-16      pkgbuild_1.0.3       
[145] tsne_0.1-3            utf8_1.1.4            lattice_0.20-38       curl_3.3              gtools_3.8.1          survival_2.43-3      
[151] limma_3.38.3          docopt_0.6.1          desc_1.2.0            dynutils_1.0.3        fastICA_1.2-1         munsell_0.5.0        
[157] iterators_1.0.10      haven_2.1.0           reshape2_1.4.3        gtable_0.3.0     

Thank you for your help!

[rcannood]

Hello @cohenp05,

Could you provide a minimum reproducible example for which your problem occurs? For now, I haven't managed to replicate it using the following code.

Robrecht

library(dyno)
library(tidyverse)

dataset <- dyntoy::generate_dataset()
traj <- infer_trajectory(dataset, ti_slingshot())

metric_ids <- c("correlation", "him", "featureimp_wcor", "F1_branches")
dyneval::calculate_metrics(dataset = dataset, model = traj, metrics = metric_ids, expression_source = dataset$expression)

# A tibble: 1 x 12
  time_waypointedgeodesic correlation time_correlation   him time_him time_featureimp featureimp_cor featureimp_wcor time_mapping_branches recovery_branches relevance_branches F1_branches
                    <dbl>       <dbl>            <dbl> <dbl>    <dbl>           <dbl>          <dbl>           <dbl>                 <dbl>             <dbl>              <dbl>       <dbl>
1                  0.0878       0.949          0.00342     1   0.0263            1.68          0.797           0.687                 0.104                 1                  1           1

[zouter]

It's a bit weird that calculate_metrics doesn't error on https://github.com/dynverse/dyneval/blob/master/R/calculate_metrics.R#L49 because the dataset in your case does not contain cell waypoints (yet).

Apart from what @rcannood suggested, could you perhaps also try to add waypoints by: monocyte_dyn <- dynwrap::add_cell_waypoints(monocyte_dyn) and see whether it errors in that case?

[zouter]

Hi @cohenp05

Apart from what @rcannood said, could you show us the names() of the dataset and model object? It looks like that neither of these objects contain a trajectory (i.e. milestone network, milestone percentages, ...)

[cohenp05]

Hello,

Thanks for your help. I've made an example below, but the code is still very lengthy and certainly overly verbose. Sorry about that. Anyway here is what I've written, and I still get the same error.

In order to run this I'm using a dataset that I had previously analyzed in the R package Seurat and also in Monocle. Those objects are called dataset_seurat and dataset_monocle.

library(dyno)
library(dyneval)
library(dyntoy)
#Check if docker is correctly installed
dynwrap::test_docker_installation(detailed = TRUE)
library(tidyverse)
library(Seurat)
library(Matrix)
library(monocle)
library(uwot)
library(rlist)

#Pull counts data from Seurat, log2 normalize, and load into Dynwrap
dataset_counts <- GetAssayData(object = dataset_seurat, slot = "counts", assay = "RNA")
dataset_norm <- log2(dataset_counts+1)
dataset_norm <- as(dataset_norm, "dgCMatrix")
dataset_norm <- t(dataset_norm)
dataset_counts <- t(dataset_counts)
dataset_dyn <- wrap_expression(expression = dataset_norm, counts = dataset_counts)

#Pull barcodes and pseudotime assignments from Monocle object of corresponding data
barcode <- rownames(pData(dataset_monocle))
pseudotime <- pData(dataset_monocle)$Pseudotime
cellsinpseudotime <- data.frame(cells = barcode, pseudotime = pseudotime)

#Use only barcodes that are shared between dynwrap object and monocle object
cellsindyn <- dataset_dyn$cell_ids
cellsinpseudotime <- cellsinpseudotime[barcode %in% cellsindyn,]

#Define start cell and end cell as minimum pseudotime value 
# and maximum pseudotime value in monocle analysis 
startcell <- as.character(cellsinpseudotime[pseudotime==min(pseudotime),1])
endcell <- as.character(cellsinpseudotime[pseudotime==max(pseudotime),1])

#Add start and end_id as prior information
dataset_dyn <- add_prior_information(dataset = dataset_dyn, start_id = startcell, end_id = endcell, start_n = 1, end_n = 1)

method_tis <- list(ti_comp1(), ti_mst(), ti_shuffle())
set.seed(seed = 456)
dataset_model <- list()

##Make 3 TI models
##Repeat these methods 3 times and store all 9 models in the list object dataset_model
for (i in seq(3)){
  model_method <- list()
  for (j in seq(length(method_tis))){
    model_method[[j]] <- infer_trajectories(dataset = dataset_dyn, method = method_tis[j], verbose = TRUE, give_priors = c("start_id", "end_id")) 
    }
  dataset_model[[i]] <- model_method
  }

#Root All Models by CD14 Expression
for (i in seq(length(dataset_model))){
  for (j in seq(length(method_tis))){
    if ("dynwrap::with_trajectory" %in% class(dataset_model[[i]][[j]][[6]][[1]])){
      dataset_model[[i]][[j]][[6]][[1]] <- add_root_using_expression(dataset_model[[i]][[j]][[6]][[1]], features_oi = "CD14", expression_source = dataset_dyn$expression)
    }
  }
}

#Calculate Pseudotime for All Models
for (i in seq(dataset_model)){
  for (j in seq(length(method_tis))){
    if ("dynwrap::with_trajectory" %in% class(dataset_model[[i]][[j]][[6]][[1]])){
      dataset_model[[i]][[j]][[6]][[1]]<- add_pseudotime(dataset_model[[i]][[j]][[6]][[1]])
    }
  }
}

#Add cell waypoints for all models
for (i in seq(length(dataset_model)){
  for (j in seq(length(method_tis))){
    if (is_wrapper_with_trajectory(dataset_model[[i]][[j]][[6]][[1]]) == TRUE){
      dataset_model[[i]][[j]][[6]][[1]] <- add_cell_waypoints(trajectory = dataset_modelcp[[i]][[j]][[6]][[1]]) 
    }
  }
}

##Select metrics to calculate: include all metrics except metrics that have the category "average"
metric_ids <- dyneval::metrics %>% filter(category != "average") %>% pull(metric_id)
metrics <- list()

##Calculate Metrics
for (i in seq(length(dataset_model))){
  metric_trial <- list()
  for (j in seq(length(method_tis))){
    if (is_wrapper_with_trajectory(dataset_model[[i]][[j]][[6]][[1]]) == TRUE){
      metric_trial[j] <- map_dfr(dataset_model, dyneval::calculate_metrics, dataset = dataset_dyn, metrics = metric_ids)
    }
  }
  metrics[i] <- metric_trial
}

The names of the dataset and (one example) model object are below:

> names(dataset_dyn)
[1] "id"                   "cell_ids"             "cell_info"            "counts"               "expression"           "expression_projected"
[7] "feature_info"         "feature_ids"          "prior_information"   
> names(dataset_model[[1]][[1]][[6]][[1]])
 [1] "id"                          "cell_ids"                    "cell_info"                   "milestone_ids"              
 [5] "milestone_network"           "divergence_regions"          "milestone_percentages"       "progressions"               
 [9] "trajectory_type"             "directed"                    "pseudotime"                  "dimred"                     
[13] "dimred_milestones"           "dimred_segment_progressions" "dimred_segment_points"       "root_milestone_id" 

[rcannood]

Hey @cohenp05,

In order to calculate any metrics, your dataset_dyn needs to have a gold standard trajectory. For this, you need to run something like:

dataset_dyn <- 
  wrap_expression(
    expression = dataset_norm, 
    counts = dataset_counts
  ) %>% 
    add_trajectory(
      milestone_network = ...,
      milestone_percentages = ...
    )

Where you define milestone network and milestone percentage according to the trajectory that is in the dataset. Check ?add_trajectory for more information of the format of the milestone network and percentages. Do you have this information?

Robrecht

[cohenp05]

Hey @rcannood,

Thanks so much for your help, and sorry for such a long lapse in response. Looking back I think perhaps my question is a bit naive and I think that I wasn't communicating it well. Ideally, I'd like to compare several trajectories generated using dynverse in an experimental setting where the ground truth is unknown. I was wondering if there are commands in dyneval (or elsewhere) that can be used to determine a "best fit" for the data. Your group has outlined a wealth of tests to run in the supplement, but in an experimental setting without a known ground truth, is there a better way to choose a trajectory (once several have been built)? Thank you again for your help

Phil C