search

drighelli / SpatialExperiment

48 stars 17 forks source link

readImgData does not find the image files (might be a winOS exclusive issue) #64

Closed lcolladotor closed 3 years ago

lcolladotor commented 3 years ago

Hi,

I'm working on updating spatialLIBD to match SpatialExperiment version 1.1.700 at https://github.com/LieberInstitute/spatialLIBD/commit/63911f463a0339cf3d0e719c8a51d7118cc670d7. I normally work on spatialLIBD from a macOS computer but today I did so from a winOS one. While working on spatialLIBD I noticed that I cannot get SpatialExperiment to load images.

spatialLIBD-based example

## Download and save a local cache of the data provided by 10x Genomics
bfc <- BiocFileCache::BiocFileCache()
lymph.url <-
    paste0(
        "https://cf.10xgenomics.com/samples/spatial-exp/",
        "1.1.0/V1_Human_Lymph_Node/",
        c(
            "V1_Human_Lymph_Node_filtered_feature_bc_matrix.tar.gz",
            "V1_Human_Lymph_Node_spatial.tar.gz"
        )
    )
lymph.data <- sapply(lymph.url, BiocFileCache::bfcrpath, x = bfc)

## Extract the files to a temporary location
## (they'll be deleted once you close your R session)
sapply(lymph.data, utils::untar, exdir = tempdir())
#> https://cf.10xgenomics.com/samples/spatial-exp/1.1.0/V1_Human_Lymph_Node/V1_Human_Lymph_Node_filtered_feature_bc_matrix.tar.gz.BFC1 
#>                                                                                                                                   0 
#>                    https://cf.10xgenomics.com/samples/spatial-exp/1.1.0/V1_Human_Lymph_Node/V1_Human_Lymph_Node_spatial.tar.gz.BFC2 
#>                                                                                                                                   0

## List the files we downloaded and extracted
## These files are typically SpaceRanger outputs
lymph.dirs <- file.path(
    tempdir(),
    c("filtered_feature_bc_matrix", "spatial", "raw_feature_bc_matrix")
)
list.files(lymph.dirs)
#> [1] "aligned_fiducials.jpg"     "barcodes.tsv.gz"          
#> [3] "detected_tissue_image.jpg" "features.tsv.gz"          
#> [5] "matrix.mtx.gz"             "scalefactors_json.json"   
#> [7] "tissue_hires_image.png"    "tissue_lowres_image.png"  
#> [9] "tissue_positions_list.csv"

## Import the data as a SpatialExperiment object
suppressPackageStartupMessages(library("SpatialExperiment"))
spe <- SpatialExperiment::read10xVisium(
    samples = tempdir(),
    sample_id = "lymph",
    type = "sparse", data = "filtered",
    images = "lowres", load = TRUE
)
## Check object
spe
#> class: SpatialExperiment 
#> dim: 36601 4035 
#> metadata(0):
#> assays(1): counts
#> rownames(36601): ENSG00000243485 ENSG00000237613 ... ENSG00000278817
#>   ENSG00000277196
#> rowData names(1): symbol
#> colnames(4035): AAACAAGTATCTCCCA-1 AAACAATCTACTAGCA-1 ...
#>   TTGTTTGTATTACACG-1 TTGTTTGTGTAAATTC-1
#> colData names(1): sample_id
#> reducedDimNames(0):
#> mainExpName: NULL
#> altExpNames(0):
#> spatialData names(3) : in_tissue array_row array_col
#> spatialCoords names(2) : pxl_col_in_fullres pxl_row_in_fullres
#> imgData names(0):

## imgData is empty =(
imgData(spe)
#> DataFrame with 0 rows and 0 columns

## Returns NULL
SpatialExperiment::readImgData(
    path = file.path(tempdir(), "spatial"),
    sample_id = "lymph"
)
#> NULL

## Re-make arguments in readImgData
path <- file.path(tempdir(), "spatial")
sample_id <- "lymph"
imageSources <- file.path(path, "tissue_lowres_image.png")
scaleFactors <- file.path(path, "scalefactors_json.json")
load <- TRUE

## Check files exist
stopifnot(all(file.exists(c(imageSources, scaleFactors))))

## Execute https://github.com/drighelli/SpatialExperiment/blob/b1e5ce753e3bd43306074442fe0a80f0bc6190b3/R/readImgData.R#L46-L56
# get sample identifiers
if (is.null(sample_id))
    stop("'sample_id' mustn't be NULL")
stopifnot(
    is.character(sample_id),
    length(unique(sample_id)) == length(path))
names(path) <- names(scaleFactors) <- sample_id

# put images into list with one element per sample
images <- lapply(path, function(.) 
    grep(., imageSources, value=TRUE))

## images is NULL =(
images
#> $lymph
#> character(0)

## Note path and imageSources are full paths in winOS
imageSources
#> [1] "C:\\Users\\fellg\\AppData\\Local\\Temp\\RtmpkTgKdg/spatial/tissue_lowres_image.png"
path
#>                                                        lymph 
#> "C:\\Users\\fellg\\AppData\\Local\\Temp\\RtmpkTgKdg/spatial"

Created on 2021-05-08 by the reprex package (v2.0.0)

SpatialExperiment docs-based example

dir <- system.file(
    file.path("extdata", "10xVisium", "section1", "spatial"),
    package = "SpatialExperiment")

# base directory contains 
# - scale factors (scalefactors_json.json)
# - one image (tissue_lowres_image.png)
list.files(dir)
#> [1] "scalefactors_json.json"    "tissue_lowres_image.png"  
#> [3] "tissue_positions_list.csv"

# read in images & scale factors
# as valid 'imgData' 'DFrame'
SpatialExperiment::readImgData(dir, sample_id = "foo")
#> DataFrame with 1 row and 4 columns
#>     sample_id    image_id   data scaleFactor
#>   <character> <character> <list>   <numeric>
#> 1         foo      lowres   ####   0.0510334

## Check path
dir
#> [1] "C:/R/R-4.1.0alpha/library/SpatialExperiment/extdata/10xVisium/section1/spatial"
## Note that it doesn't have \\

Created on 2021-05-08 by the reprex package (v2.0.0)

Solutions?

Could normalizePath() resolve this?

SpatialExperiment::readImgData(
    path = normalizePath(file.path(tempdir(), "spatial")),
    sample_id = "lymph"
)
#> NULL
normalizePath(path)
#> [1] "C:\\Users\\fellg\\AppData\\Local\\Temp\\RtmpApKoPq\\spatial"
path
#> [1] "C:\\Users\\fellg\\AppData\\Local\\Temp\\RtmpApKoPq/spatial"

Hmm, nope.

What about grep(fixed = TRUE)?

> lapply(path, function(.) 
+     grep(., imageSources, value=TRUE, fixed = TRUE))
$lymph
[1] "C:\\Users\\fellg\\AppData\\Local\\Temp\\RtmpApKoPq/spatial/tissue_lowres_image.png"

Ok, that seems to work!

lcolladotor commented 3 years ago

I forgot the R session info

- Session info -------------------------------------------------------------------------------------------------------
 setting  value                                    
 version  R version 4.1.0 alpha (2021-04-20 r80202)
 os       Windows 10 x64                           
 system   x86_64, mingw32                          
 ui       RStudio                                  
 language (EN)                                     
 collate  English_United States.1252               
 ctype    English_United States.1252               
 tz       America/New_York                         
 date     2021-05-08                               

- Packages -----------------------------------------------------------------------------------------------------------
 ! package                * version    date       lib source                             
   AnnotationDbi            1.53.1     2021-03-09 [1] Bioconductor                       
   AnnotationHub            2.99.6     2021-05-06 [1] Bioconductor                       
   assertthat               0.2.1      2019-03-21 [1] CRAN (R 4.1.0)                     
   attempt                  0.3.1      2020-05-03 [1] CRAN (R 4.1.0)                     
   backports                1.2.1      2020-12-09 [1] CRAN (R 4.1.0)                     
   beachmat                 2.7.7      2021-03-08 [1] Bioconductor                       
   beeswarm                 0.3.1      2021-03-07 [1] CRAN (R 4.1.0)                     
   benchmarkme              1.0.7      2021-03-21 [1] CRAN (R 4.1.0)                     
   benchmarkmeData          1.0.4      2020-04-23 [1] CRAN (R 4.1.0)                     
   Biobase                * 2.51.0     2021-03-09 [1] Bioconductor                       
   BiocFileCache          * 1.99.8     2021-05-06 [1] Bioconductor                       
   BiocGenerics           * 0.37.5     2021-05-04 [1] Bioconductor                       
   BiocIO                   1.1.2      2021-03-08 [1] Bioconductor                       
   BiocManager              1.30.13    2021-05-07 [1] CRAN (R 4.1.0)                     
   BiocNeighbors            1.9.4      2021-03-08 [1] Bioconductor                       
   BiocParallel             1.25.5     2021-03-09 [1] Bioconductor                       
   BiocSingular             1.7.2      2021-03-08 [1] Bioconductor                       
   BiocVersion              3.13.1     2021-03-08 [1] Bioconductor                       
   Biostrings               2.59.2     2021-03-09 [1] Bioconductor                       
   bit                      4.0.4      2020-08-04 [1] CRAN (R 4.1.0)                     
   bit64                    4.0.5      2020-08-30 [1] CRAN (R 4.1.0)                     
   bitops                   1.0-7      2021-04-24 [1] CRAN (R 4.1.0)                     
   blob                     1.2.1      2020-01-20 [1] CRAN (R 4.1.0)                     
   bslib                    0.2.4      2021-01-25 [1] CRAN (R 4.1.0)                     
   cachem                   1.0.4      2021-02-13 [1] CRAN (R 4.1.0)                     
   callr                    3.7.0      2021-04-20 [1] CRAN (R 4.1.0)                     
   cli                      2.5.0      2021-04-26 [1] CRAN (R 4.1.0)                     
   clipr                    0.7.1      2020-10-08 [1] CRAN (R 4.1.0)                     
   codetools                0.2-18     2020-11-04 [1] CRAN (R 4.1.0)                     
   colorspace               2.0-1      2021-05-04 [1] CRAN (R 4.1.0)                     
   config                   0.3.1      2020-12-17 [1] CRAN (R 4.1.0)                     
   cowplot                  1.1.1      2020-12-30 [1] CRAN (R 4.1.0)                     
   crayon                   1.4.1      2021-02-08 [1] CRAN (R 4.1.0)                     
   curl                     4.3.1      2021-04-30 [1] CRAN (R 4.1.0)                     
   data.table               1.14.0     2021-02-21 [1] CRAN (R 4.1.0)                     
   DBI                      1.1.1      2021-01-15 [1] CRAN (R 4.1.0)                     
   dbplyr                 * 2.1.1      2021-04-06 [1] CRAN (R 4.1.0)                     
   DelayedArray             0.17.13    2021-05-02 [1] Bioconductor                       
   DelayedMatrixStats       1.13.6     2021-04-22 [1] Bioconductor                       
   desc                     1.3.0      2021-03-05 [1] CRAN (R 4.1.0)                     
   devtools               * 2.4.1      2021-05-05 [1] CRAN (R 4.1.0)                     
   digest                   0.6.27     2020-10-24 [1] CRAN (R 4.1.0)                     
   dockerfiler              0.1.3      2019-03-19 [1] CRAN (R 4.1.0)                     
   doParallel               1.0.16     2020-10-16 [1] CRAN (R 4.1.0)                     
   dotCall64                1.0-1      2021-02-11 [1] CRAN (R 4.1.0)                     
   dplyr                    1.0.6      2021-05-05 [1] CRAN (R 4.1.0)                     
   dqrng                    0.3.0      2021-05-01 [1] CRAN (R 4.1.0)                     
   DropletUtils             1.11.12    2021-03-08 [1] Bioconductor                       
   DT                       0.18       2021-04-14 [1] CRAN (R 4.1.0)                     
   edgeR                    3.33.3     2021-03-09 [1] Bioconductor                       
   ellipsis                 0.3.2      2021-04-29 [1] CRAN (R 4.1.0)                     
   evaluate                 0.14       2019-05-28 [1] CRAN (R 4.1.0)                     
   ExperimentHub            1.99.7     2021-05-06 [1] Bioconductor                       
   fansi                    0.4.2      2021-01-15 [1] CRAN (R 4.1.0)                     
   fastmap                  1.1.0      2021-01-25 [1] CRAN (R 4.1.0)                     
   fields                   11.6       2020-10-09 [1] CRAN (R 4.1.0)                     
   filelock                 1.0.2      2018-10-05 [1] CRAN (R 4.1.0)                     
   foreach                  1.5.1      2020-10-15 [1] CRAN (R 4.1.0)                     
   fs                       1.5.0      2020-07-31 [1] CRAN (R 4.1.0)                     
   generics                 0.1.0      2020-10-31 [1] CRAN (R 4.1.0)                     
   GenomeInfoDb           * 1.27.11    2021-04-13 [1] Bioconductor                       
   GenomeInfoDbData         1.2.5      2021-05-08 [1] Bioconductor                       
   GenomicAlignments        1.27.2     2021-03-09 [1] Bioconductor                       
   GenomicRanges          * 1.43.4     2021-04-04 [1] Bioconductor                       
   ggbeeswarm               0.6.0      2017-08-07 [1] CRAN (R 4.1.0)                     
   ggplot2                  3.3.3      2020-12-30 [1] CRAN (R 4.1.0)                     
   glue                     1.4.2      2020-08-27 [1] CRAN (R 4.1.0)                     
   golem                    0.3.1      2021-04-17 [1] CRAN (R 4.1.0)                     
   gridExtra                2.3        2017-09-09 [1] CRAN (R 4.1.0)                     
   gtable                   0.3.0      2019-03-25 [1] CRAN (R 4.1.0)                     
   HDF5Array                1.19.16    2021-05-02 [1] Bioconductor                       
   highr                    0.9        2021-04-16 [1] CRAN (R 4.1.0)                     
   hms                      1.0.0      2021-01-13 [1] CRAN (R 4.1.0)                     
   htmltools                0.5.1.1    2021-01-22 [1] CRAN (R 4.1.0)                     
   htmlwidgets              1.5.3      2020-12-10 [1] CRAN (R 4.1.0)                     
   httpuv                   1.6.1      2021-05-07 [1] CRAN (R 4.1.0)                     
   httr                     1.4.2      2020-07-20 [1] CRAN (R 4.1.0)                     
   interactiveDisplayBase   1.29.0     2021-03-08 [1] Bioconductor                       
   IRanges                * 2.25.11    2021-05-05 [1] Bioconductor                       
   irlba                    2.3.3      2019-02-05 [1] CRAN (R 4.1.0)                     
   iterators                1.0.13     2020-10-15 [1] CRAN (R 4.1.0)                     
   jquerylib                0.1.4      2021-04-26 [1] CRAN (R 4.1.0)                     
   jsonlite                 1.7.2      2020-12-09 [1] CRAN (R 4.1.0)                     
   KEGGREST                 1.31.2     2021-05-05 [1] Bioconductor                       
   knitr                    1.33       2021-04-24 [1] CRAN (R 4.1.0)                     
   later                    1.2.0      2021-04-23 [1] CRAN (R 4.1.0)                     
   lattice                  0.20-44    2021-05-02 [1] CRAN (R 4.1.0)                     
   lazyeval                 0.2.2      2019-03-15 [1] CRAN (R 4.1.0)                     
   lifecycle                1.0.0      2021-02-15 [1] CRAN (R 4.1.0)                     
   limma                    3.47.13    2021-05-02 [1] Bioconductor                       
   locfit                   1.5-9.4    2020-03-25 [1] CRAN (R 4.1.0)                     
   lubridate                1.7.10     2021-02-26 [1] CRAN (R 4.1.0)                     
   magick                   2.7.2      2021-05-02 [1] CRAN (R 4.1.0)                     
   magrittr                 2.0.1      2020-11-17 [1] CRAN (R 4.1.0)                     
   maps                     3.3.0      2018-04-03 [1] CRAN (R 4.1.0)                     
   Matrix                   1.3-3      2021-05-04 [1] CRAN (R 4.1.0)                     
   MatrixGenerics         * 1.3.1      2021-03-08 [1] Bioconductor                       
   matrixStats            * 0.58.0     2021-01-29 [1] CRAN (R 4.1.0)                     
   memoise                  2.0.0      2021-01-26 [1] CRAN (R 4.1.0)                     
   mime                     0.10       2021-02-13 [1] CRAN (R 4.1.0)                     
   munsell                  0.5.0      2018-06-12 [1] CRAN (R 4.1.0)                     
   pillar                   1.6.0      2021-04-13 [1] CRAN (R 4.1.0)                     
   pkgbuild                 1.2.0      2020-12-15 [1] CRAN (R 4.1.0)                     
   pkgconfig                2.0.3      2019-09-22 [1] CRAN (R 4.1.0)                     
   pkgload                  1.2.1      2021-04-06 [1] CRAN (R 4.1.0)                     
   plotly                   4.9.3      2021-01-10 [1] CRAN (R 4.1.0)                     
   png                      0.1-7      2013-12-03 [1] CRAN (R 4.1.0)                     
   Polychrome               1.2.6      2020-11-11 [1] CRAN (R 4.1.0)                     
   prettyunits              1.1.1      2020-01-24 [1] CRAN (R 4.1.0)                     
   processx                 3.5.2      2021-04-30 [1] CRAN (R 4.1.0)                     
   promises                 1.2.0.1    2021-02-11 [1] CRAN (R 4.1.0)                     
   pryr                   * 0.1.4      2018-02-18 [1] CRAN (R 4.1.0)                     
   ps                       1.6.0      2021-02-28 [1] CRAN (R 4.1.0)                     
   purrr                    0.3.4      2020-04-17 [1] CRAN (R 4.1.0)                     
   R.methodsS3              1.8.1      2020-08-26 [1] CRAN (R 4.1.0)                     
   R.oo                     1.24.0     2020-08-26 [1] CRAN (R 4.1.0)                     
   R.utils                  2.10.1     2020-08-26 [1] CRAN (R 4.1.0)                     
   R6                       2.5.0      2020-10-28 [1] CRAN (R 4.1.0)                     
   rappdirs                 0.3.3      2021-01-31 [1] CRAN (R 4.1.0)                     
   RColorBrewer             1.1-2      2014-12-07 [1] CRAN (R 4.1.0)                     
   Rcpp                     1.0.6      2021-01-15 [1] CRAN (R 4.1.0)                     
   RCurl                    1.98-1.3   2021-03-16 [1] CRAN (R 4.1.0)                     
   remotes                  2.3.0      2021-04-01 [1] CRAN (R 4.1.0)                     
   reprex                 * 2.0.0      2021-04-02 [1] CRAN (R 4.1.0)                     
   restfulr                 0.0.13     2017-08-06 [1] CRAN (R 4.1.0)                     
   rhdf5                    2.35.2     2021-03-08 [1] Bioconductor                       
 D rhdf5filters             1.3.5      2021-04-30 [1] Bioconductor                       
   Rhdf5lib                 1.13.6     2021-05-04 [1] Bioconductor                       
   rjson                    0.2.20     2018-06-08 [1] CRAN (R 4.1.0)                     
   rlang                    0.4.11     2021-04-30 [1] CRAN (R 4.1.0)                     
   rmarkdown                2.8        2021-05-07 [1] CRAN (R 4.1.0)                     
   roxygen2                 7.1.1      2020-06-27 [1] CRAN (R 4.1.0)                     
   rprojroot                2.0.2      2020-11-15 [1] CRAN (R 4.1.0)                     
   Rsamtools                2.7.2      2021-04-13 [1] Bioconductor                       
   RSQLite                  2.2.7      2021-04-22 [1] CRAN (R 4.1.0)                     
   rsthemes                 0.2.1.9000 2021-04-22 [1] Github (gadenbuie/rsthemes@19299e5)
   rstudioapi               0.13       2020-11-12 [1] CRAN (R 4.1.0)                     
   rsvd                     1.0.5      2021-04-16 [1] CRAN (R 4.1.0)                     
   rtracklayer            * 1.51.5     2021-05-04 [1] Bioconductor                       
   S4Vectors              * 0.29.19    2021-05-06 [1] Bioconductor                       
   sass                     0.3.1      2021-01-24 [1] CRAN (R 4.1.0)                     
   ScaledMatrix             0.99.2     2021-03-08 [1] Bioconductor                       
   scales                   1.1.1      2020-05-11 [1] CRAN (R 4.1.0)                     
   scater                   1.19.11    2021-03-08 [1] Bioconductor                       
   scatterplot3d            0.3-41     2018-03-14 [1] CRAN (R 4.1.0)                     
   scuttle                  1.1.18     2021-03-08 [1] Bioconductor                       
   sessioninfo              1.1.1      2018-11-05 [1] CRAN (R 4.1.0)                     
   shiny                    1.6.0      2021-01-25 [1] CRAN (R 4.1.0)                     
   shinyWidgets             0.6.0      2021-03-15 [1] CRAN (R 4.1.0)                     
   SingleCellExperiment   * 1.13.14    2021-04-13 [1] Bioconductor                       
   spam                     2.6-0      2020-12-14 [1] CRAN (R 4.1.0)                     
   sparseMatrixStats        1.3.8      2021-04-28 [1] Bioconductor                       
   SpatialExperiment      * 1.1.700    2021-05-05 [1] Bioconductor                       
 R spatialLIBD            * 1.3.18     <NA>       [?] <NA>                               
   stringi                  1.5.3      2020-09-09 [1] CRAN (R 4.1.0)                     
   stringr                  1.4.0      2019-02-10 [1] CRAN (R 4.1.0)                     
   styler                   1.4.1      2021-03-30 [1] CRAN (R 4.1.0)                     
   SummarizedExperiment   * 1.21.3     2021-04-13 [1] Bioconductor                       
   suncalc                  0.5.0      2019-04-03 [1] CRAN (R 4.1.0)                     
   testthat               * 3.0.2      2021-02-14 [1] CRAN (R 4.1.0)                     
   tibble                   3.1.1      2021-04-18 [1] CRAN (R 4.1.0)                     
   tidyr                    1.1.3      2021-03-03 [1] CRAN (R 4.1.0)                     
   tidyselect               1.1.1      2021-04-30 [1] CRAN (R 4.1.0)                     
   usethis                * 2.0.1      2021-02-10 [1] CRAN (R 4.1.0)                     
   utf8                     1.2.1      2021-03-12 [1] CRAN (R 4.1.0)                     
   vctrs                    0.3.8      2021-04-29 [1] CRAN (R 4.1.0)                     
   vipor                    0.4.5      2017-03-22 [1] CRAN (R 4.1.0)                     
   viridis                  0.6.0      2021-04-15 [1] CRAN (R 4.1.0)                     
   viridisLite              0.4.0      2021-04-13 [1] CRAN (R 4.1.0)                     
   withr                    2.4.2      2021-04-18 [1] CRAN (R 4.1.0)                     
   xfun                     0.22       2021-03-11 [1] CRAN (R 4.1.0)                     
   XML                      3.99-0.6   2021-03-16 [1] CRAN (R 4.1.0)                     
   xml2                     1.3.2      2020-04-23 [1] CRAN (R 4.1.0)                     
   xtable                   1.8-4      2019-04-21 [1] CRAN (R 4.1.0)                     
   XVector                  0.31.1     2021-03-08 [1] Bioconductor                       
   yaml                     2.2.1      2020-02-01 [1] CRAN (R 4.1.0)                     
   zlibbioc                 1.37.0     2021-03-09 [1] Bioconductor                       

[1] C:/R/R-4.1.0alpha/library

 D -- DLL MD5 mismatch, broken installation.
 R -- Package was removed from disk.
lcolladotor commented 3 years ago

Resolved in #65.

lcolladotor commented 3 years ago

I was telling Lukas that it's weird that https://github.com/drighelli/SpatialExperiment/blob/b6d1f6cdc20a50bf95c001ee64ceebc5521c7c02/tests/testthat/test_read10xVisium.R#L53-L59 didn't fail on BBS winOS. But as we can see above,

imgData(spe)
#> DataFrame with 0 rows and 0 columns

So it might actually give you an object of the class you want, but not of the length you want (aka a class with NULL data).