r-hyperspec / hySpc.read.txt

Import ASCII formatted data into hyperSpec
https://r-hyperspec.github.io/hySpc.read.txt/
MIT License
0 stars 1 forks source link

Shouldn't `read_txt_Shimadzu()` be faster? #60

Open GegznaV opened 2 years ago

GegznaV commented 2 years ago

On my machine, it takes arround 2 minutes to read GCxGC-qMS.txt file in read_txt_Shimadzu() unit test:

 system.time({
     filename <- system.file(
         "extdata",
         "txt.Shimadzu/GCxGC-qMS.txt",
         package = "hySpc.read.txt"
     )

     spc <- read_txt_Shimadzu(filename)
 })
#>   user  system elapsed 
#> 124.50   17.56  143.90 

143.90/60 ≈ 2.398 min

image

It is way too long.

Either the function is too slow or the file is too large. What should we do about this, @sangttruong, @bryanhanson, @cbeleites? Should we skip this test for now?


Related:

GegznaV commented 2 years ago

This is a piece of RSudio profiling information:

image

The bottleneck seems to be here (49.4 is memory and 71540 is time usage):

image

GegznaV commented 2 years ago

image

image

GegznaV commented 2 years ago

The test file is large: image

GegznaV commented 2 years ago

Can we have a smaller test file? @cbeleites