Closed ghost closed 7 years ago
The ideal solution is also use fread
for zip file. There are several approaches:
data.table
fread(input = 'zcat < data.gz')
. However windows doesn't have zcat
gzip
installed by default. It's difficult to create a simple cross platform solution without needs user to install software first.tar
which cannot be recognized by R internal function unzip
. R.utils
used R connections method to uncompress zip to files, but you still need to identify compression method first.Right now the parameter method is simplest without need of much change to existing code. We can further improve this depend on new usage or development in related packages.
I think the most important thing is that as.telemetry "just work" with default arguments. I put in some code that checks to see if the filename looks like a CSV, then attempts fread. If the filename doesn't look like a CSV or fread fails, then the slower read.table is used instead.
data < NULL
# fread doesn't work on compressed files yet
if(endsWith(tolower(object),".csv"))
{ data <- try(data.table::fread(object,data.table=FALSE,check.names=TRUE,...)) }
# if fread fails, then fall back on read.table
if(class(data)!="data.frame")
{ data <- utils::read.csv(object,...) }
We could add in more logic for different compression formats, but I don't know that the command & pipe notation is the same across platforms.
@chfleming This is a much better solution compared to extra parameter.
I think there is no need to check compression formats since there are many possibilities and platform compatibility problems.
@chfleming I think we can actually just fread
the first 5 rows without the file name check. It's possible the csv file have different file name (I saw .txt before). How about this:
data <- try(data.table::fread(object, data.table = FALSE, check.names = TRUE, nrows = 5),
silent = TRUE)
if (class(data) == "data.frame") {
data <- data.table::fread(object,data.table=FALSE,check.names=TRUE,...)
} else {
data <- utils::read.csv(object,...)
}
I think the direct read test should be fast enough that comparable to the file name check, and it will handle all possible cases without complex logic.
That seems to work well. Pushed.
zipfile = FALSE
forfread
, fall back toread.csv
when zip file is needed.For 160M csv,
fread
took 2.64s whileread.csv
took 21s.