file reading and management

[franknarf1]

Reading files

• fread("unzip -p file.zip subfile.csv") or fread("tar -zxvf multiple_files.csv.gz wanted_file.csv"). thanks to xz, Axeman https://chat.stackoverflow.com/transcript/message/45164278#45164278

[franknarf1]

Maybe also cover reading files in other formats (json, matrix)... though maybe that's broad enough for a separate set of notes.

• matrix stored as csv: https://stackoverflow.com/a/55165750 or https://stackoverflow.com/a/14594999 or https://stat.ethz.ch/pipermail/r-help/2006-June/108226.html

[dracodoc]

Thanks for the great tutorial. I consider myself an advanced data.table user but still learned some new stuff.

Two suggestions on the fread part:

• fread support reading .gz directly

?fread Compressed files with extension .gz and .bz2 are supported if the R.utils package is installed.

• I used to use rbindlist a lot, later I found in many cases I can combine tables in one line and get a proper result. for example

DT = fileDT[, rbindlist(setNames(contents, id), idcol="file_id")]

can be

fileDT[, fread(fn), by = .(year, id)]

and the result will be all table combined together with year, id column.