search

davebraze / FDBeye

R tools for eyetracker workflows.
https://davebraze.github.io/FDBeye/
Other
118 stars 14 forks source link

Serialize ELascii objects #19

Open davebraze opened 8 years ago

davebraze commented 8 years ago

Look into schemes for serializing ELascii objects (readELascii.R). Consider packages jsonlite or XML. Just now, I'm leaning toward the former.

davebraze commented 8 years ago

I've been rethinking this. Rather than go for a text serialization I think it would be better to just build thin wrappers for reading and writing ELascii objects to/from rdata files. After all, the point is to provide efficient read/write to file, not inter-operability as such.

davebraze commented 6 years ago

Also consider the feather format, implemented in the eponymous R package: https://cran.r-project.org/web/packages/feather/index.html. I think, that feather would be preferable to the HDF5 format, although should probably think about this a bit more.

There is some discussion of relative merits here: https://news.ycombinator.com/item?id=11384577.

One concern about feather is whether it will have staying power, or end up abandonware. HDF5, at least, has been around for a long time and is widely used. Note there are two R packages for interacting with HDF5 files: H5, and rhdf5, the latter on bioconductor. An older package, HDF5, has been removed from CRAN and seems deprecated.

Still shouldn't rule out json or XML.

davebraze commented 6 years ago

Here, https://github.com/wesm/feather/issues/318, it says of feather that "you shouldn't rely on it for long-term storage, at least for now." That pretty much rules out use of feather for my purposes.

In that same place "Parquet files" are recommended for safe long term storage., cf https://rud.is/b/2017/01/22/create-parquet-files-from-r-data-frames-with-sergeant-apache-drill-a-k-a-make-parquet-files-great-again-in-r/