fstpackage / fst

Lightning Fast Serialization of Data Frames for R
http://www.fstpackage.org/fst/
GNU Affero General Public License v3.0
614 stars 42 forks source link

Move `fstlib` bindings into a separate package #240

Closed MarcusKlik closed 4 years ago

MarcusKlik commented 4 years ago

I'm thinking of splitting the fst package in (at first) 2 components:

The reason is that this setup would allow for independent updates to the LZ4, ZSTD and fstlib libraries and to the R code in the fst package. This is important because each CRAN submission is quite involved at the moment due to the C++ components not passing the CRAN (pre-)tests.

For example, the LZ4 and ZSTD components throw a lot of UBSAN / ASAN errors (see here) and get compiler warnings on the (Clang 10) CRAN build infrastructure. It's fantastic for code quality and security that CRAN uses the very latest build tools infrastructure but it can be tricky for packages that pack external components :-). With 2 components we can alternate updates to the fst package and it's backend to streamline the submission process.

The idea is to have a fst package and a fstcore package. The latter takes care of all C++ code and the former can be 95% R code (maybe 100% at some point). Perhaps later, also the LZ4 and ZSTD components could be hosted in a separate package to further decouple the dependencies...

MarcusKlik commented 4 years ago

done, for now the Travis builds are run using fstcore in the Remotes field (from Github). Next step is to publish fstcore to CRAN so that it can be added as a standard dependency.