I'm thinking of splitting the fst package in (at first) 2 components:
one component with the 'core' functionality (the R bindings to the fstlib C++ library)
one component with the R API (such as presented in the fst package)
The reason is that this setup would allow for independent updates to the LZ4, ZSTD and fstlib libraries and to the R code in the fst package. This is important because each CRAN submission is quite involved at the moment due to the C++ components not passing the CRAN (pre-)tests.
For example, the LZ4 and ZSTD components throw a lot of UBSAN / ASAN errors (see here) and get compiler warnings on the (Clang 10) CRAN build infrastructure. It's fantastic for code quality and security that CRAN uses the very latest build tools infrastructure but it can be tricky for packages that pack external components :-). With 2 components we can alternate updates to the fst package and it's backend to streamline the submission process.
The idea is to have a fst package and a fstcore package. The latter takes care of all C++ code and the former can be 95% R code (maybe 100% at some point). Perhaps later, also the LZ4 and ZSTD components could be hosted in a separate package to further decouple the dependencies...
done, for now the Travis builds are run using fstcore in the Remotes field (from Github). Next step is to publish fstcore to CRAN so that it can be added as a standard dependency.
I'm thinking of splitting the
fst
package in (at first) 2 components:fstlib
C++ library)R
API (such as presented in thefst
package)The reason is that this setup would allow for independent updates to the
LZ4
,ZSTD
andfstlib
libraries and to theR
code in thefst
package. This is important because each CRAN submission is quite involved at the moment due to the C++ components not passing the CRAN (pre-)tests.For example, the
LZ4
andZSTD
components throw a lot of UBSAN / ASAN errors (see here) and get compiler warnings on the (Clang 10) CRAN build infrastructure. It's fantastic for code quality and security that CRAN uses the very latest build tools infrastructure but it can be tricky for packages that pack external components :-). With 2 components we can alternate updates to thefst
package and it's backend to streamline the submission process.The idea is to have a
fst
package and afstcore
package. The latter takes care of all C++ code and the former can be 95%R
code (maybe 100% at some point). Perhaps later, also theLZ4
andZSTD
components could be hosted in a separate package to further decouple the dependencies...