Figure out relationship between rextendr and cargo R packages

[clauswilke]

There is an R package {cargo} (now on CRAN, GitHub repo here) that tackles some of the same issues as rextendr. For example, it defined rust_fn() which takes Rust code, compiles it, and makes it available to R.

We should have a community conversation to see if we can come up with a clear separation of responsibilities between the two packages, so we're not duplicating effort. The R-Rust community is still very small and there's a million things to be done and so we're all going to be better off if everybody is working on their own specific slice of the overall space. Having said that, I realize that sometimes there are competing technical approaches to the same problem and it certainly is useful to explore those and have more than one way of doing something.

Both {cargo} and {rextendr} are MIT-licensed so there's no problem with moving code back and forth if that's helpful.

Tagging the cargo maintainer @dbdahl.

[dbdahl]

Thanks @clauswilke for mentioning my cargo package. I have been interested in extendr project and admire the community you have built. I hope that both of our projects encourage more interest in using Rust among the R community and perhaps these will lower some of the barriers.

For those interested in learning more about my project, please see the paper Writing R Extensions in Rust or see the README.md at the GitHub repository. It has been working well for me on R packages already on CRAN (namely salso and caviarpd) and for some R packages still in development, so I wanted to share it with the R community.

I don't have the bandwidth right now to hash out a separation or integration, but I'm open in principle to collaborations. For the record, I am particularly interested in supporting the development of R packages using Rust (as opposed to, say, embedding R in Rust programs). Here are some of the things that I don't believe rextendr has right now that I believe could be helpful for package developers: setup_rust, cross_compile, target, and run. Also, package developers can copy the cargo.R script to their packages so R can find and run the Cargo package manager according to CRAN policies by, for example, using no more than two CPU threads and downloading dependencies to a temporary directory. As @clauswilke points out, code sharing should be easy since the rextendr and cargo packages share the MIT license.

[clauswilke]

@dbdahl Just a few quick thoughts: First, we need to distinguish between extendr and rextendr. extendr is a Rust crate that connects R and Rust, and allows you to call Rust from R and/or R from Rust. It does all the heavy lifting. rextendr is an R package that provides a few helper functions that make it easier to work with the extendr crate. You don't need rextendr to use extendr, and any R packages written with rextendr will not depend on that package when published. This is different from, say, Rcpp, which is a required dependency if you want to write C++ code using it. I feel your paper muddles these issues at places.

With that out of the way, a logical separation would be:

• cargo: Anything Rust related that doesn't depend on the extendr crate or at least isn't specific to it.
• rextendr: Any additional functionality that uses extendr.

Not sure what this means in practice, but roughly this feels appropriate to me.

[dbdahl]

@clauswilke Thanks for noting the distinctions between extendr and rextendr and that packages developed with rextendr don't depend on it. (That last point is true of the cargo package too, by the way.) I will be more careful when I revise the paper. Thanks!

And, yes, that separation might work well. I am not super familiar with developing Rust-based packages with extendr, but I am glad to work on common Rust-related functionality that could benefit both those developing using extendr or those developing with our own framework.

[Ilia-Kosenkov]

Looking at the public API of cargo, I can pinpoint the following things that both packages try to achieve:

• cargo::new_package() + cargo::register_calls() vs rextendr::use_extendr() + rextendr::register_extendr(): obviously different as different Rust frameworks are used
• cargo::rust_fn() vs rextendr::rust_source() and friends: the same reason as above

Things that cargo has that we do not

• cargo::setup_rust(): installing rust correctly is tricky, so we make no attempts to do so
• cargo::target(): we do cross-arch-compile, but within one OS, so targets are determined by the environment in which R runs
• cargo::cross_compile(): same reason, while a user can change the toolchain, the targets are pre-defined by the environment
• cargo::run() : runs cargo with configs appropriate for CRAN.

The last item on the list is very interesting. We have our own cargo-running code, but right now it is very limited. I promised to provide a better implementation by utilizing {processx}, and I have a working prototype, but I never merged it to master (see open issues in this repository). This is a topic for potential collaboration (imo).

A package named {cargo} should definitely be an entry-point to Rust infrastructure and provide necessary tools to work with Rust. However, it is not yet clear to me if it is possible to achieve as different frameworks (such as extendr) may have different requirements. If cargo::run() is flexible, I see no problem in introducing a dependency and offloading all calls to cargo to this function, or borrowing/sharing codebase.