Add `arrow` to the list of packages with non-exportable objects?

[etiennebacher]

Wish or feature request

Hello, I recently ran into an issue about non-exportable external pointers when calling some arrow functions in future_lapply(). I think it would be useful to add this package to the list of packages with non-exportable objects:

library(arrow, warn.conflicts = FALSE)
library(future)

plan(multisession)

### arrow error
x <- future({
  as_arrow_table(iris)
})
value(x)
#> Table
#> Error: Invalid <Table>, external pointer to null

### future error
options(future.globals.onReference = "error")  
x <- future({
  as_arrow_table(iris)
})
value(x)
#> Error: Detected a non-exportable reference ('externalptr') in the value (of class 'Table') of the resolved future

as_arrow_table() creates only a pointer to some data that is not actually present in R. The user can then perform some operations before pulling the data in R. I'm not an arrow expert so here are more docs about this: https://arrow.apache.org/docs/r/articles/data_objects.html

Related to https://github.com/apache/arrow/issues/40231

---

Note that this is also an issue for the polars package: https://github.com/pola-rs/r-polars/issues/851. This package is not on CRAN, so I don't know if you want to add it to the list as well.

Thanks

[eitsupi]

I would say that arrow is covered for developers since it is built on cpp11, but it may not be obvious to users.

[HenrikBengtsson]

Good idea. I'll try to include an example of this.

Note that this is also an issue for the polars package: pola-rs/r-polars#851. This package is not on CRAN, so I don't know if you want to add it to the list as well.

I'm happy to do that; do you a small example for how to do this?

[HenrikBengtsson]

Added arrow example, cf. commit 6c041c2. See also https://future.futureverse.org/articles/future-4-non-exportable-objects.html#package-arrow

[HenrikBengtsson]

Note that this is also an issue for the polars package: pola-rs/r-polars#851. This package is not on CRAN, so I don't know if you want to add it to the list as well.

I'm happy to do that; do you a small example for how to do this?

I see that your https://github.com/pola-rs/r-polars/issues/851 enough for me to work on something.

[HenrikBengtsson]

I've added an example for polars too, cf. https://future.futureverse.org/articles/future-4-non-exportable-objects.html#package-polars

Thanks for the suggestions.