Add arm64 cross compilation github release

[sorhawell]

Close #4 Close #371

Add source packages containing arm64 Linux or arm64 macOS Rust binary to the GitHub release.

[sorhawell]

test release here https://github.com/pola-rs/r-polars/releases/tag/pr_377_test_release

[sorhawell]

I can split it into a chore PR. It is just from running fmt document on main as it is now.

[sorhawell]

latest test release https://github.com/pola-rs/r-polars/releases/tag/pr_377_test_release_2

[sorhawell]

added full_features = true + remove hardcoded choices from release.yaml

https://github.com/pola-rs/r-polars/releases/tag/pr_377_test_release_4

[eitsupi]

@sorhawell It is not clear to me what this process is doing. Does this exist for Swatinem/rust-cache? https://github.com/pola-rs/r-polars/blob/ebf5fc32d26e1bfef9ac07e4ae24e83e3792e6e4/src/Makevars#L31-L38

If so, it would be better to use the workspace option of Swatinem/rust-cache.

[sorhawell]

@sorhawell It is not clear to me what this process is doing. Does this exist for Swatinem/rust-cache?

 if [ -f "$(STATLIB)" ]; then \ 

It mainly needed for speeding local R cmd check I think.

In the case of performing a R cmd check locally. The binaries may already be compiled or nearly compiled. No need to recompile rust-polars for every check, cargo cache can decide if needed. Cargo was called just above these lines but is compiling/caching in the original source folder not in a temp R cmd check folder. RPOLARS_RUST_SOURCE and rpolars_ext_binary points back to the original target folder. If no binary is found, even though cargo should just have produced a binary, then a symlink is created to original folder.

It speeds up a R cmd check locally from 20min to about 3min for my mediocre MacBook.

I run via inst/misc/develop_polars.R : check_polars() which also sets environment variables right and automatically point to the right original compilation.

If working with multiple forks and/or checked out repos, it can also be used to share a single cache to save diskspace and time.

[sorhawell]

@eitsupi many thx for taking a look at this! I have travel for some weeks. I will try to follow up whenever I can.

[eitsupi]

It seems working now. (Not tested yet)

I also want to add linux arm64 build.

[sorhawell]

Not sure about the linux target error. Does libR-sys support these targets?

[eitsupi]

Not sure about the linux target error. Does libR-sys support these targets?

Since cross builds inside a Linux container, I think it failed because there is no R inside the container.

[eitsupi]

extendr/extendr#566 seems causing this error.

 error[E0308]: mismatched types
   --> /home/runner/.cargo/git/checkouts/extendr-f64cf2fdda5197e8/ee8d375/extendr-api/src/lib.rs:630:17
    |
630 |         Rprintf(PRINTF_NO_FMT_CSTRING.as_ptr(), cs.as_ptr());
    |         ------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `*const u8`, found `*const i8`
    |         |
    |         arguments to this function are incorrect
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`
note: function defined here
   --> /home/runner/work/r-polars/r-polars/src/rust/target/aarch64-unknown-linux-gnu/release-optimized/build/libR-sys-d7c4c89c6e4571b8/out/bindings.rs:876:12
    |
876 |     pub fn Rprintf(arg1: *const ::std::os::raw::c_char, ...);
    |            ^^^^^^^

error[E0308]: mismatched types
   --> /home/runner/.cargo/git/checkouts/extendr-f64cf2fdda5197e8/ee8d375/extendr-api/src/lib.rs:638:18
    |
638 |         REprintf(PRINTF_NO_FMT_CSTRING.as_ptr(), cs.as_ptr());
    |         -------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `*const u8`, found `*const i8`
    |         |
    |         arguments to this function are incorrect
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`
note: function defined here
   --> /home/runner/work/r-polars/r-polars/src/rust/target/aarch64-unknown-linux-gnu/release-optimized/build/libR-sys-d7c4c89c6e4571b8/out/bindings.rs:879:12
    |
879 |     pub fn REprintf(arg1: *const ::std::os::raw::c_char, ...);
    |            ^^^^^^^^

See remacs/remacs#1393

[sorhawell]

Nice swift fix to print extendr char/u8 !!!

[eitsupi]

It seems working fine on my Raspberry Pi4! (just run rocker/r-ver:latest)

R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: aarch64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> install.packages("https://github.com/pola-rs/r-polars/releases/download/pr_377_test_release_5/polars_cross_aarch64-unknown-linux-gnu.tar.gz", repos = NULL)
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://github.com/pola-rs/r-polars/releases/download/pr_377_test_release_5/polars_cross_aarch64-unknown-linux-gnu.tar.gz'
Content type 'application/octet-stream' length 31324944 bytes (29.9 MB)
==================================================
downloaded 29.9 MB

* installing *source* package ‘polars’ ...
** using staged installation
** libs
using C compiler: ‘gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0’
rm -Rf polars.so ./rust/target/release/libr_polars.a entrypoint.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include    -fPIC  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c entrypoint.c -o entrypoint.o
if [ -f "/tmp/RtmpPqR1Kl/R.INSTALL18609e7909/polars/src/../inst/libr_polars.a" ]; then \
        echo "refer directly to a precompiled object file + skip cargo build"; \
        mkdir -p ./rust/target/release ; \
        mv /tmp/RtmpPqR1Kl/R.INSTALL18609e7909/polars/src/../inst/libr_polars.a ./rust/target/release/libr_polars.a ; \
        exit 0 ; \
fi && \
if [ "true" != "true" ]; then \
        export CARGO_HOME=/tmp/RtmpPqR1Kl/R.INSTALL18609e7909/polars/src/.cargo; \
fi && \
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.cargo/bin" && \
if [ "" == "true" ]; then \
        cargo build --lib --profile release --manifest-path="./rust/Cargo.toml" --features "full_features"; \
else \
        cargo build --lib --profile release --manifest-path="./rust/Cargo.toml"; \
fi
refer directly to a precompiled object file + skip cargo build
if [ "true" != "true" ]; then \
        rm -Rf /tmp/RtmpPqR1Kl/R.INSTALL18609e7909/polars/src/.cargo && \
        rm -Rf ./rust/target/release/build; \
fi
if [ -f "./rust/target/release/libr_polars.a" ]; then \
        echo "file is there: "; \
elif [ -f ""./rust/target/release/libr_polars.a"" ]; then \
        echo "file is '"./rust/target/release/libr_polars.a"'"; \
        mkdir -p ./rust/target/release ; \
        echo "trying to symlink in "./rust/target/release/libr_polars.a""; \
        ln -s "./rust/target/release/libr_polars.a" ./rust/target/release/libr_polars.a ; \
fi
file is there:
if [ "" == "true" ]; then \
        echo "cleanup!!" ; \
        mv ./rust/target/release/libr_polars.a ./rust/target/release/../temp_binary.a; \
        rm -rf ./rust/target/release; \
        mkdir ./rust/target/release; \
        mv ./rust/target/release/../temp_binary.a ./rust/target/release/libr_polars.a; \
        rm -rf ./src/.cargo; \
else \
        echo "hands off!!" ; \
fi
hands off!!
gcc -shared -L/usr/local/lib/R/lib -L/usr/local/lib -o polars.so entrypoint.o -L./rust/target/release -lr_polars -L/usr/local/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-polars/00new/polars/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (polars)
> polars::pl$DataFrame(mtcars)
shape: (32, 11)
┌──────┬─────┬───────┬───────┬───┬─────┬─────┬──────┬──────┐
│ mpg  ┆ cyl ┆ disp  ┆ hp    ┆ … ┆ vs  ┆ am  ┆ gear ┆ carb │
│ ---  ┆ --- ┆ ---   ┆ ---   ┆   ┆ --- ┆ --- ┆ ---  ┆ ---  │
│ f64  ┆ f64 ┆ f64   ┆ f64   ┆   ┆ f64 ┆ f64 ┆ f64  ┆ f64  │
╞══════╪═════╪═══════╪═══════╪═══╪═════╪═════╪══════╪══════╡
│ 21.0 ┆ 6.0 ┆ 160.0 ┆ 110.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0  ┆ 4.0  │
│ 21.0 ┆ 6.0 ┆ 160.0 ┆ 110.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0  ┆ 4.0  │
│ 22.8 ┆ 4.0 ┆ 108.0 ┆ 93.0  ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0  ┆ 1.0  │
│ 21.4 ┆ 6.0 ┆ 258.0 ┆ 110.0 ┆ … ┆ 1.0 ┆ 0.0 ┆ 3.0  ┆ 1.0  │
│ …    ┆ …   ┆ …     ┆ …     ┆ … ┆ …   ┆ …   ┆ …    ┆ …    │
│ 15.8 ┆ 8.0 ┆ 351.0 ┆ 264.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 4.0  │
│ 19.7 ┆ 6.0 ┆ 145.0 ┆ 175.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 6.0  │
│ 15.0 ┆ 8.0 ┆ 301.0 ┆ 335.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0  ┆ 8.0  │
│ 21.4 ┆ 4.0 ┆ 121.0 ┆ 109.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0  ┆ 2.0  │
└──────┴─────┴───────┴───────┴───┴─────┴─────┴──────┴──────┘

[eitsupi]

I think we should use ubuntu-20.04 to build binaries for Linux to fix #86. (I recently saw this problem in other repositories. gnu targets are difficult.)

[sorhawell]

haha nice way of testing with RaspberryPi ^^

I think we should use ubuntu-20.04 to build binaries for Linux to fix https://github.com/pola-rs/r-polars/issues/86.

Sounds like a good idea. The full implications are likely hard to test now. We could either go all-in and see if any issues reported. If releasing some "legacy-support" release first I suspect, there will probably be little feedback.

[eitsupi]

For example, the compatibility is improved here by changing the runner for the build from Ubuntu 22.04 to Ubuntu 20.04. rust-lang/mdBook#1954

(Although for normal applications, choosing the musl target instead of the gnu target is definitely superior)

[eitsupi]

ubuntu-20.04 does not seem working... I have reverted that.

[eitsupi]

I think we can merge now.