search

pola-rs / r-polars

Polars R binding
https://pola-rs.github.io/r-polars/
Other
473 stars 35 forks source link

`make docs` compiles the Rust crates twice #613

Closed etiennebacher closed 10 months ago

etiennebacher commented 10 months ago

When I run make docs, the makefile calls build and then install:

https://github.com/pola-rs/r-polars/blob/d6cc4dfe7e1045d172ae5aee4d11ba8e8f72a8a5/Makefile#L68

This compiles the Rust deps twice, which is a waste of time. Am I the only one with this? Is it possible to fix this?

sorhawell commented 10 months ago

It may be due to a change of working directory as with R CMD check. I can try to see if symlinking will fix it.

sorhawell commented 10 months ago

I think there is a bad habbit in many R build tools like extendr, roxygen2 etc. to automatically and with no opt-out to call devtools::load_all() to ensure package is loaded. This is ok for small packages, but for a mammuth like polars, even the R part slows down the process significantly.

I ran make docs in the Rstudio terminal which has pandocs.

for make docs I count loading polars 4 times in the log. I saw one build of rust code before an error with forkcluster with the future package

(.venv) Srens-MacBook-Pro:r-polars sorenwelling$ make docs
export NOT_CRAN=true \
        && export LIBR_POLARS_BUILD=true \
        && export RPOLARS_FULL_FEATURES=true \
        && Rscript -e 'if (!(require(arrow) && require(nanoarrow) && require(knitr))) warning("could not load arrow/nanoarrow/knitr, ignore changes to nanoarrow.Rd or knit_print.Rd"); rextendr::document()'
Loading required package: arrow

Attaching package: ‘arrow’

The following object is masked from ‘package:utils’:

    timestamp

Loading required package: nanoarrow
Loading required package: knitr
ℹ Generating extendr wrapper functions for package: polars.
✔ Writing 'R/extendr-wrappers.R'
ℹ Updating polars documentation
ℹ Loading polars
export NOT_CRAN=true \
        && export LIBR_POLARS_BUILD=true \
        && export RPOLARS_FULL_FEATURES=true \
        && R CMD INSTALL --no-multiarch --with-keep.source .
* installing to library ‘/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library’
* installing *source* package ‘polars’ ...
** using staged installation

--------------------------- [RUST FOUND] ---------------------------
cargo 1.73.0-nightly (7ac9416d8 2023-07-24)

rustc 1.73.0-nightly (0d95f9132 2023-07-26)
binary: rustc
commit-hash: 0d95f9132909ae7c5f2456748d0ffd1c3ba4a8e8
commit-date: 2023-07-26
host: x86_64-apple-darwin
release: 1.73.0-nightly
LLVM version: 16.0.5
--------------------------------------------------------------------

** libs
using C compiler: ‘Apple clang version 14.0.0 (clang-1400.0.29.202)’
using SDK: ‘’
rm -Rf polars.so /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a entrypoint.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c entrypoint.c -o entrypoint.o
if [ -f "/Users/sorenwelling/Documents/projs/r-polars/src/../tools/libr_polars.a" ]; then \
                mkdir -p "/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release" ; \
                mv "/Users/sorenwelling/Documents/projs/r-polars/src/../tools/libr_polars.a" "/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a" ; \
                exit 0; \
        fi && \
        if [ "true" != "true" ]; then \
                export CARGO_HOME=/Users/sorenwelling/Documents/projs/r-polars/src/.cargo; \
        fi && \
        export PATH="/Users/sorenwelling/Documents/projs/r-polars/.venv/bin:/usr/local/opt/python@3.10/libexec/bin:/usr/local/git/bin:/Applications/RStudio.app/Contents/Resources/app/resources/terminal/bash/.local/state/nix/profile/bin:/nix/var/nix/profiles/default/bin:/usr/local/opt/python@3.9/libexec/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/sorenwelling/.cargo/bin/:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/sorenwelling/Documents/projs/r-polars/.venv/bin:/usr/local/opt/python@3.10/libexec/bin:/usr/local/git/bin:/Applications/RStudio.app/Contents/Resources/app/resources/terminal/bash/.local/state/nix/profile/bin:/nix/var/nix/profiles/default/bin:/Users/sorenwelling/.cargo/bin:/usr/ucb:/Applications/Docker.app/Contents/Resources/bin/:/Users/sorenwelling/Applications/quarto/bin:/usr/texbin:/Applications/RStudio.app/Contents/Resources/app/quarto/bin:/Applications/RStudio.app/Contents/Resources/app/bin/postback:/usr/local/gfortran/bin:/usr/local/gfortran/bin:/Users/sorenwelling/.cargo/bin" && \
        if [ "true" = "true" ]; then \
                cargo build --target=x86_64-apple-darwin --lib --profile release --manifest-path="./rust/Cargo.toml" --features "full_features"; \
        else \
                cargo build --target=x86_64-apple-darwin --lib --profile release --manifest-path="./rust/Cargo.toml"; \
        fi
    Finished release [optimized] target(s) in 0.35s
if [ "true" != "true" ]; then \
                rm -Rf /Users/sorenwelling/Documents/projs/r-polars/src/.cargo && \
                rm -Rf /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/build; \
        fi
if [ -f "/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a" ]; then \
                echo "file is there: "; \
        elif [ -f "./rust/target/x86_64-apple-darwin/release/libr_polars.a" ]; then \
                echo "file is './rust/target/x86_64-apple-darwin/release/libr_polars.a'"; \
                mkdir -p /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release ; \
                echo "trying to symlink in ./rust/target/x86_64-apple-darwin/release/libr_polars.a"; \
                ln -s ./rust/target/x86_64-apple-darwin/release/libr_polars.a /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a ; \
        fi
file is there: 
if [ "" = "true" ]; then \
                echo "cleanup!!" ; \
                mv /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/../temp_binary.a; \
                rm -rf /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release; \
                mkdir /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release; \
                mv /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/../temp_binary.a /Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a; \
                rm -rf ./src/.cargo; \
        else \
                echo "hands off!!" ; \
        fi
hands off!!
clang -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib -o polars.so entrypoint.o -L/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release -lr_polars -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(adler32.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(adler32_fold.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(chunkset.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(compare256.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(compress.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(cpu_features.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(crc32_braid.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(crc32_braid_comb.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(crc32_fold.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate_fast.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate_huff.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate_medium.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate_quick.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate_rle.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate_slow.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(deflate_stored.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(functable.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(infback.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(inflate.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(inftrees.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(insert_string.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(insert_string_roll.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(slide_hash.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(trees.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(uncompr.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(zutil.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(x86_features.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(slide_hash_avx2.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(chunkset_avx2.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(compare256_avx2.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(adler32_avx2.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(adler32_avx512.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(adler32_avx512_vnni.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(adler32_sse42.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(insert_string_sse42.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(chunkset_sse2.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(compare256_sse2.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(slide_hash_sse2.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(adler32_ssse3.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(chunkset_ssse3.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(crc32_pclmulqdq.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(crc32_vpclmulqdq.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(gzlib.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(gzread.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: object file (/Users/sorenwelling/Documents/projs/r-polars/src/rust/target/x86_64-apple-darwin/release/libr_polars.a(gzwrite.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
installing to /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/00LOCK-r-polars/00new/polars/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (polars)
Rscript -e 'devtools::load_all(); rmarkdown::render("README.Rmd")'
ℹ Loading polars

processing file: README.Rmd

output file: README.knit.md

/Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/pandoc +RTS -K512m -RTS README.knit.md --to gfm+tex_math_dollars-yaml_metadata_block --from markdown+autolink_bare_uris+tex_math_single_backslash --output README.md --template /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/rmarkdown/rmarkdown/templates/github_document/resources/default.md 

Output created: README.md
Rscript -e 'devtools::load_all(); rmarkdown::render("altdoc/reference_home.Rmd")'
ℹ Loading polars

processing file: reference_home.Rmd

output file: reference_home.knit.md

/Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/pandoc +RTS -K512m -RTS reference_home.knit.md --to gfm+tex_math_dollars-yaml_metadata_block --from markdown+autolink_bare_uris+tex_math_single_backslash --output reference_home.md --template /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/rmarkdown/rmarkdown/templates/github_document/resources/default.md 

Output created: reference_home.md
Rscript -e 'future::plan(future::multicore); source("altdoc/altdoc_preprocessing.R"); altdoc::render_docs(freeze = FALSE, parallel = TRUE)'
here() starts at /Users/sorenwelling/Documents/projs/r-polars
ℹ Loading polars

── Basic files ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ NEWS imported.
✔ LICENSE imported.
✔ README imported.

── Man pages ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
ℹ Found 577 man pages to convert.
objc[50388]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[50388]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[50386]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[50386]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[50390]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[50390]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
Error: Failed to retrieve the result of MulticoreFuture (future_sapply-2) from the forked worker (on localhost; PID 50386). Post-mortem diagnostic: No process exists with this PID, i.e. the forked localhost worker is no longer alive. The total size of the 10 globals exported is 597.23 KiB. The three largest globals are ‘.rd2qmd’ (227.27 KiB of class ‘function’), ‘...future.FUN’ (166.24 KiB of class ‘function’) and ‘.qmd2md’ (80.45 KiB of class ‘function’)
In addition: Warning message:
In mccollect(jobs = jobs, wait = TRUE) :
  1 parallel job did not deliver a result
Execution halted
make: *** [docs] Error 1
sorhawell commented 10 months ago

On a sidenote, in my opinion the future package has a poorly optimized inter-process communication relying literally on reading/writing .rds/.Rdata-files () to disk. It would help the R community, if someone wrote an implementation relying on process shared memory, instead of disk. We have that in polars via rust, but it should probably be written in C to be adopted in parallel package.

I would personally only pick future if:

Otherwise parallel::clusterApplyLB is a pretty good alternative when each job may very in length and parallel is a core package.

sorhawell commented 10 months ago

I tried to call with parallel = FALSE 2-5 seconds per documents seems like a lot of time. Is a new R process being spawned for each document? 

> altdoc::render_docs(freeze = FALSE, parallel = FALSE)

── Basic files ──────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ NEWS imported.
✔ LICENSE imported.
✔ README imported.

── Man pages ────────────────────────────────────────────────────────────────────────────────────────────────────────────
ℹ Found 577 man pages to convert.
✔ Converting function reference 1/577: as_polars_df [4.2s]
✔ Converting function reference 2/577: as_polars_lf [3.5s]
✔ Converting function reference 3/577: as_polars_series [2.9s]
✔ Converting function reference 4/577: DataFrame_class [2.5s]
✔ Converting function reference 5/577: DataFrame_clone [3.1s]
✔ Converting function reference 6/577: DataFrame_columns [2.8s]
✔ Converting function reference 7/577: DataFrame_describe [2.5s]
✔ Converting function reference 8/577: DataFrame_drop_in_place [2.9s]
✔ Converting function reference 9/577: DataFrame_drop_nulls [2.6s]
✔ Converting function reference 10/577: DataFrame_drop [2.7s]
✔ Converting function reference 11/577: DataFrame_dtype_strings [2.7s]
etiennebacher commented 10 months ago

altdoc uses quarto to render the man pages and the vignettes, and it calls one quarto process per file. This is why it takes a few seconds for each file. I'm not happy with this but for now there's no native way in quarto to parallelize rendering of several files so using future seemed logical.

Re your point on future, most packages don't have to render more than 100 man pages, polars is the exception with 500+ man pages. You might be interested in mirai.

Finally I notice that even when you recompile the package, you don't need to download or compile Rust crates. When I switch branches, it often has to download and recompile all polars crates (polars-compute, etc.). Currently, this recompilation happens twice with make docs. devtools::load_all() is indeed called several times but only takes a few seconds each to it's lower priority.

sorhawell commented 10 months ago

oh yes I see now it the call to quarto_render which is slow, which executes quarto_bin which spawns a new process which likely spawn another new R process. Yeah then quarto is completely the bottleneck. I'm a bit surprised quarto is ok with this.

You might be interested in mirai.

ohh mirai that looks exciting ! :)

When I switch branches, it often has to download and recompile all polars crates

I was very tired of this too. I wrote load_polars(), build_polars(), check_polars(). Do they work for you? Only when we change rust dependencies should some crates be rebuild. When we bump rust-polars, most crate in practices will be bumped and rebuilt too.

etiennebacher commented 10 months ago

Do they work for you?

No idea, before I used rextendr::document() or devtools::load_all() but that wasn't good, and now I use the makefile.

Yeah then quarto is completely the bottleneck.

https://github.com/etiennebacher/altdoc/issues/242

sorhawell commented 10 months ago

To be sure extendr-wrappers are updated we need to run rextendr::document() I run it via load_polars().

When using multi process rendering the polars -package must also be build, because other processes will likely load the last built package from library.

After this no more load_all() should be needed anywhere and perhaps could be disabled.

unfortunately R devtools/roxygen/extendr just love calling load_all everywhere :/

sorhawell commented 10 months ago

@etiennebacher can you write the exact sequence of calls you make and from where? Slow development is a kill joy.

load_polars / check_polars sets the following envvars for me

RPOLARS_RUST_SOURCE = paste0(getwd(), "/src/rust") RPOLARS_FULL_FEATURES = "true" NOT_CRAN = "true" RPOLARS_CARGO_CLEAN_DEPS = "false" RPOLARS_PROFILE = "release"

RPOLARS_RUST_SOURCE can be used when building from some other path where the normal rust polars binary is not there, and the rust cache is not accessible.

RPOLARS_CARGO_CLEAN_DEPS = "false" can protect the already built binary from being deleted. A lot of these inconveniences is finally due to CRAN being terrible for developers.

etiennebacher commented 10 months ago

can you write the exact sequence of calls you make and from where?

Now I just do make build in the terminal so that it sets the correct envvars. I didn't know whether your scripts were still active or if you switched to using the makefile too. It would be nice to harmonize our dev process, that would also be easier for potential new contributors to have a clear document about how to compile

eitsupi commented 10 months ago

RPOLARS_RUST_SOURCE can be used when building from some other path where the normal rust polars binary is not there, and the rust cache is not accessible.

I recommend that you don't use those old hacks. Specifying the binary path with LIBR_POLARS_PATH is much more versatile. https://github.com/pola-rs/r-polars/blob/d6cc4dfe7e1045d172ae5aee4d11ba8e8f72a8a5/.github/workflows/release-lib.yaml#L148

eitsupi commented 10 months ago

And I think these are not CRAN issues, but simply that Rust's cache doesn't work because R CMD check tries to build from source multiple times.

sorhawell commented 10 months ago

@eitsupi

And I think these are not CRAN issues, but simply that Rust's cache doesn't work because R CMD check tries to build from source multiple times.

With check_polars() I reuse cache every time with no mental effort as a one liner. I can warmly recommend it.

The specific CRAN issues was that we had a loaded footgun that deleted the rust target dir and prebuilt binary if an envvar is set wrong. That is 20 min compile time lost on my slow machine. That could happen to me 3 times a day. There I appreciate to run build commands where all environment variables are preset the right way every time as defaulted function arguments.

etiennebacher commented 10 months ago

This strayed a bit from the original issue, but for information I don't have this "double-compilation" problem when I use Task instead of Make so this can be closed on my side.

eitsupi commented 10 months ago

Glad to hear that. I will remove the Makefile #556