search

openpharma / staged.dependencies

R package to implement development stages for package development
https://openpharma.github.io/staged.dependencies/
MIT License
12 stars 3 forks source link

Any way to install all dependencies in one call, instead of installing them for every package? #180

Closed m7pr closed 1 year ago

m7pr commented 1 year ago

I just run

x <- staged.dependencies::dependency_table(".")
staged.dependencies::install_deps(x, upgrade = "always")

while being in teal repository. It looks like dependencies are getting installed for each package separately causing the installation process to install the same packages multiple times. This extends the time of overall installation.

Maybe there is a way to change the code so that digest/Rcpp gets installed once, and not multiple times like below, e.g. during teal.slice , teal.data, scda.2022, teal.widgets, teal.transform installations.

I haven't checked the code but it looks like we are running lapply(packages, install_) instead of install_(packages)

Installing 6 packages: vctrs, fs, digest, Rcpp, jsonlite, curl

  There is a binary version available but the source version is later:
   binary source needs_compilation
fs  1.6.2  1.6.3              TRUE

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/vctrs_0.6.3.zip'
Content type 'application/zip' length 1333025 bytes (1.3 MB)
downloaded 1.3 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/fs_1.6.2.zip'
Content type 'application/zip' length 391427 bytes (382 KB)
downloaded 382 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/digest_0.6.33.zip'
Content type 'application/zip' length 206255 bytes (201 KB)
downloaded 201 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/Rcpp_1.0.11.zip'
Content type 'application/zip' length 2873706 bytes (2.7 MB)
downloaded 2.7 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/jsonlite_1.8.7.zip'
Content type 'application/zip' length 1105409 bytes (1.1 MB)
downloaded 1.1 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/curl_5.0.1.zip'
Content type 'application/zip' length 2654638 bytes (2.5 MB)
downloaded 2.5 MB

package ‘vctrs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘vctrs’
Warning: restored ‘vctrs’
package ‘fs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘fs’
Warning: restored ‘fs’
package ‘digest’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘digest’
Warning: restored ‘digest’
package ‘Rcpp’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘Rcpp’
Warning: restored ‘Rcpp’
package ‘jsonlite’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘jsonlite’
Warning: restored ‘jsonlite’
package ‘curl’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘curl’
Warning: restored ‘curl’

The downloaded binary packages are in
    C:\Users\kosinsm4\AppData\Local\Temp\RtmpkZngfg\downloaded_packages
* installing *source* package 'teal.widgets' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (teal.widgets)
vctrs    (0.6.2  -> 0.6.3 ) [CRAN]
fs       (1.6.2  -> 1.6.3 ) [CRAN]
digest   (0.6.31 -> 0.6.33) [CRAN]
jsonlite (1.8.4  -> 1.8.7 ) [CRAN]
Rcpp     (1.0.10 -> 1.0.11) [CRAN]
Installing 5 packages: vctrs, fs, digest, jsonlite, Rcpp

  There is a binary version available but the source version is later:
   binary source needs_compilation
fs  1.6.2  1.6.3              TRUE

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/vctrs_0.6.3.zip'
Content type 'application/zip' length 1333025 bytes (1.3 MB)
downloaded 1.3 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/fs_1.6.2.zip'
Content type 'application/zip' length 391427 bytes (382 KB)
downloaded 382 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/digest_0.6.33.zip'
Content type 'application/zip' length 206255 bytes (201 KB)
downloaded 201 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/jsonlite_1.8.7.zip'
Content type 'application/zip' length 1105409 bytes (1.1 MB)
downloaded 1.1 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/Rcpp_1.0.11.zip'
Content type 'application/zip' length 2873706 bytes (2.7 MB)
downloaded 2.7 MB

package ‘vctrs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘vctrs’
Warning: restored ‘vctrs’
package ‘fs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘fs’
Warning: restored ‘fs’
package ‘digest’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘digest’
Warning: restored ‘digest’
package ‘jsonlite’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘jsonlite’
Warning: restored ‘jsonlite’
package ‘Rcpp’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘Rcpp’
Warning: restored ‘Rcpp’

The downloaded binary packages are in
    C:\Users\kosinsm4\AppData\Local\Temp\RtmpkZngfg\downloaded_packages
Skipping installation of C:/Users/kosinsm4/Documents/.staged.dependencies/packages_cache/insightsengineering_teal.logger_db3be93ac2413ef16521b0cf8fe202ad since same commit sha already installed
* installing *source* package 'scda.2022' ...
** using staged installation
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (scda.2022)
Skipping installation of C:/Users/kosinsm4/Documents/.staged.dependencies/packages_cache/insightsengineering_scda_6ea836b807b2d80ac7feb032e00c8aa2 since same commit sha already installed
vctrs    (0.6.2  -> 0.6.3 ) [CRAN]
fs       (1.6.2  -> 1.6.3 ) [CRAN]
digest   (0.6.31 -> 0.6.33) [CRAN]
jsonlite (1.8.4  -> 1.8.7 ) [CRAN]
Rcpp     (1.0.10 -> 1.0.11) [CRAN]
Skipping 2 packages not available: SummarizedExperiment, MultiAssayExperiment
Installing 5 packages: vctrs, fs, digest, jsonlite, Rcpp

  There is a binary version available but the source version is later:
   binary source needs_compilation
fs  1.6.2  1.6.3              TRUE

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/vctrs_0.6.3.zip'
Content type 'application/zip' length 1333025 bytes (1.3 MB)
downloaded 1.3 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/fs_1.6.2.zip'
Content type 'application/zip' length 391427 bytes (382 KB)
downloaded 382 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/digest_0.6.33.zip'
Content type 'application/zip' length 206255 bytes (201 KB)
downloaded 201 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/jsonlite_1.8.7.zip'
Content type 'application/zip' length 1105409 bytes (1.1 MB)
downloaded 1.1 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/Rcpp_1.0.11.zip'
Content type 'application/zip' length 2873706 bytes (2.7 MB)
downloaded 2.7 MB

package ‘vctrs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘vctrs’
Warning: restored ‘vctrs’
package ‘fs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘fs’
Warning: restored ‘fs’
package ‘digest’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘digest’
Warning: restored ‘digest’
package ‘jsonlite’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘jsonlite’
Warning: restored ‘jsonlite’
package ‘Rcpp’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘Rcpp’
Warning: restored ‘Rcpp’

The downloaded binary packages are in
    C:\Users\kosinsm4\AppData\Local\Temp\RtmpkZngfg\downloaded_packages
* installing *source* package 'teal.data' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
Warning message:
package 'shiny' was built under R version 4.3.1 
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Warning: package 'shiny' was built under R version 4.3.1
** testing if installed package can be loaded from final location
Warning: package 'shiny' was built under R version 4.3.1
** testing if installed package keeps a record of temporary installation path
* DONE (teal.data)
vctrs    (0.6.2  -> 0.6.3 ) [CRAN]
fs       (1.6.2  -> 1.6.3 ) [CRAN]
digest   (0.6.31 -> 0.6.33) [CRAN]
jsonlite (1.8.4  -> 1.8.7 ) [CRAN]
Rcpp     (1.0.10 -> 1.0.11) [CRAN]
curl     (5.0.0  -> 5.0.1 ) [CRAN]
Skipping 2 packages not available: SummarizedExperiment, MultiAssayExperiment
Installing 6 packages: vctrs, fs, digest, jsonlite, Rcpp, curl

  There is a binary version available but the source version is later:
   binary source needs_compilation
fs  1.6.2  1.6.3              TRUE

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/vctrs_0.6.3.zip'
Content type 'application/zip' length 1333025 bytes (1.3 MB)
downloaded 1.3 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/fs_1.6.2.zip'
Content type 'application/zip' length 391427 bytes (382 KB)
downloaded 382 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/digest_0.6.33.zip'
Content type 'application/zip' length 206255 bytes (201 KB)
downloaded 201 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/jsonlite_1.8.7.zip'
Content type 'application/zip' length 1105409 bytes (1.1 MB)
downloaded 1.1 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/Rcpp_1.0.11.zip'
Content type 'application/zip' length 2873706 bytes (2.7 MB)
downloaded 2.7 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/curl_5.0.1.zip'
Content type 'application/zip' length 2654638 bytes (2.5 MB)
downloaded 2.5 MB

package ‘vctrs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘vctrs’
Warning: restored ‘vctrs’
package ‘fs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘fs’
Warning: restored ‘fs’
package ‘digest’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘digest’
Warning: restored ‘digest’
package ‘jsonlite’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘jsonlite’
Warning: restored ‘jsonlite’
package ‘Rcpp’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘Rcpp’
Warning: restored ‘Rcpp’
package ‘curl’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘curl’
Warning: restored ‘curl’

The downloaded binary packages are in
    C:\Users\kosinsm4\AppData\Local\Temp\RtmpkZngfg\downloaded_packages
* installing *source* package 'teal.slice' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
Warning message:
package 'shiny' was built under R version 4.3.1 
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Warning: package 'shiny' was built under R version 4.3.1
** testing if installed package can be loaded from final location
Warning: package 'shiny' was built under R version 4.3.1
** testing if installed package keeps a record of temporary installation path
* DONE (teal.slice)
vctrs    (0.6.2  -> 0.6.3 ) [CRAN]
fs       (1.6.2  -> 1.6.3 ) [CRAN]
jsonlite (1.8.4  -> 1.8.7 ) [CRAN]
curl     (5.0.0  -> 5.0.1 ) [CRAN]
digest   (0.6.31 -> 0.6.33) [CRAN]
Rcpp     (1.0.10 -> 1.0.11) [CRAN]
Installing 6 packages: vctrs, fs, jsonlite, curl, digest, Rcpp

  There is a binary version available but the source version is later:
   binary source needs_compilation
fs  1.6.2  1.6.3              TRUE

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/vctrs_0.6.3.zip'
Content type 'application/zip' length 1333025 bytes (1.3 MB)
downloaded 1.3 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/fs_1.6.2.zip'
Content type 'application/zip' length 391427 bytes (382 KB)
downloaded 382 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/jsonlite_1.8.7.zip'
Content type 'application/zip' length 1105409 bytes (1.1 MB)
downloaded 1.1 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/curl_5.0.1.zip'
Content type 'application/zip' length 2654638 bytes (2.5 MB)
downloaded 2.5 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/digest_0.6.33.zip'
Content type 'application/zip' length 206255 bytes (201 KB)
downloaded 201 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/Rcpp_1.0.11.zip'
Content type 'application/zip' length 2873706 bytes (2.7 MB)
downloaded 2.7 MB

package ‘vctrs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘vctrs’
Warning: restored ‘vctrs’
package ‘fs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘fs’
Warning: restored ‘fs’
package ‘jsonlite’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘jsonlite’
Warning: restored ‘jsonlite’
package ‘curl’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘curl’
Warning: restored ‘curl’
package ‘digest’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘digest’
Warning: restored ‘digest’
package ‘Rcpp’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘Rcpp’
Warning: restored ‘Rcpp’

The downloaded binary packages are in
    C:\Users\kosinsm4\AppData\Local\Temp\RtmpkZngfg\downloaded_packages
* installing *source* package 'teal.transform' ...
** using staged installation
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (teal.transform)
vctrs    (0.6.2  -> 0.6.3 ) [CRAN]
fs       (1.6.2  -> 1.6.3 ) [CRAN]
Rcpp     (1.0.10 -> 1.0.11) [CRAN]
digest   (0.6.31 -> 0.6.33) [CRAN]
jsonlite (1.8.4  -> 1.8.7 ) [CRAN]
Installing 5 packages: vctrs, fs, Rcpp, digest, jsonlite

  There is a binary version available but the source version is later:
   binary source needs_compilation
fs  1.6.2  1.6.3              TRUE

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/vctrs_0.6.3.zip'
Content type 'application/zip' length 1333025 bytes (1.3 MB)
downloaded 1.3 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/fs_1.6.2.zip'
Content type 'application/zip' length 391427 bytes (382 KB)
downloaded 382 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/Rcpp_1.0.11.zip'
Content type 'application/zip' length 2873706 bytes (2.7 MB)
downloaded 2.7 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/digest_0.6.33.zip'
Content type 'application/zip' length 206255 bytes (201 KB)
downloaded 201 KB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.3/jsonlite_1.8.7.zip'
Content type 'application/zip' length 1105409 bytes (1.1 MB)
downloaded 1.1 MB

package ‘vctrs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘vctrs’
Warning: restored ‘vctrs’
package ‘fs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘fs’
Warning: restored ‘fs’
package ‘Rcpp’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘Rcpp’
Warning: restored ‘Rcpp’
package ‘digest’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘digest’
Warning: restored ‘digest’
package ‘jsonlite’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘jsonlite’
Warning: restored ‘jsonlite’

The downloaded binary packages are in
    C:\Users\kosinsm4\AppData\Local\Temp\RtmpkZngfg\downloaded_packages
* installing *source* package 'teal.reporter' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
Melkiades commented 1 year ago

I am not sure this is the case. It seems that it is unable to install the newer dependencies: vctrs (0.6.2 -> 0.6.3 ) [CRAN] is not updated: 

package ‘vctrs’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘vctrs’
Warning: restored ‘vctrs’

Hence the problem is not that it reinstalls everything all the time, but more that it is not able to update them on your local machine for some reasons. We need to investigate if this is happening for others (on my ubuntu installation this does not happen)

m7pr commented 1 year ago

@Melkiades still, it shouldn't try do that for every package IMHO. It might not happen on your machine, as you may have the newest versions of those packages : )

Melkiades commented 1 year ago

@Melkiades still, it shouldn't try do that for every package IMHO. It might not happen on your machine, as you may have the newest versions of those packages : )

I think this boils down to stopping installing dependencies in the install.packages call here in git_tools.R: utils::install.packages(repo_dir, repos = NULL, type = "source")

It seems too from the comments here that turning the installation of deps off is not something convenient and is not the default R behavior. I know you suggested trying only once and then stopping it from retrying but it seems really not a problem related to staged.dependencies, but more something that can be solved with renv and so on. I think until someone else has this problem we can close this

m7pr commented 1 year ago

I lived in a world where I assumed you can set upgrade = "never" in install.packages, but it looks like it's a devtools::install argument, nor install.packages.

I am not saying we should defer from regular install.packages() behavior. What I am saying here that this install.packages() gets called multiple times (for every dependent package), instead of once. If it would be called once, then also the upgrade attempt would be called once.

m7pr commented 1 year ago

For the upgrade install_deps has upgrade parameter as it uses remotes::install_deps() for the installation. Can you open back the issue? I think there is a misunderstanding between what I planned to report and what you investigated

Melkiades commented 1 year ago

We can use devtools for installing, that is true. Maybe @pawelru can drive us towards a common solution ^^

Btw I think it is kinda of slow atm, so less deps installations the faster. Still I would add it as an option

m7pr commented 1 year ago

Your comment was missleading and made me think we use install.packages() for the installation. However I checked that we use remotes::install_deps() in staged.dependencies::install_deps().

The primary issue that I was trying to describe is that all dependencies are installed in this loop https://github.com/openpharma/staged.dependencies/blob/fb124997306b35d44a0225bb4b400bf7258c4c75/R/dependencies_helper.R#L63

in run_package_actions where we could combine packages into one vector if the only considered action is the installation so that they get installed with one call.

Anyway it's not a big issue, as it only appears when upgrade = "always" and there is an issue with user session, where some packages are loaded and it looks like they can not be updated. I think we can close this one. As we aligned on what was the initial cause.