Open torbjorn opened 5 years ago
To the best of my knowledge, Packrat won't ever try to automatically remove any downloaded source packages from the src
folder. Of course, I could be wrong.
Is there any chance you could distill this into a small, reproducible example? E.g. using a snippet of code that I could run on my own machine to reproduce the problem.
I tried creating a minimal setup with only a locally installed package. That did not reproduce this behaviour.
I did notice however that after a couple of iterations, my locally installed package has multiple source tar balls on the dev system under packrat/src, while on the prod system, after restore, there is only the one current tar ball. Even if I rsync'ed them all over.
Could that be at play above? The mechanism which removes sources not used or needed?
I had another stab at this one, this time running the ::restore with strace -e trace=unlink, below is what unfolded. As you can see, right under Replacing ga.data (upgrade 1.3.9-5 to 1.3.9-9) ...
it goes on to delete all the sources, including the one it should upgrade to. Once I run rsync again it does it just fine. Is it picky on file mtimes? Could there be slight time sync differences between two systems that causes this?
strace -e unlink R -e 'packrat::restore()'
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28943, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28944, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28947, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
unlink("/tmp/tmpfnBdNQq") = 0
R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28951, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
> packrat::restore()
unlink("/tmp/RtmplM6VjB/file710e4deaa62b") = 0
unlink("/tmp/RtmplM6VjB/file710e1807a70a") = 0
unlink("/tmp/RtmplM6VjB/file710e434ebd67") = 0
unlink("/tmp/RtmplM6VjB/file710e7bcf92b5") = 0
unlink("/tmp/RtmplM6VjB/file710e4bdebb10") = 0
unlink("/tmp/RtmplM6VjB/file710e40ff47a") = 0
unlink("/tmp/RtmplM6VjB/file710e2a75150e") = 0
unlink("/tmp/RtmplM6VjB/file710e74ce4d51") = 0
unlink("/tmp/RtmplM6VjB/file710e1aee383a") = 0
unlink("/tmp/RtmplM6VjB/file710e4a30f515") = 0
unlink("/tmp/RtmplM6VjB/file710e5f217cdf") = 0
unlink("/tmp/RtmplM6VjB/file710e27bc877c") = 0
unlink("/tmp/RtmplM6VjB/file710e1b108ab5") = 0
unlink("/tmp/RtmplM6VjB/file710e6aa25025") = 0
unlink("/tmp/RtmplM6VjB/file710e566db79b") = 0
unlink("/tmp/RtmplM6VjB/file710e31a5eb87") = 0
unlink("/tmp/RtmplM6VjB/file710e779f773e") = 0
unlink("/tmp/RtmplM6VjB/file710e348d3edb") = 0
unlink("/tmp/RtmplM6VjB/file710e3468d6e") = 0
unlink("/tmp/RtmplM6VjB/file710e504a6836") = 0
unlink("/tmp/RtmplM6VjB/file710e7125aa80") = 0
unlink("/tmp/RtmplM6VjB/file710e2990a8ec") = 0
unlink("/tmp/RtmplM6VjB/file710e3eb30f20") = 0
unlink("/tmp/RtmplM6VjB/file710e39cd47f4") = 0
unlink("/tmp/RtmplM6VjB/file710e15359ca4") = 0
unlink("/tmp/RtmplM6VjB/file710e4fbe396f") = 0
unlink("/tmp/RtmplM6VjB/file710e7ab1ac9e") = 0
unlink("/tmp/RtmplM6VjB/file710e33651309") = 0
unlink("/tmp/RtmplM6VjB/file710e33357736") = 0
unlink("/tmp/RtmplM6VjB/file710e2d85800c") = 0
unlink("/tmp/RtmplM6VjB/file710e19dff764") = 0
unlink("/tmp/RtmplM6VjB/file710e1201d61") = 0
unlink("/tmp/RtmplM6VjB/file710e458d2716") = 0
unlink("/tmp/RtmplM6VjB/file710e5d2eb4cb") = 0
unlink("/tmp/RtmplM6VjB/file710e7cefb016") = 0
unlink("/tmp/RtmplM6VjB/file710e116be227") = 0
unlink("/tmp/RtmplM6VjB/file710e613ea945") = 0
unlink("/tmp/RtmplM6VjB/file710e2764c525") = 0
unlink("/tmp/RtmplM6VjB/file710e63a2f78") = 0
unlink("/tmp/RtmplM6VjB/file710e7c2ce17f") = 0
unlink("/tmp/RtmplM6VjB/file710e7195ba3a") = 0
unlink("/tmp/RtmplM6VjB/file710e655bac57") = 0
unlink("/tmp/RtmplM6VjB/file710e23e968fc") = 0
unlink("/tmp/RtmplM6VjB/file710eca644f0") = 0
unlink("/tmp/RtmplM6VjB/file710e4ffdfc7c") = 0
unlink("/tmp/RtmplM6VjB/file710e7a572097") = 0
unlink("/tmp/RtmplM6VjB/file710e3e4c3077") = 0
unlink("/tmp/RtmplM6VjB/file710e479d73bb") = 0
unlink("/tmp/RtmplM6VjB/file710e2ee45f72") = 0
unlink("/tmp/RtmplM6VjB/file710e4192bde5") = 0
unlink("/tmp/RtmplM6VjB/file710e17e7dbf1") = 0
unlink("/tmp/RtmplM6VjB/file710e200a09f2") = 0
unlink("/tmp/RtmplM6VjB/file710e6b2366d1") = 0
unlink("/tmp/RtmplM6VjB/file710e569aeb11") = 0
unlink("/tmp/RtmplM6VjB/file710e59d751e7") = 0
unlink("/tmp/RtmplM6VjB/file710e590376") = 0
unlink("/tmp/RtmplM6VjB/file710e26592480") = 0
unlink("/tmp/RtmplM6VjB/file710e5488fe85") = 0
unlink("/tmp/RtmplM6VjB/file710e33be167f") = 0
unlink("/tmp/RtmplM6VjB/file710e598e9bb6") = 0
unlink("/tmp/RtmplM6VjB/file710e20e7e92") = 0
unlink("/tmp/RtmplM6VjB/file710e4d9e0de3") = 0
unlink("/tmp/RtmplM6VjB/file710e5aaeb917") = 0
unlink("/tmp/RtmplM6VjB/file710e479ba5a8") = 0
unlink("/tmp/RtmplM6VjB/file710e2accc2ae") = 0
unlink("/tmp/RtmplM6VjB/file710e579e692e") = 0
unlink("/tmp/RtmplM6VjB/file710e590787cf") = 0
unlink("/tmp/RtmplM6VjB/file710ec0b6bf3") = 0
unlink("/tmp/RtmplM6VjB/file710e7f032e53") = 0
unlink("/tmp/RtmplM6VjB/file710e5f41b747") = 0
unlink("/tmp/RtmplM6VjB/file710e8384d73") = 0
unlink("/tmp/RtmplM6VjB/file710e7098e88d") = 0
unlink("/tmp/RtmplM6VjB/file710e449d639e") = 0
unlink("/tmp/RtmplM6VjB/file710e2c21b66f") = 0
unlink("/tmp/RtmplM6VjB/file710e7d3f2d7d") = 0
unlink("/tmp/RtmplM6VjB/file710e149b601b") = 0
unlink("/tmp/RtmplM6VjB/file710e2678d706") = 0
unlink("/tmp/RtmplM6VjB/file710e3b8b5df5") = 0
unlink("/tmp/RtmplM6VjB/file710e5c38d3d6") = 0
unlink("/tmp/RtmplM6VjB/file710e555d3678") = 0
unlink("/tmp/RtmplM6VjB/file710e7d1e1bda") = 0
unlink("/tmp/RtmplM6VjB/file710e7420afc7") = 0
unlink("/tmp/RtmplM6VjB/file710e7567406b") = 0
unlink("/tmp/RtmplM6VjB/file710e684182ac") = 0
unlink("/tmp/RtmplM6VjB/file710e4abb9ad9") = 0
unlink("/tmp/RtmplM6VjB/file710e4f3e9252") = 0
unlink("/tmp/RtmplM6VjB/file710e689a8622") = 0
unlink("/tmp/RtmplM6VjB/file710e7114bf59") = 0
unlink("/tmp/RtmplM6VjB/file710e23c790d7") = 0
unlink("/tmp/RtmplM6VjB/file710e1c589ca1") = 0
unlink("/tmp/RtmplM6VjB/file710e4aa35b10") = 0
unlink("/tmp/RtmplM6VjB/file710e25d60f69") = 0
unlink("/tmp/RtmplM6VjB/file710e69f6aa85") = 0
unlink("/tmp/RtmplM6VjB/file710e25521427") = 0
unlink("/tmp/RtmplM6VjB/file710e6d71b512") = 0
unlink("/tmp/RtmplM6VjB/file710e14c36d33") = 0
unlink("/tmp/RtmplM6VjB/file710e7cf07d55") = 0
unlink("/tmp/RtmplM6VjB/file710e46793ce1") = 0
unlink("/tmp/RtmplM6VjB/file710e20ced927") = 0
unlink("/tmp/RtmplM6VjB/file710e7bf3aba8") = 0
unlink("/tmp/RtmplM6VjB/file710e25baf429") = 0
unlink("/tmp/RtmplM6VjB/file710e2907269a") = 0
unlink("/tmp/RtmplM6VjB/file710e6c8c9436") = 0
unlink("/tmp/RtmplM6VjB/file710e6a5857c7") = 0
unlink("/tmp/RtmplM6VjB/file710e5528dd09") = 0
unlink("/tmp/RtmplM6VjB/file710e69cbc1b3") = 0
unlink("/tmp/RtmplM6VjB/file710e7ef3b7e2") = 0
unlink("/tmp/RtmplM6VjB/file710e7ba1b40f") = 0
unlink("/tmp/RtmplM6VjB/libloc_228_ccadf3d65a81c52b.rds") = -1 ENOENT (No such file or directory)
Replacing ga.data (upgrade 1.3.9-5 to 1.3.9-9) ... unlink("/var/www/ga-biobank/perl/R_Home/packrat/src/ga.data/ga.data_1.3.9-5.tar.gz") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/src/ga.data/ga.data_1.3.7-30.tar.gz") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/src/ga.data/ga.data_1.3.7-20.tar.gz") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/src/ga.data/ga.data_1.3.9-9.tar.gz") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/src/ga.data/ga.data_1.3.7-24.tar.gz") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/biobank_db.json") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/kitlot_mapping.json") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_probe_levels.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_lx200_probe_diff.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_probe_categories.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_model_pca.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_kitlot_map.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_customer_bacteria_names.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/data/bacteria_limits_table_revision2.RData") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/data/bacteria_limits_table_revision3.RData") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_kitlots.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_genus_annotations.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_bacteria_names.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_probe_set.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_batch_correction.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat/test_model_data.r") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/tests/testthat.R") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/help/ga.data.rdx") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/help/AnIndex") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/help/ga.data.rdb") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/help/aliases.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/help/paths.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/Meta/links.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/Meta/hsearch.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/Meta/package.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/Meta/nsInfo.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/Meta/Rd.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/Meta/features.rds") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/bacteria_db.json") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/probe_categories.json") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/INDEX") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/html/R.css") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/html/00Index.html") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/DESCRIPTION") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/NAMESPACE") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/R/ga.data.rdx") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/R/sysdata.rdx") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/R/ga.data") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/R/ga.data.rdb") = 0
unlink("/var/www/ga-biobank/perl/R_Home/packrat/lib/x86_64-redhat-linux-gnu/3.5.1/ga.data/R/sysdata.rdb") = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28960, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
Error in installPkg(pkgRecord, project, repos, lib) :
Failed to install ga.data (1.3.9-9): sources missing at /var/www/ga-biobank/perl/R_Home/packrat/src/ga.data/ga.data_1.3.9-9.tar.gz
Calls: <Anonymous> -> restoreImpl -> playActions -> installPkg
In addition: Warning message:
In packrat::restore() :
The most recent snapshot was generated using R version 3.5.3
Execution halted
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28971, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 1 +++
Thanks -- I think I understand what's going on now.
When a package is upgraded / downgraded, Packrat attempts to remove the old version of the package, and then install the new one. Part of that implies calling removePkgs()
, e.g.
https://github.com/rstudio/packrat/blob/ffd810657948739d1e0d7c364d2b97dbe093633b/R/restore.R#L812
and that will also imply removing all cached sources for that package:
It looks like this was intentional:
https://github.com/rstudio/packrat/issues/19
but in hind-sight it seems over-aggressive. I don't think we should remove the package sources during a restore; they should only be cleaned up manually (e.g. by the user or by a smart-enough clean()
function)
Ok, then.
So is my chosen route for deploying my R based app wrong then?
My production system typically doesn't have access to a checked out dev. directory of package sources that my development system has, and from which it can install_local, and I'd be hesitant to provide it.
The other R-package-bundle approach is definitely an over kill. We make incremental updates to select few packages, recreating and reinstalling the entire bundle would be an insane waste of time for a small update. If I understand that approach right.
But why does it work the other time around? All I have to do, annoying as it is, is to have the restore() process remove the sources, complain loudly that they can't be found after that, then rsync them back, and run restore again, and this time its super happy.
The intended pattern for how this should work may or may not be optimal (goes without saying I think it isn't), but it ought to be consistent.
I don't think you're doing anything wrong. Packrat seems to have some unexpected / sub-optimal behavior here, and I think it should work without the extra orchestration that seems to be required in your case. I'll plan to fix this in Packrat directly.
I'm experiencing odd behavior.
I using packrat to control the production R environment, but deploying to production and calling restore does strange things. At first go it removes sources of the package I want to restore. It then complains it can't find the sources.
I do the following:
Odd. In my setup this is reproducing every time.
Here is the text output from a terminal as I go through the steps: