Feature Request: Offline Package Build System

[JaredCrean2]

Would it be possible to add a feature to the package manager that would download but not install all dependencies of a particular package (possibly putting them in a specified directory)?

The motivation for this is getting my research code running on a compute cluster that cannot access the internet. I would like to download all dependencies to my local machine and scp them to the cluster.

One difficulty with this is packages that download and install external libraries using the download function. This would have to be done during the download step, but the package manager (currently) has no way of knowing what individual packages might download.

[tkelman]

What does "download but not install" mean here? Pkg has no control over what packages choose to do in their deps/build.jl script.

[JaredCrean2]

What does "download but not install" mean here

Clone the git repos, but don't execute deps/build.jl.

To make this work, there would have to be some mechanism for packages to register their download requests and let Pkg handle the actual downloading, so that it could be done during the download stage if needed.

[tkelman]

At present, Pkg.clone does not execute deps/build.jl, but you'd have to then free and resolve to get the same version as the latest tag that Pkg.add would use.

[wildart]

Would it be possible to add a feature to the package manager that would download but not install all dependencies of a particular package (possibly putting them in a specified directory)?

This is what Pkg does. It downloads all dependencies to local machine.

@JaredCrean2 It looks like you need a distributed installer which would install a package and its dependencies on the main host (Pkg does this), and then distribute/clone the installed package with dependencies to the rest of the cluster machines. This is clearly beyond the scope of the package manager. You'd better start writing you own package for that.

[davidanthoff]

I guess what @JaredCrean2 is suggesting is that there are more ways to hook up custom code than just the deps/build.jl script. For example, the package manager could first look for a deps/download.jl script, call that if present, and then in a second step call deps/build.jl, if present. The convention for package authors would then be that all downloading happens in deps/download.jl, and all building happens in deps/build.jl. Specifically, the convention would have to be that deps/build.jl never ever downloads anything.

With that setup, one could "prep" a package directory for offline building by calling the code in download.jl, and then copying the whole package directory over to some other machine that doesn't have internet access, and then call all the build.jl scripts on that offline machine.

[JaredCrean2]

@davidanthoff Yes, that's a better way of describing it.

[JaredCrean2]

Pkg.clone does not execute deps/build.jl,

It does not execute the deps/build.jl of the cloned package, but it does for the cloned package's dependencies. For example, cloning Petsc.jl installs MPI.jl, and MPI.jl builds a little C/Fortran library as part of its installation. The problem is the C/Fortran library needs to be build on the cluster, not my local machine, because it needs to be customized for the build environment.

Here is the output from cloning on a clean machine (no other packages installed):

julia> Pkg.clone("git@github.com:JuliaParallel/PETSc.jl.git")
INFO: Cloning PETSc from git@github.com:JuliaParallel/PETSc.jl.git
Warning: Permanently added the RSA host key for IP address '192.30.252.122' to the list of known hosts.
INFO: Computing changes...
INFO: Cloning cache of BaseTestNext from git://github.com/JuliaCI/BaseTestNext.jl.git
INFO: Updating cache of MPI...
INFO: Installing BaseTestNext v0.2.1
INFO: Installing BinDeps v0.3.21
INFO: Installing Compat v0.7.11
INFO: Installing MPI v0.3.2
INFO: Installing SHA v0.1.2
INFO: Installing URIParser v0.1.2
INFO: Building MPI
INFO: Recompiling stale cache file /home/dev1/.julia/lib/v0.4/BinDeps.ji for module BinDeps.
INFO: Recompiling stale cache file /home/dev1/.julia/lib/v0.4/Compat.ji for module Compat.
INFO: Recompiling stale cache file /home/dev1/.julia/lib/v0.4/URIParser.ji for module URIParser.
INFO: Recompiling stale cache file /home/dev1/.julia/lib/v0.4/SHA.ji for module SHA.
INFO: Attempting to Create directory /home/dev1/.julia/v0.4/MPI/deps/build
INFO: Changing Directory to /home/dev1/.julia/v0.4/MPI/deps/build
-- The Fortran compiler identification is GNU
-- The C compiler identification is GNU 4.8.2
-- Check for working Fortran compiler: /usr/bin/f95
-- Check for working Fortran compiler: /usr/bin/f95  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/bin/f95 supports Fortran 90
-- Checking whether /usr/bin/f95 supports Fortran 90 -- yes
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Found Git: /usr/bin/git (found version "1.9.1") 
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/libmpich.so;/usr/lib/x86_64-linux-gnu/libopa.so;/usr/lib/x86_64-linux-gnu/libmpl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/libcr.so;/usr/lib/x86_64-linux-gnu/libpthread.so  
-- Found MPI_Fortran: /usr/lib/x86_64-linux-gnu/libmpichf90.so;/usr/lib/x86_64-linux-gnu/libmpich.so;/usr/lib/x86_64-linux-gnu/libopa.so;/usr/lib/x86_64-linux-gnu/libmpl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/libcr.so;/usr/lib/x86_64-linux-gnu/libpthread.so  
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Configuring done
-- Generating done
-- Build files have been written to: /home/dev1/.julia/v0.4/MPI/deps/build
Scanning dependencies of target gen_constants
[ 20%] Building Fortran object CMakeFiles/gen_constants.dir/gen_constants.f90.o
Linking Fortran executable gen_constants
[ 20%] Built target gen_constants
Scanning dependencies of target version
[ 20%] Built target version
Scanning dependencies of target gen_functions
[ 40%] Building C object CMakeFiles/gen_functions.dir/gen_functions.c.o
Linking C executable gen_functions
[ 40%] Built target gen_functions
Scanning dependencies of target mpijl-build
[ 60%] Generating mpi-build.jl
[ 60%] Built target mpijl-build
Scanning dependencies of target mpijl
[ 80%] Generating compile-time.jl
[ 80%] Built target mpijl
Scanning dependencies of target juliampi
[100%] Building Fortran object CMakeFiles/juliampi.dir/test_mpi.f90.o
Linking Fortran shared library libjuliampi.so
[100%] Built target juliampi
[ 20%] Built target gen_constants
[ 20%] Built target version
[ 40%] Built target gen_functions
[ 60%] Built target mpijl-build
[ 80%] Built target mpijl
[100%] Built target juliampi
Install the project...
-- Install configuration: ""
-- Installing: /home/dev1/.julia/v0.4/MPI/deps/src/./compile-time.jl
-- Installing: /home/dev1/.julia/v0.4/MPI/deps/usr/lib/libjuliampi.so

[tkelman]

Ah, right, so if you really wanted to avoid all deps/build.jl executions you'd need to crawl the dependencies and manually clone from leaves back up. Kinda messy. Maybe a keyword argument to Pkg.add and/or Pkg.clone that controls whether the build script gets executed (and/or whether the build script of dependencies gets executed) would be a useful feature.

[wildart]

I started to work on PkgDev.bundle command [#36] which would create a package bundle with its dependencies.

[JaredCrean2]

Great, thanks