search

commercialhaskell / stack

The Haskell Tool Stack
http://haskellstack.org
BSD 3-Clause "New" or "Revised" License
3.97k stars 846 forks source link

Proposal: Dependency vendoring for Stack #3813

Open loewenstein opened 6 years ago

loewenstein commented 6 years ago

Dependency vendoring for Stack

Use cases

With vendored dependencies we get the capability to build, test & run a Haskell project without access to either Hackage or Stackage.

  1. Get a project from version control and start working without internet connectivity (local, offline development)
  2. Build a project on a remote machine without internet access (Bosh release)
  3. Build a project on a remote machine without internet access and with limited local disk space (Cloud Foundry buildpack)

Problem

Stack has to get the Hackage index and download all dependency packages prior to building. As a workaround for use case use case 1 we can stack build --prefetch --dry-run to get everything local. As a workaround for use case 2 we can create a tarball from stack-root. However the creation is rather inconvenient and the result larger than we want.

Solution

The recently released extensible snapshot work can be used as basis for complete offline builds with limited file system usage. We can download all dependency packages (stack list-dependencies and remove ghc package db packages) and list them in a snapshot referring only to ghc-x.y.z snapshot. Given an existing GHC we can use the snapshot and --system-ghc to build without any internet access.

Doing what is described as solution from outside of stack is inconvenient though. That is why we propose to add a command to stack:

stack vendor-dependencies [--path <path>] <upstream snapshot>

The command should download the dependency packages for all targets to (which should default to <project root>/vendored) and create a snapshot YAML at <path>/vendored.yml including all the downloaded packages and referring to the ghc resolver used by .

Limitations

We do not propose to vendor ghc itself because that is system dependent. As a consequence GHC has to be provided out of band.

borsboom commented 6 years ago

Related: #1594 #3106 #359

borsboom commented 6 years ago

Would this approach support the vendored packages living in a directory outside? And could it be made to support getting vendored packages from an http(s) server on a company intranet (disconnected rom the Internet)?

loewenstein commented 6 years ago

Not sure I get the questions right.

As long as a project can refer to a resolver somewhere there is no limitation to having the vendored snapshot local to the project. @snoyberg is there any limitation where custom snapshots can exist?

Is the second question: "Can I call stack vendor-dependencies when in a shielded network?" I would suppose vendoring would reuse the package download facilities of stack itself. If stack can get packages from a server within the intranet than vendoring can.

Does that answer the questions?

snoyberg commented 6 years ago

is there any limitation where custom snapshots can exist?

I'm not sure what that means.

loewenstein commented 6 years ago

@snoyberg I was trying to answer @borsboom s question.

Would this approach support the vendored packages living in a directory outside?

The idea of the vendoring proposal is to create a custom snapshot with local archives at , so as long as I can reference a snapshot in any location, i.e. not necessarily relative to the stack.yml, this should be no problem.

snoyberg commented 6 years ago

Yes, you should be able to refer to a different filepath for a custom snapshot definition.