Spec: Lockfile

[zkochan]

yarn's lockfile concept is pretty good. Here's its documentation: https://yarnpkg.com/en/docs/yarn-lock.

I suggest we document our own lockfile format in the specs.

[zkochan]

I think there is no need to save the whole URL to the tarball. It should be enough just to have a dictionary that converts fuzzy dependencies to exact ones:

registry: https://registry.npmjs.org
package-1:
  ^1.0.0: 1.0.3
package-2:
  ^2.0.0: 2.0.1
package-3:
  ^3.0.0: 3.1.9
package-4:
  ^4.0.0: 4.5.1
  ^4.5.0: 4.5.1
ied:
  alexanderGugel/ied#2.3.4: alexanderGugel/ied#373c7fa787e486438a998b4b4574ac5684d42e5f

Also tarball/local dependencies can be ignored, because they are exact dependencies already.

[billiegoose]

We need to keep the SHAsums though for security, so we can verify the package integrity. The whole point of the lockfile is to make sure you can duplicate exactly the setup you had before, which means we need to hash the content, not just map the location, of packages.

I'm also a little worried about how you resolved package-4 there, lol.

[billiegoose]

Are we going to embrace a hierarchy where there's a concept of "top-level packages" and "supporting packages"? I would think we have to to support uninstalling / garbage collecting unused packages. Each "store" could be considered a virtual package, and all the top-level packages in the store the virtual package's dependencies.

[billiegoose]

Also, just gonna toss this out there, can we avoid inventing a new file format that requires a specialized 300 LOC parser? It'll be easier to interoperate with if we use a format that is already popular, like YAML or JSON or TOML.

[zkochan]

I vote for YAML.

I agree on the SHAsums, lets add them as well.

Are we going to embrace a hierarchy where there's a concept of "top-level packages" and "supporting packages"? I would think we have to to support uninstalling / garbage collecting unused packages. Each "store" could be considered a virtual package, and all the top-level packages in the store the virtual package's dependencies.

Is this related to the lockfile? The store.yaml file has a dependency graph that allows to really easily uninstall packages from a store (it is already implemented in pnpm) and to prune it if necessary.

[alexanderGugel]

:+1: for YAML. Much easier to update for humans + you can add comments (I think you should be able to manually edit those lock files just in case).

Also there are plenty of great parsers for it now.

[zkochan]

So with the shasums I believe it would look something like this:

checksums:
  package-1:
    1.0.3: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
  package-2:
    2.0.1: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
  package-3:
    3.1.9: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
  package-4:
    4.5.1: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
registry: https://registry.npmjs.org
resolutions:
  ied:
    alexanderGugel/ied#2.3.4: alexanderGugel/ied#373c7fa787e486438a998b4b4574ac5684d42e5f
  package-1:
    ^1.0.0: 1.0.3
  package-2:
    ^2.0.0: 2.0.1
  package-3:
    ^3.0.0: 3.1.9
  package-4:
    ^4.0.0: 4.5.1
    ^4.5.0: 4.5.1

The file is a yaml file with sorted keys. The file name maybe dependencies-lock.yaml

[zkochan]

It appears that lockfiles won't work with shared stores. See this comment from @phestermcs.

[billiegoose]

Don't dismiss the lockfile just yet! It's still vital, but the issue phestermcs outlines is one I've been trying to work out. I think that's why I was wondering about some modules being "top level" installed modules. My assumption was that each top level package has its own (locked) dependency tree and the store lockfile would basically be a superset of those lockfiles.

The lockfiles could accurately describe the desired dependency tree. Whether that dependency tree was realizable in a directory structure with no duplication of modules was a problem I'd set aside for now, having come to the conclusion that new require semantics were necessary. (Hence, wmhilton/require.node-modules.io) But I am thrilled to see that @phestermcs is tackling this issue!

[zkochan]

@wmhilton, @alexanderGugel. pnpm currently uses a format similar to the package-lock.json but still different. Anyway, I am thinking about renaming the filename before v1 of pnpm. Currently, it is called shrinkwrap.yaml and I think it might be confusing because now npm has both a shrinkwrap and a lockfile.

The word lock is confusing as well because locks are used by pnpm (and not only) for locking filesystem.... So my idea is node_modules_state.yaml. What do you think, is it a good idea or better to leave it shrinkwrap.yaml?