Structure? - Githubissues

ari-becker commented 5 years ago

@f-f how do you envision that packages here should be structured? i.e. how should pull requests be made?

I imagine something like adding a file, probably one whose name equals the package, whose contents are a record with a field pointing to a hashed/frozen public URL, but where should that file be placed?

ari-becker commented 5 years ago

Another idea: the record itself holds multiple fields according to a convention, each pointing to a different version. i.e.

{ latest = https://example.com/v2/package.dhall sha256:abcdef
, v2 = https://example.com/v2/package.dhall sha256:abcdef
, v1 = https://example.com/v1/package.dhall sha256:123456
}

f-f commented 5 years ago

@ari-becker sorry for missing documentation, I'll add something to the readme later today.

Some thoughts:

I think "a package" is just a key in the packages.dhall record
And this key should always point to the latest version of the package, as it's not usual that one needs to use different versions of the same package at the same time We can release tags of this repo quite often so people can point at tags instead of master

ari-becker commented 5 years ago

I'm not sure if it's a good thing for end users to be pulling in the entire packages.dhall? It would help with performance testing, I'm sure, but in real-world usage I think people will only want to pull in the packages they need.

While individuals won't need multiple versions of the same package at the same time, it goes to reason that we can't make heavy users upgrade all of their packages at the same time, so for any given tag of packages, different users will want to pin to different versions of a package.

Maybe it would be best to have each package be a folder in the repository - packages/<package_name> and then each folder can have a latest.dhall , v1.dhall etc. so that end users aren't required to pull all versions of the package at the same time?

ari-becker commented 5 years ago

To add another point against a unified packages.dhall - if the main rationale for Dhall to have hashed expressions in the first place was security (caching being a side benefit), then don't we jeopardize users' security by forcing them to download all packages, whether they want to or not, whenever they update their tag pointing to packages.dhall? Wouldn't it be contrary to the language's goals?

f-f commented 5 years ago

@ari-becker all fair points, I'll detail on this a bit more:

Much of my stance on this comes from thinking a lot about this problem for purescript/package-sets, so I'm happy to further expand on any assumptions I'm postulating here
We should look at this keeping in mind what's the goal for this repo: to provide a discoverable collection of packages that are CI-guaranteed to be working with the latest Dhall version (of course we can change this goal if so we wish) Having a single record with all packages means that the CI check is very low maintenance: dhall <<< "./packages.dhall"
In this sense, it's not compulsory for us to provide a single record that you can just import, and if we do it's not compulsory for users to import it:
- if you need to import only one package, you can just copy one import
- if you need to import more than one package, you could either:
- import the whole record and access the packages in the record
- copy different remote imports over to your project, and have a project-wide package set - note that this solution would be the only one available if we had different records, as users would need to either copy the imports, or import each record
Since this is just a Dhall record, anyone wanting to override package versions can do so with normal record merge: ./packages.dhall // { Prelude = .. }, so I wouldn't say users are forced to upgrade the whole thing at once
This said, it's totally possible to have both solution: a unified packages.dhall and separate records for packages
"pulling all packages at the same time" is not a problem IMO (that is, once we ship the new evaluation. In the current situation any big package is borderline too slow even just by itself), the only delay is when users have a "cold cache" Having "trusted cache servers" a-là Nix would fix this too
Re security: again users are not being forced to upgrade, or to upgrade everything. Packages in the packages.dhall have hashes on them, so it's easy to diff between commits/tags to audit what has changed

ari-becker commented 5 years ago

So to import Prelude and dhall-kubernetes, you imagine people doing something like let imports = (https://raw.githubusercontent.com/dhall-lang/packages/some-tag/packages.dhall sha:abcdef).(Prelude, kubernetes)? Doesn't that still require the entire normalized contents of packages.dhall to be fetched? If the new evaluation algorithm takes care of this concern, then great, but I can still imagine a situation where the sheer size of such a cache could become a problem (but maybe that's a nice problem to have?), and anyways are we sure it's not a security concern? Although I will say that I really like the ./packages.dhall // { Prelude = override } idiom, that's smooth.

I think there could be a pretty simple PR check which, for a proposed packages/<package_name>/x.dhall addition, checks if (./packages.dhall).package_name.x evaluates.

I don't mean 'forced' in the sense of, people's machines being bricked if they don't upgrade or some foreign entity making the change for you without approval. I mean 'forced' in the sense that nobody really has the resources to evaluate every change made to every tool (particularly when you're being 'encouraged' to import everything), so it's safer and easier to just not import dependencies you don't need, even if you wouldn't take a performance hit by doing so.

If the goal is to provide a guarantee by CI that the packages are working, then what happens with packages whose locations bit-rot? Should the CI only accept GitHub locations (not really a guarantee because commits and tags can be deleted)? Should the CI be able to remove broken packages, so that the head packages.dhall can still resolve? Should the CI need to provide a cache layer whereby the imports of broken packages fall back to the cache provided by the CI?

singpolyma commented 5 years ago

I, too, am concerned about the time to load this form. You can say "oh, it'll be cached" but if we get any sizable ecosystem of libraries that cache would be huge.

joneshf commented 5 years ago

So to import Prelude and dhall-kubernetes, you imagine people doing something like let imports = (https://raw.githubusercontent.com/dhall-lang/packages/some-tag/packages.dhall sha:abcdef).(Prelude, kubernetes)?

Based on this point:

This said, it's totally possible to have both solution: a unified packages.dhall and separate records for packages

I'd imagine you could do something like:

let Prelude = https://raw.githubusercontent.com/dhall-lang/packages/some-tag/packages/Prelude.dhall sha:abcdef
let kubernetes = https://raw.githubusercontent.com/dhall-lang/packages/some-tag/packages/kubernetes.dhall sha:fedcba

If you wanted those individual packages, as well as import the whole thing in one go.

dhall-lang / packages

Structure? #1