Proposal for a nicer way to manage dependencies

[anchpop]

Hi, I want to leave this issue as a spot for people to talk about their ideas for dependency management. I think unison's unique architecture has the potential to make great strides in this area.

Here are some common use cases:

1) Someone wants to add a new dependency to their project they found on the internet

2) Someone wants to see all the packages in their project to see if they're out of date

3) Someone wants to write a library and share their code with others

4) Someone wants to make an edit to a library they're using, and maybe try and merge those changes back upstream

Unison's current system struggles most with the second, because there's no way to see all the external dependencies you've pulled in. Here's my proposal for a change.

The current system:

To pull some code in your project, type something like this into ucm:

pull https://github.com/unisonweb/refactoring-example.git:.cooking .cooking

Which will look at the git repo at github.com/unisonweb/refactoring-example.git, download the namespace .cooking and drop it into the .cookingnamespace in your project. Then you can use.cookingto refer to that code, i.e..cooking.cinnamon`.

If you run that twice, it will tell you Looks like .cooking is up to date..

My proposal:

Instead of needing to unload the git repo into a namespace in your project, we say that every git repo is already in a namespace. So anywhere in my project I can write the following:

(git@https://github.com/unisonweb/refactoring-example.git).cooking.cinnamon

And without needing to do anything in explicit in ucm, when this code is loaded it will access https://github.com/unisonweb/refactoring-example.git, load it into your project (but not into any namespace you'd normally see from ucm), and then understand that when try to access (git@https://github.com/unisonweb/refactoring-example.git) you're accessing that new namespace. When you put this code into your project with the ucm add command, it remembers this is code from an external git repo so the next time you edit that code you'll see something like:

(git@https://github.com/unisonweb/refactoring-example.git#2fb4ec84).cooking.cinnamon

Where 2fb4ec8443cfa0041062e5dbd5fc326ca0d63fbf is the hash of the commit that was grabbed.

(git@ here is special syntax to tell unison to grab a git repo at that address instead of looking in your project. If there were some kind of centralized unison package repo it would be easy to extend this syntax with upm@ or anything else.)

Obviously, that's too unwieldy to write every time, so like a normal namespace you would also be able to put this at the top of any file:

use (git@https://github.com/unisonweb/refactoring-example.git#2fb4ec844)

mystuff = cooking.cinnamon

Or, for less ambiguity where stuff is coming from, perhaps some more new syntax could be added like this:

use (git@https://github.com/unisonweb/refactoring-example.git#2fb4ec84) as refactoring-example

mystuff = refactoring-example.cooking.cinnamon

Where the use x as y syntax means when I refer to the namespacey, I mean the namespacex`. My idea was that the name here is automatically chosen from the name of the git repo, but there may be a better way.

The advantage are:

1) When you use ls or cd, etc. to look inside your codebase, you only see your code, it's never cluttered up by all your dependencies.

2) It's obvious when you import something whether it's coming from your codebase or the internet. You could also share a just code snippet with someone who's having an issue online, even if it needs a library, by telling them "try the function (git@https://github.com/unisonweb/refactoring-example.git).mogrify if you're having trouble mogrifying something". It's sort of more declarative this way.

3) It makes it possible to have a structured way to manage and examine external dependencies

4) The code is still downloaded to your project as normal, this isn't a fundamental change to the way dependencies work. More of a change to the UI to encourage a bit of organization.

So that's the first part of my proposal. The second part is a new deps ucm command. I'm thinking it would work something like this:

.> deps

  Your project has 3 external dependencies:

    1. refactoring-example  -  up to date with master on git@https://github.com/unisonweb/refactoring-example.git
    2. base#7ec9cf1         -  up to date with master on git@https://github.com/unisonweb/base.git
    3. base#149a9af         -  1 commit behind master on git@https://github.com/unisonweb/base.git  -- What's happening here is that different parts of your code depend on different versions
                                                                                          --   of base, so the git commit hash is used to disambiguate them (as they have the same name).

.> view 1

  refactoring-example, from git@https://github.com/unisonweb/refactoring-example.git #2fb4ec84, is referencd by these values:

    1. app.baking.ingredientsMenu
    2. simulation.baking.util

  refactoring-example itself has 4 direct dependencies and 23 transitive dependencies. 

.> deps update base#149a9af

  Updating your project's usage of base from #7ec9cf1 (one commit behind) to the latest `master`, #7ec9cf1...

  Here's what's changed in your codebase in after the update:

    Updates:

      1. mylibrary.releases.v1.square : Nat -> Nat
         ↓
      2. mylibrary.releases.v1.square : Nat -> Nat

      3. square : Nat -> Nat
         ↓
      4. square : Nat -> Nat

      There were 2 auto-propagated updates.

      5. patch patch (added 2 updates,
      deleted 2)

  Tip: You can use `todo` to see if this generated any work to do and `test` to
       run the tests. Or you can use `undo` or `reflog` to undo the results of this uptate.

.> deps

  Your project has 2 external dependencies:

    1. refactoring-example  -  up to date with from git@https://github.com/unisonweb/refactoring-example.git
    2. base                 -  up to date with      git@https://github.com/unisonweb/base.git

The only dependencies listed are those that are depended on by code currently inside your codebase. Since we updated our usages of base from the old version to the new, it stopped showing up in our deps command. Running update attemts to change your project's usage of a dependency to the latest version on that branch. deps should also have similar commands for updating your codebase's use of a dependency from one git repo to another, or changing to a particular hash, or only updating usages in a particular namespace.

Stuff visible from your dependencies is also accessible from find and view. But trying to use edit or update to change stuff exported by your dependencies probably shouldn't work - if you need to do that, you should still be able to pull the code and drop it directly into your project like you do now.

Thoughts? I could try to get started implementing this if you all think it's a good idea.

[aryairani]

I agree it would probably be good to have some sort of way to track a library author's updates to your dependencies. I also think it could be good to have a way to reference external definitions without requiring a manual pull step first. I like the idea for listing, viewing, updating deps too.

We have previously considered viewing git repos as a read-only overlay on the namespace, similar to what you described ("every git repo is already in a namespace."), including support in imports.

The notion of an "out-of-date package" (or a "package" at all) isn't fully sorted out for Unison though, so we would need to workshop that a bit. And before implementing anything, we should have a plan for how these external deps would be represented in the codebase.

I think we'll want to take a look at this again after M2!