Question: possible to mirror garden locally

[jhoogeboom]

I'm wondering if there is any possibility to mirror a garden locally, similar to the travel function in ohmyrepos?

[davvid]

Yeah, there are multiple ways to clone gardens. The simplest way is to copy garden.yaml somewhere else and then ask garden to grow all of the repos in that new location. e.g.

cp garden.yaml /media/$USER/storage
garden -C /media/$USER/storage grow '@*'

Another way is to use the garden --root <path> option to override the garden root on the command-line.

garden -r /media/$USER/storage grow '@*'

That'll grow all of the trees in the current garden file into the /media/$USER/storage directory without having to copy a garden file over there.

You can get a lot of mileage out of overriding "garden variables" and the garden.root value on the command-line. If you use variables in your garden file then you can garden -D name=value to override the variable called name with the specified value.

Another option is to configure bare repositories for the external storage and then use the remotes field in each non-bare tree to add a storage git remote so that we can push to / pull from the external storage. That's the most "gittish" way of keeping things in sync across multiple mediums.

e.g.

variables:
    storage: /media/${USER}/storage

templates:
    storage:
        url: git@example.org:projects/${TREE_NAME}.git
        remotes:
            storage: ${storage}/${TREE_NAME}.git
        commands:
            setup: |
                mkdir -p ${storage}
                git init --bare ${storage}/${TREE_NAME}.git
            push-storage: git push storage "$@"
            push-var: git push ${storage} "$@"

trees:
    project-one:
        templates: storage
    project-two:
         templates: storage

groups:
    projects: project-*

When you get new storage you do this to recreate your setup:

# Clone projects and configure the "storage" remotes
garden grow projects

# Create the initial bare repositories
garden setup projects

# Day-to-day: push into the "storage" remotes from all of the repos:
garden push-storage projects

# You can use ad-hoc locations by overriding the ${storage} variable
garden -D storage=/media/$USER/new-storage setup projects
garden -D storage=/media/$USER/new-storage push-var projects

If the projects already exist then garden grow can be used to retrofit existing project-* trees and add the storage remotes:

garden grow projects 

If you have many repos then garden exec and custom commands can help automate a lot of these kinds of use cases.

garden grow will create full-blown worktrees in arbitrary locations using the garden -r | --root option, which may or may not be what you want. If you could elaborate on, "mirror a garden locally," I can offer some other approaches.

One thing that could be helpful would be to teach garden grow to create empty bare repositories when the tree has no url or remotes configured. In other words, it would be cool if we could if we made it so that this yaml:

trees:
    project-one.git:
        bare: true
        path: ${storage}/project-one.git

... creates an empty bare repositories when garden grow '@project-*.git' is run. Right now the url field is required so that doesn't currently work without specifying url, but we can change that. If you think that'd be useful I can look at adding support for that. We already support bare repos so making the url field optional would be pretty straightforward.

Let me know what you think and whether any of these options are more compelling than the others.

[jhoogeboom]

Thank you so much for this very elaborate answer! I think with this example, it could work:

variables:
    storage: /media/${USER}/storage

templates:
    storage:
        url: git@example.org:projects/${TREE_NAME}.git
        remotes:
            storage: ${storage}/${TREE_NAME}.git
        commands:
            setup: |
                mkdir -p ${storage}
                git init --bare ${storage}/${TREE_NAME}.git
            push-storage: git push storage "$@"
            push-var: git push ${storage} "$@"

trees:
    project-one:
        templates: storage
    project-two:
         templates: storage

groups:
    projects: project-*

Let's say I need to restore my repos from 'storage', how would I do that, can I point the url to the storage path?

One thing I was wondering is, if templates can be assigned on the group level? In this case so that the trees don't all individually need the templates: storage. Also interesting what you show here with the project-* , I had to reread that a bit here.

[davvid]

Let's say I need to restore my repos from 'storage', how would I do that, can I point the url to the storage path?

Yeah, that'd be the way to do it. Once you point the url there then you'd garden grow projects.

Later if you want to swap over to the network url you can edit the url field and rerun garden grow projects to apply the updated git configuration.

One thing I was wondering is, if templates can be assigned on the group level? In this case so that the trees don't all individually need the templates: storage.

They can only be assigned bottom-up from the tree definition. Groups are moreso aggregates over trees rather than something that defines a tree. Trees can be in multiple gardens and groups, for example.

This does bring up a scenario where maybe we could shortcut things a bit further. Right now we currently support a shorthand notation for trees:

trees:
    name: url

This is equivalent to writing:

trees:
    name:
        url: url

Which is basically the minimum for defining a tree that can be cloned.

It wouldn't be a far stretch to introduce a shorthand for trees that don't need to specify a URL for trees that are able to leverage templates to provide their full definition.

Some possible options:

trees:
    # allow a single key=value, useful in case we extend this to unknown other scenarios later 
    project-x: templates=storage,other

    # or we can hijack the array syntax because `templates` can use multiple templates.
    project-y: [storage, other]

I kind of like the latter because it uses valid yaml syntax rather than string tricks.

If we did that then writing this:

trees:
    project-one: [storage]
    project-two: [storage]

would be equivalent to writing this:

trees:
    project-one:
        templates: [storage]
    project-two:
        templates: [storage]

which seems like a nice space savings when you have lots of trees.

I can't really see into the future to know if we'd ever want a different non-template meaning for this shorthand array notation, but from here it does seem like this could be a good use for it.

What do you think @jhoogeboom? If you'd find that useful I can look at extending the tree spec to interpret lists/arrays as template references.

[jhoogeboom]

If I understand it correctly, the shorthand is so that any single value after a treename, is interpreted as the url right now (trees: name: url), and any yaml list could be interpreted as template (trees: project-one: [storage])?

Could be useful I guess, but since I work with a lot of repositories I might have to go with something more automated anyway and then it does not matter so much.

Some things I was wondering:

If a tree exists, for example when I've done a garden grow then I don't need to enter the full treename for subsequent commands. However when a tree does not exist, it needs the full treename.

for example

trees:
  projects/garden/repo1:

When it exists, I can do garden grow repo1 when it doesn't I need enter the full path garden grow projects/garden/repo1 or a tree query like garden grow projects*

Is this on purpose?

When I plant a tree, for example to update the remotes, then an absolute path is added to the tree, even if there is already an url in the template.

variables:
  source: "${HOME}"
  storage: "${HOME}/projects/garden/storage"
garden:
  root: "${HOME}"
templates:
  storage:
    url: "${source}/${TREE_NAME}"
    remotes:
      storage: "${storage}/${TREE_NAME}.git"

trees:
  projects/garden/repo1:
    templates:
      - storage

when I do garden plant repo1, the tree is edited with:

trees:
  projects/garden/repo1:
    templates:
      - storage
    remotes:
      storage: /Users/jhoogeboom/projects/garden/storage/projects/garden/repo1.git
    url: /Users/jhoogeboom/projects/garden/repo1

Can this be prevented? Or is it up the user to undo this again if needed?

[davvid]

I think we should be able to detect this situation and retain the original expressions instead of replacing them.

We'd just need to skip setting the value when the current expression evaluates to the same value as what's already in the config. I'll take a stab at improving this before the next release.

[davvid]

Please test out what's in git if you can ~ this should make things better for the url: field and remotes: block when using variables to define things.

This same strategy can be applied to other parts of garden plant as well, but we'll see whether we end up needing to do that.

[davvid]

even if there is already an url in the template

I think this part isn't handled yet, at least. I think garden plant is still going to add a url since it's not really taking templates into consideration and is just looking at the raw yaml and it might be unconditionally adding a url field.

It might just work out because we're querying the configuration, but I'll have to test this to see if we really do avoid writing the url field. That might need a further tweak. It is still worth testing what's currently in git anyways since it did improve things. I'll see if it also handled the template scenario as well (TBD).

[davvid]

I just tested this and it looks like the logic worked out to avoid writing the url field with templates, so this is good to go as-is. Now I just gotta write a test for this.