Porch: Unable to have two git repos with same URL and different directories

Original issue URL: https://github.com/kptdev/kpt/issues/3601 Original issue user: https://github.com/LiptonB Original issue created at: 2022-10-05T19:10:31Z Original issue last updated at: 2023-01-26T04:38:30Z Original issue body: ### Expected behavior I would like to store multiple groups of configs in the same git repository. I thought I might be able to do so by creating multiple porch Repositories pointing to the same git repo. So I have the following repositories with the same repo URL but different directory values:

apiVersion: config.porch.kpt.dev/v1alpha1
kind: Repository
metadata:
  name: repo1
  namespace: default
spec:
  content: Package
  deployment: true
  description: Repo1
  git:
    branch: master
    createBranch: true
    directory: /repo1
    repo: https://github.com/[redacted]
    secretRef:
      name: github-pat
  type: git

apiVersion: config.porch.kpt.dev/v1alpha1
kind: Repository
metadata:
  name: repo2
  namespace: default
spec:
  content: Package
  deployment: true
  description: Repo2
  git:
    branch: master
    createBranch: true
    directory: /repo2
    repo: https://github.com/[same redacted URL]
    secretRef:
      name: github-pat
  type: git

I expected to be able to clone a package into either one of these repos:

kpt alpha rpkg clone porch-test-blueprints-cd096908f5390cc134a7b4d631cd3717b94b1b5e newpackage --repository repo1 -ndefault
kpt alpha rpkg clone porch-test-blueprints-cd096908f5390cc134a7b4d631cd3717b94b1b5e newpackage --repository repo2 -ndefault

Actual behavior

Though the above is the behavior I'd most prefer (package name independent of the repository), the commands above both produce the error message:

Error: Internal error occurred: cannot create newpackage outside of repository directory "repo1"

To work around this, I tried including the directory name in the package name. This one worked:

kpt alpha rpkg clone porch-test-blueprints-cd096908f5390cc134a7b4d631cd3717b94b1b5e repo1/newpackage --repository repo1 -ndefault

However this one:

kpt alpha rpkg clone porch-test-blueprints-cd096908f5390cc134a7b4d631cd3717b94b1b5e repo2/newpackage --repository repo2 -ndefault

still produces an error message that references the repo1 directory:

Error: Internal error occurred: cannot create repo2/newpackage outside of repository directory "repo1"

I was not able to find a way to clone into this second repository.

Information

kpt version: 1.0.0-beta.19 porch version: 1de5acf9cc65e99a9b491db2e5dd643c231767b0 porch UI version: https://github.com/GoogleContainerTools/kpt-backstage-plugins/commit/6334354fb4e880ef316ef447609b4f8765dccfa5

Steps to reproduce the behavior

Use Porch UI to register two git repositories with the same repo URL and different directories
Attempt to clone a package from a separate blueprint repository (doesn't matter whether it's in the same or a different git repo) into these two Porch repositories

Original issue comments: Comment user: https://github.com/droot Comment created at: 2022-10-05T21:17:12Z Comment last updated at: 2022-10-05T21:17:12Z Comment body: Given that both the repositories are deployment repos, I suspect cloning restrictions are kicking in. If repo1 was a not a deployment repository, it would have probably worked.

/cc @ChristopherFry @mortent @natasha41575

Comment user: https://github.com/droot Comment created at: 2022-10-05T21:22:57Z Comment last updated at: 2022-10-05T21:22:57Z Comment body: I do want to point out that we are not testing this mode (registering separate porch repos for different directories in a single github repo) regularly so some more surprises may be in the store. Thanks for reporting it.

Comment user: https://github.com/LiptonB Comment created at: 2022-10-05T21:43:11Z Comment last updated at: 2022-10-05T21:43:11Z Comment body: Thanks for the quick answer! In case this is useful information, I also saw similar behavior when a blueprint repo was involved. I didn't test as many permutations there, but I was unable to clone a package in blueprint repo blueprints into deployment repo repo2, getting an error message that it couldn't create the package outside of repository directory "blueprints".

Comment user: https://github.com/droot Comment created at: 2022-10-05T22:02:14Z Comment last updated at: 2022-10-05T22:02:14Z Comment body: > Thanks for the quick answer! In case this is useful information, I also saw similar behavior when a blueprint repo was involved. I didn't test as many permutations there, but I was unable to clone a package in blueprint repo blueprints into deployment repo repo2, getting an error message that it couldn't create the package outside of repository directory "blueprints".

ok, so it is broken in all cases then. Thanks for confirming this.

@LiptonB I am curious about how are you planning to use kpt in general (use-case, what time configs, personas involved etc.). Feel free to share as much as detail possible, it can help us prioritize better.

Comment user: https://github.com/LiptonB Comment created at: 2022-10-06T15:25:07Z Comment last updated at: 2022-10-06T15:25:07Z Comment body: I'll start by saying that I think I don't actually need multiple repositories. I do think they would be useful if they provided namespacing such that the repo name didn't have to be in the package name. But I think that I will be ok with one big repository for the time being, and will just name the packages clustername/packagename. The context here is that I would like to maintain many deployments of the same blueprint, where each one will be deployed to either a different cluster or a different namespace within a cluster.

What I'm trying to do is manage a repo that will be the source of a GitOps sync for a large number of GKE clusters. I want to be able to automate the following:

Standing up a new cluster and the applications that go on it. That means setting up a spot in the repo for it to sync from and putting the relevant application configs into that sync location.
(Gradually) rolling out new application versions when they are built in CI by updating the image tag
(Gradually) rolling out configuration changes to all the clusters

I think kpt, and porch in particular, is relevant because:

Deployments of the same blueprint to different clusters can be in different states but there is still a process to roll out a change to all of them
Puts an API surface in front of edits to the git repo, facilitating writes from CI jobs and custom kubernetes controllers
Kpt functions provide a nice way to structure those writes (rpkg pull | kpt fn eval | rpkg push)
UI opens up new possibilities for editing objects (in this case mainly company-internal CRDs) without much kubernetes knowledge

Comment user: https://github.com/droot Comment created at: 2022-10-06T18:48:57Z Comment last updated at: 2022-10-06T18:48:57Z Comment body: > What I'm trying to do is manage a repo that will be the source of a GitOps sync for a large number of GKE clusters. I want to be able to automate the following:

Standing up a new cluster and the applications that go on it. That means setting up a spot in the repo for it to sync from and putting the relevant application configs into that sync location.

(Gradually) rolling out new application versions when they are built in CI by updating the image tag

(Gradually) rolling out configuration changes to all the clusters

I think kpt, and porch in particular, is relevant because:

Deployments of the same blueprint to different clusters can be in different states but there is still a process to roll out a change to all of them

Puts an API surface in front of edits to the git repo, facilitating writes from CI jobs and custom kubernetes controllers

Kpt functions provide a nice way to structure those writes (rpkg pull | kpt fn eval | rpkg push)

UI opens up new possibilities for editing objects (in this case mainly company-internal CRDs) without much kubernetes knowledge

Spot on. Makes perfect sense. The value prop of kpt stack is articulated even better than our copy of the website.

I'll start by saying that I think I don't actually need multiple repositories. I do think they would be useful if they provided namespacing such that the repo name didn't have to be in the package name. But I think that I will be ok with one big repository for the time being, and will just name the packages clustername/packagename. The context here is that I would like to maintain many deployments of the same blueprint, where each one will be deployed to either a different cluster or a different namespace within a cluster.

How do you setups roles (and permissions) for managing this configs or that is not a requirement in your case ?

Comment user: https://github.com/johnbelamaric Comment created at: 2022-10-06T19:35:10Z Comment last updated at: 2022-10-06T19:35:10Z Comment body: @LiptonB we have many of these same use cases in Nephio, which is a new OSS project for edge fleet management, with a focus on Telco. When relevant, I am trying to upstream the things we need for Nephio into Porch, because I see - as you do - many other use cases for the same functionality.

In your case, take a look at https://github.com/GoogleContainerTools/kpt/issues/3488, which is all about generation of variants of an upstream package. For the broader question of rollouts and - in the Nephio case - multi-actor, coordinated editing of the deployment config, you may want to see the issues referenced here: https://docs.google.com/document/d/1E-pMeb8u46C-UayBSaYG_WeqDcRrpq8OhocW3mr65ws/edit?usp=sharing (you'll need to join kpt-contribx@googlegroups.com to view and comment).

Comment user: https://github.com/johnbelamaric Comment created at: 2022-10-06T19:36:44Z Comment last updated at: 2022-10-06T19:36:44Z Comment body: There is also a demo PoC referenced in the doc, that is sort of the precursor to the functionality in #3488. In that case, we assume separate repos per cluster, but you'll see in #3488 we generalize targets beyond that.

Comment user: https://github.com/LiptonB Comment created at: 2022-10-06T22:27:32Z Comment last updated at: 2022-10-06T22:27:32Z Comment body: > How do you setups roles (and permissions) for managing this configs or that is not a requirement in your case ?

Well, this is all somewhat exploratory right now so I'm not completely sure 😄 We do need some access control over these configs - we'd want review of manual changes before applying them, though we might potentially be ok with system-driven rollouts happening automatically. Since I wasn't seeing those access controls in porch as it exists today and wasn't sure how things were going to evolve, I was thinking of starting by only using porch to create and edit draft revisions, and using GitHub PRs (with CODEOWNERS to control approvals) to make changes actually take effect. Haven't tested this workflow yet, though, so not sure how well it'll work.

nephio-project / nephio