Enable multiple integrations of the same provider

[LvffY]

Feature Suggestion

It would be great to be able to have multiples integrations for the same provider. For exemple to be able to define :

integrations:
  github:
    - host: github.com/myOrg
      token: ${GITHUB_MYORG_TOKEN}
  github:
    - host: github.com/myOtherOrg
      token: ${GITHUB_MYOTHERORG_TOKEN}

Context

My team is working with multiple azure devops organisation and, AFAIK, we can't define multiple times a same integration type (in my case dev.azure.com).

It would be great to add this feature to allow organizations with multiple clients on the same providers to be able to define a token by client.

[jhaals]

Hi! Multiple integrations for the same host is not supported but we have something similar for Github where a single integration can have multiple Github apps and we need to use the correct GitHub app for the corresponding organization. We have abstracted that with a CredentialsProvider which might be something that could be used to draw inspiration from. There's currently logic that relies on host for lookups so adding a /org will probably cause issues for existing code.

[LvffY]

Hi @jhaals

Thanks for your reply :)

What I understand is that this could be quite long to implement, right ?

I don't know if this could help but I will explain a bit more my context : my team is working with multiple clients. Because we work on our client architecture, we don't decide which git provider to use. So right now, we have 2 gitlab instances and 2 Azure devops organizations. In order to aggregate our team developer portal, we need to be able to define something like this :

integrations:
  gitlab:
    - host: my.gitlab.provider.com/teamA
      token: ${GITLAB_A_TOKEN}
    - host: my.othergitlab.provider.com/teamB
      token: ${GITLAB_B_TOKEN}
  azure:
    - host: dev.azure.com/orgA
      token: ${AZDO_A_TOKEN}
    - host: dev.azure.com/orgB
      token: ${AZDO_B_TOKEN}

Another use case that I imagine is that multiple teams are working on the same Backstage instance but they don't know each others. So we could imagine that the backstage team would have to deal with one token to access code of the team A and another token to access code of the team B.

I don't know if it changes something but Github is not my personnal priority.

Last but not least, this problem is linked to https://github.com/backstage/backstage/issues/10431

[jhaals]

But in your first example with Gitlab the two instances have different hostnames, Backstage should be able to handle that without any additional features. Or is something not working as expected with just this configuration?

integrations:
  gitlab:
    - host: my.gitlab.provider.com
      token: ${GITLAB_A_TOKEN}
    - host: my.othergitlab.provider.com
      token: ${GITLAB_B_TOKEN}

For the azure config example there is currently no support for having different tokens for authentication to the same host. It feels like this is something that needs supporting. I don't know if you can reuse the same token between orgs. @awanlin do you know how this works today with Backstage and multiple orgs?

I understand that Github is not your personal priority. I brought that up as prior art since GitHub support one integration section containing multiple apps for authentication :)

[LvffY]

@jhaals Thanks for the clarification :)

To be honest, I wasn't able to try it for now (because one of my gitlab is in a private network and we didn't call the network team), so I don't know if it's working right now. But you're right, because it's two separate hostnames, it should work.

But actually, I also have the same request for gitlab : be able to connect to the same gitlab host with multiple tokens (i.e to access multiple groups which don't have common developers and so can't see the code of each others).

To be explicit, I could have the following integrations :

integrations:
  gitlab:
    - host: my.gitlab.provider.com/teamA
      token: ${GITLAB_A_TOKEN}
    - host: my.gitlab.provider.com/teamB
      token: ${GITLAB_B_TOKEN}
    ## As you said previously, this should work out-of-the-box, because hostname is different
    - host: my.othergitlab.provider.com/teamC
      token: ${GITLAB_C_TOKEN}
  azure:
    - host: dev.azure.com/orgA
      token: ${AZDO_A_TOKEN}
    - host: dev.azure.com/orgB
      token: ${AZDO_B_TOKEN}

[benjdlambert]

Yep, I think we're going to want to take a little step back here and think about adding CredentialProviders for other integration types, so that we can do some additional logic when acting with the integrations depending on the URL's. @Rugvip or @freben do you have any other ideas on how we could support this?

[freben]

yeah, we probably want to shield this all behind credential providers, and the tricky part of course being that it's all host based in the config now rather than baseUrl based - so we'll have to come up with some transition plan there

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[LvffY]

@freben @benjdlambert is there any update on this point ?

[Rugvip]

I think we need to go down the credential provider route tbh, perhaps a bit like we've ended up in the frontend with multiple providers behind a similar API. Bit more tricky in the backend space though I would think.

The GitHub interface is pretty simple though, probably something we can replicate:

type GithubCredentialType = 'app' | 'token';

type GithubCredentials = {
  headers?: { [name: string]: string };
  token?: string;
  type: GithubCredentialType;
};

interface GithubCredentialsProvider {
  getCredentials(opts: { url: string }): Promise<GithubCredentials>;
}

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[LvffY]

@Rugvip do we have any update on this issue ?

[Rugvip]

@LvffY nothing except that figuring out a pattern for GithubCredentialsProvider-like things in general is in scope for #13082

[FreshlyBrewedCode]

Hi! We are currently running a Backstage PoC in our team and not having the ability to import from multiple Azure DevOps organizations is a major deal breaker for us. I am currently looking through the code and evaluating if it would be possible for me to come up with a basic implementation.

I would consider contributing but it looks like you are already considering a more general approach (similar to GitHubCredentialsProvider as @Rugvip mentioned). I don't have the time and overview of the codebase to contribute a generalized approach but I would rather focus specifically on the Azure integration (there is also an open issue related to multiple orgs in Azure #10431).

Would you consider accepting a PR that focuses on solving the problem for a single integration or are you already set on going with a more generalized approach?

[benjdlambert]

I think that we're going to head down the credential provider router either way, so I think go ahead and try to implement. Let's try to make it as similar as possible to the GithubCredentialsProvider something small like getCredentials({ url: string }) or whatever and see where we get to. :pray:

[awanlin]

@FreshlyBrewedCode if you do move forward with this please tag me in the PR as I'll gladly test this with Azure DevOps Server 2020, the on-prem version. Most of the time it's fine but never hurts to double check. I'll also help out with review 👍

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[FreshlyBrewedCode]

I already have a basic implementation to support multiple Azure DevOps Orgs in my fork. Just did not have time for further testing or to draft a PR during the last weeks. I hope I can look back into it in the coming weeks.

[LvffY]

@FreshlyBrewedCode It would be a great feature to have ! :)

Hope you'll have some time to share this feature :)

[Mercanuis]

Hi there.

My organization is also starting to explore Backstage and I was coming across this conversation via searching for "Backstage multiple repositories"

The context is that my team recently integrated with GH and is using the Discovery Tool to have Backstage scrape the repository for catalog-info YAML and host our documents. We would like to extend this to other teams in the same company but under separate organizations, repos, etc.

Similar to @LvffY 's prior comment, we have a GH App, and an Oauth App for the purposes of leveraging software templates, and have set up our integration secrets for Kubernetes deployment like so

integrations:
  github:
    - host: <MY_HOST>
      apiBaseUrl: https://<MY_HOST>/api/v3
      rawBaseUrl: https://<MY_HOST>/raw
      apps:
      - $include: ../secrets/secret-github-credentials.yaml

 ...

auth:
  providers: 
    github:
      $include: ../secrets/secret-github-oauth-credentials.yaml

catalog:
  readonly: true
  locations:
    - type: github-discovery
      target: https://<MY_HOST>/<ORG>/<REPO>/blob/main/catalog-info.yaml
    - type: github-org
      target: https://<MY_HOST>/<ORG>
      rules:
        - allow: [Group, User]

We are trying to solve the following use cases

• Can we scrape from a separate organization? <ORG>
• Can we scrape from targeted repos? <REPO>
• Do we need to provide separate credentials per organization?

I'm just trying to understand the nature of what's possible currently with Backstage. For what its worth, this would be a feature the company would be interested in for sure

Thanks! :)

[jhaals]

Hi @Mercanuis reading your config example it looks like you have a GitHub App configured and not different tokens per organization. Given that you have one GitHub App you should be able to ingest from multiple GitHub organizations as long as the GitHub App you use for authentication is installed in both.

[Mercanuis]

Hi @jhaals and thanks for the input. I could look into this. I wasn't aware I could do this (still fresh to learning how GH apps exactly work and this is code that we're still trying to further understand)

If this is the case, it might be possible already given our current setup for our org. thank you for letting me know

[jhaals]

@Mercanuis please give it a go and report back! Note that the app need to be installed in both organisations and I believe for it to be available the app must be made public

[Mercanuis]

@Mercanuis please give it a go and report back! Note that the app need to be installed in both organisations and I believe for it to be available the app must be made public

Thanks. I'll give this a try and see if that helps!

[Mercanuis]

@jhaals I was able to experiment and with the GH App installed on my company's GHE we were able to pull from multiple repositories. This fits our use case quite nicely. I wonder if there are implications with regards to the safety of using the app publically, but for now, this is a huge win. Thanks to the team for the advice

[fabiano-amaral]

Hi @Mercanuis reading your config example it looks like you have a GitHub App configured and not different tokens per organization. Given that you have one GitHub App you should be able to ingest from multiple GitHub organizations as long as the GitHub App you use for authentication is installed in both.

I've same question that @Mercanuis, can we have some problem with using GitHub apps public? Because I think that the only way to use the same github app across multiples organizations is changing the github app to public, correct?

[fabiano-amaral]

Ah, in time, I've a related question. If I have two organization and the same user belongs to both, we have a conflict, I saw in some issue that it is possible to customize the default namespace to reflect an org, but I didn't find this documentation, if someone can help some stuff to avoid, will be helpful

[Rugvip]

Wanna highlight that there's a configuration that allows you to limit the allowed owners (org or user) that an app can be used for, https://backstage.io/docs/integrations/github/github-apps#limiting-the-github-app-installations. It was missing from the config schema but I'm fixing that in #14682.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[marct83]

The issue I'm having is only being able to use one token for a gitlab host. Since token's differ between gitlab groups we can only scrape one group. Definitely not ideal.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bandgren]

I already have a basic implementation to support multiple Azure DevOps Orgs in my fork. Just did not have time for further testing or to draft a PR during the last weeks. I hope I can look back into it in the coming weeks.

@FreshlyBrewedCode, did you get anywhere with this? Saw your "local" fork and seems stale, did your org end up dropping Backstage or did you solve this in any other way? Or would you still like to push forward with your implementation?

[FreshlyBrewedCode]

@Bandgren Backstage is still used by our org but I only have little involvement in the project now. Also, this is currently not a must have feature for us. Last time I checked I got Azure DevOps multi org support working in my fork, however, I struggled to integrate the modified packages into our backstage instance for further testing.

[palktonderAtpowelno]

@FreshlyBrewedCode @jhaals @Rugvip We are using multiple Azure DevOps organizations and not having support for this in Backstage could be a showstopper for us. Seeing the code that @FreshlyBrewedCode have created in his fork, the solution seems so within reach, and I would gladly help in anyway I can to make sure to get a solution, but I'm so new to Backstage, its code and frankly React in general, so I can not do it on my own. Any suggestions on how we can move forward?

[awanlin]

This has come up at our Org recently but a slightly different use case - we want to use Azure DevOps Server 2020 on-prem as well as a Azure DevOps Services cloud instance. If I can find some time I'll try and work on this, no promises but it's something I've thought about on and off for some time now.

@FreshlyBrewedCode how would you feel if I used your changes as a base? I would co-sign the commits with you or make sure you were mentioned in the PR. This would help me get going faster

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Rugvip]

Closing this as a duplicate of #10431 as that's the original ask. Please create separate issues if you would like to see this outside of the Azure integration. This is being worked on in #18213

[Andrei-Predoiu]

@Rugvip You closed this as a duplicate of itself.