Submodules are resolved into incorrect URL

chrislovett03 commented 4 years ago

Sourcegraph version 3.21.2

When browsing submodules under a meta-repo - users can view the main/trunk meta-repo, the commit dropdown lists an accurate history of commits. When the user clicks on a given commit, nothing happens. Meta-repo has submodules, the submodules are searchable directly but not when navigating through the meta-repo. The desired behavior is access the meta-repo, search at a universe hash, and see the submodules and their corresponding commits.

Our typical handling of submodules is, if we detect them, and we have the submodule repo itself synced, we simply list them in the file tree with a custom icon, and clicking on that entry takes the user to the submodule on Sourcegraph [https://sourcegraph.com/github.com/ezEngine/submodule-example]

The user sees the submodule in the file tree, with a valid commit hash, but clicking on it doesn’t do anything. The URL doesn’t change, the page doesn’t change, etc..

One hint: the user is using several repo name transformations in their GitLab connection config (e.g. to remove a .d-git from the end of repo names), which appear to not be respected by the submodule link in the file tree when the user hovers over it. Perhaps a place to check to start?

We can gather debug info if needed.

Customer: https://app.hubspot.com/contacts/2762526/company/578600789

github-actions[bot] commented 4 years ago

Heads up @tsenart - the "team/cloud" label was applied to this issue.

tsenart commented 4 years ago

@keegancsmith: Can you provide some direction please?

flying-robot commented 4 years ago

Slack convo: https://sourcegraph.slack.com/archives/CHPC7UX16/p1604350347483800

flying-robot commented 4 years ago

@christinelovett I chatted w/@tsenart and discussed a few bits of debug info that would be helpful to get:

what specific transformations are being applied?
are they using repositoryPathPattern?
can we get their external service configuration (w/secrets elided)?
can we get the clone URL for the submodule?

chrislovett03 commented 4 years ago

Thanks for looking into it @flying-robot! I'll check with the user and get you that info if possible.

chrislovett03 commented 4 years ago

@flying-robot :

GitLab config is:

{
  "projectQuery": [
    "?membership=false"
  ],
  "token": "redacted",
  "url": "redacted",
  "repositoryPathPattern": "{pathWithNamespace}",
  "nameTransformations": [
      {
        "regex": "\\.d/",
        "replacement": "/"
      },
      {
        "regex": "-git$",
        "replacement": ""
      }
    ]
}

And an example of a submodule path as it appears in the browser: <sg-url>/<project>/<subdir1>/<subdir2>/<submodule>

Example of our top level meta repo as it appears in the browser: <sg-url>/<project>/<meta-repo>

From the meta repo level, if I click on a subdir the url as it appears in the browser: <sg-url>/<project>/<meta-repo>/-/tree/<subdir1>

And from the metarepo level, with a subdir open, if I hover over a submodule that looks like: <submodule> @ 09sk323

The url that pops up in the hover box is: Submodule: ../../<project>/<subdir1>.d/<subdir2>.d/<submodule>-git

And these submodules do nothing when we click on them.

When trying to click on submodules in the meta-repo, we see errors from the frontend service like:

t=2020-11-02T18:25:36+0000 lvl=warn msg="linksForRepository failed to RepoLookup" repo=<project>/<meta-repo> error="Post \"http://repo-updater:<port>/repo-lookup\": context canceled"

t=2020-11-02T18:37:10+0000 lvl=eror msg="Failed to resolve submodule repository name from clone URL" cloneURL=../../<project>/<subdir1>.d/<submodule>-git err="No matching code host found for ../../<project>/<subdir1>.d/<submodule>-git"

flying-robot commented 4 years ago

@christinelovett awesome, this is extremely helpful. I believe that the issue might be that their submodule is configured via a local/relative path and not a URL. Would it be possible to get the contents of their .gitmodules file to confirm?

As an example, I have a repo on GitLab right now with a submodule configured like so:

[submodule "test-repo-2"]
        path = test-repo-2
        url = ../test-repo-2

I've also configured an external service locally to match their overall config. When the frontend tries to resolve that repo of mine, the error message is as follows and I get all the behavior described (the link doesn't do anything, etc.):

17:10:01 frontend | ERROR Failed to resolve submodule repository name from clone URL, cloneURL: , err: No matching code host found for

chrislovett03 commented 3 years ago

@flying-robot reopening this issue per customer's note:

That was causing link rendering to break since it wasn't able to resolve the code host URL with the repo, resulting in submodule links in the UI that went nowhere.

I think that sounds like part of the problem. We can click on links in the repo file dir and they go to the right place

However, the problem is that the file dir doesn't load in the first place (for many folders). So when I try to click on main/foo/bar to open up the folder, it leads to: Request to https://sourcegraph.com/.api/graphql?TreeEntries failed with 504 Gateway Time-out

Performance issues: for some file dir it opens but it takes a few minutes

muratsu commented 3 years ago

Another instance of this bug can be observed in the freedesktop-sdk repository.

Navigate to https://sourcegraph.com/gitlab.com/freedesktop-sdk/freedesktop-sdk
From the file nav bar, navigate to Utils > buildstream-abi-checker

Expected

Submodule to be displayed

Actual

We try to load the submodule on github.com and give a repository not found error
The following url is the resolved url: https://sourcegraph.com/github.com/buildstream-abi-checker@a44550a1c65c527fb9d337852b48ad5e82b9a842

chrislovett03 commented 3 years ago

This is the console error being returned:

Could not resolve file info for code view RepoNotFoundError: repo gitlab.companyname.com/main/foo.d/bar.d/app-git not found
    at t.project (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:8:158)
    at t._next (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:2:18982)
    at t.next (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:2:3281)
    at t._next (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:2:19080)
    at t.next (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:2:3281)
    at t.notifyNext (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:59:30607)
    at t._next (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:2:17825)
    at t.next (chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:2:3281)
    at chrome-extension://dgjhfomjieaadpoljlnidmbgkdffpack/js/inject.bundle.js:59:2311

chrislovett03 commented 3 years ago

Hey @muratsu any movement on this? Any info we can collect from the customer?

muratsu commented 3 years ago

@unknwon is looking into this

unknwon commented 3 years ago

Thanks @muratsu! Your example is super helpful. I can already tell for some reasons, we resolved it to GitHub.com (should be GitLab.com), and is missing the "owner" (ie. "freedesktop-sdk").

unknwon commented 3 years ago

Confirmed working for https://sourcegraph.com/gitlab.com/freedesktop-sdk/freedesktop-sdk

muratsu commented 3 years ago

The cusomer is still experiencing this issue on 3.28. Adding relevant slack thread: https://sourcegraph.slack.com/archives/C01LZKLRF0C/p1626118429460000?thread_ts=1625769829.424600&cid=C01LZKLRF0C

marcleblanc2 commented 1 year ago

Customer 8205 is requesting submodule support for GitLab repos in support ticket 13881.

They're also using nameTransformations, and on v5.0.1.

Can we please reprioritize this issue? 🙏

sourcegraph / sourcegraph-public-snapshot

Submodules are resolved into incorrect URL #15286