Error when installing plugins through mirror

[LongLiveCHIEF]

Description

Looks like i missed some things when we added the ability for plugins to download through a mirror. During the install, we're getting a did not respond with an X-Version header errror, which I believe is originating from the https://github.com/apollographql/rover/blob/main/installers/binstall/src/install.rs#L77-L102

Looks like it's using a client that blocks redirects, which won't work if downloading the plugin through a mirror.

Steps to reproduce

1. Setup a proxy-mirror for rover.apollo.dev
2. run rover install --plugin supergraph@latest-2

Expected result

Plugin should download and install

Actual result

Error: did not respond with an X-Version header

Environment

Rover v0.18.1

[LongLiveCHIEF]

I'm thinking we can change to a custom redirect policy. From reqwest docs - struct Policy

let custom = redirect::Policy::custom(|attempt| {
    if attempt.previous().len() > 5 {
        attempt.error("too many redirects")
    } else if attempt.url().host_str() == Some("example.domain") {
        // prevent redirects to 'example.domain'
        attempt.stop()
    } else {
        attempt.follow()
    }
});
let client = reqwest::Client::builder()
    .redirect(custom)
    .build()?;

[EverlastingBugstopper]

Redirects are definitely allowed by our client - Rover is just expecting to be able to parse the version of the binary from that header. The requests Rover makes during the plugin install look like this:

1) HEAD to https://rover.apollo.dev/tar/supergraph/x86_64-apple-darwin/v2.0.0 which parses the X-Version header (which is v2.0.0) and returns a 302 1) GET to the same URL which returns a 302 with a Location header equal to https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.0.0/supergraph-v2.0.0-x86_64-apple-darwin.tar.gz 1) GET to that URL which returns yet another redirect to an https://objects.githubusercontent.com URL with X-Amz-Algorithm and X-Amz-Credential URL parameters 1) One last GET to the githubusercontent.com URL which finally results in a 200 and the binary content.

[LongLiveCHIEF]

It's this part which is confusing me: https://github.com/apollographql/rover/blob/e2ba8f1756d2c403b35ab21c1a370c43d769c5f5/installers/binstall/src/install.rs#L79

The client var is even called the no_redirect_client

[LongLiveCHIEF]

My first thought was to check to make sure we weren't stripping headers by our network appliances, but when I saw the no_redirect_client I just figured naturally that that was the issue since the tarbal_url was originally derived from rover.apollo.dev.

I just figured the rover.apollo.dev did steps 2 and 3 on their server instead of serving 302 responses and attaching new headers.

[EverlastingBugstopper]

Oh yeah that no_redirect_client is so it does that initial HEAD request at first - if you try to do a HEAD request with the redirect policy set you can't read the header value it gives back. Subsequent requests (that actually get the binary) do follow redirects for sure.

The reason that HEAD request was implemented was in order to get the version of a latest plugin (like supergraph@latest) - it's not super useful when installing a pinned version. A change could be made to only execute that HEAD request when installing the latest plugin version rather than a pinned version, or the mirror could be configured to allow HEAD requests and include that version header.

And no, the rover.apollo.dev endpoint does not resolve any redirects - it just returns them directly. I attached HTTP Toolkit to my terminal and ran the command to see the request/response flow as Rover sent/received them:

[LongLiveCHIEF]

or the mirror could be configured to allow HEAD requests and include that version header.

Even if the mirror supports HEAD requests... the request made through a mirror will return the headers for the mirror and not the proxied repo, since the destination url in a mirrored request is the mirror and not the upstream binary source repo.

[LongLiveCHIEF]

could we not use the github api instead of doing a HEAd request, to do latest release in a get request? This would allow mirror/proxies to work and would still return version information, along with a whole bunch of other useful information (like commit id/sha, etc...)

example: https://api.github.com/repos/apollographql/federation-rs/releases/latest

{
  "url": "https://api.github.com/repos/apollographql/federation-rs/releases/119498753",
  "assets_url": "https://api.github.com/repos/apollographql/federation-rs/releases/119498753/assets",
  "upload_url": "https://uploads.github.com/repos/apollographql/federation-rs/releases/119498753/assets{?name,label}",
  "html_url": "https://github.com/apollographql/federation-rs/releases/tag/supergraph%40v2.5.4",
  "id": 119498753,
  "author": {
    "login": "apollo-bot2",
    "id": 50160705,
    "node_id": "MDQ6VXNlcjUwMTYwNzA1",
    "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/apollo-bot2",
    "html_url": "https://github.com/apollo-bot2",
    "followers_url": "https://api.github.com/users/apollo-bot2/followers",
    "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
    "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
    "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
    "repos_url": "https://api.github.com/users/apollo-bot2/repos",
    "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
    "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
    "type": "User",
    "site_admin": false
  },
  "node_id": "RE_kwDOGit3qc4HH2gB",
  "tag_name": "supergraph@v2.5.4",
  "target_commitish": "main",
  "name": "supergraph@v2.5.4",
  "draft": false,
  "prerelease": false,
  "created_at": "2023-08-31T18:24:04Z",
  "published_at": "2023-08-31T18:51:14Z",
  "assets": [
    {
      "url": "https://api.github.com/repos/apollographql/federation-rs/releases/assets/123988018",
      "id": 123988018,
      "node_id": "RA_kwDOGit3qc4HY-gy",
      "name": "md5sums.txt",
      "label": "",
      "uploader": {
        "login": "apollo-bot2",
        "id": 50160705,
        "node_id": "MDQ6VXNlcjUwMTYwNzA1",
        "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/apollo-bot2",
        "html_url": "https://github.com/apollo-bot2",
        "followers_url": "https://api.github.com/users/apollo-bot2/followers",
        "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
        "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
        "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
        "repos_url": "https://api.github.com/users/apollo-bot2/repos",
        "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
        "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "text/plain; charset=utf-8",
      "state": "uploaded",
      "size": 330,
      "download_count": 0,
      "created_at": "2023-08-31T18:51:12Z",
      "updated_at": "2023-08-31T18:51:13Z",
      "browser_download_url": "https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.5.4/md5sums.txt"
    },
    {
      "url": "https://api.github.com/repos/apollographql/federation-rs/releases/assets/123988016",
      "id": 123988016,
      "node_id": "RA_kwDOGit3qc4HY-gw",
      "name": "sha1sums.txt",
      "label": "",
      "uploader": {
        "login": "apollo-bot2",
        "id": 50160705,
        "node_id": "MDQ6VXNlcjUwMTYwNzA1",
        "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/apollo-bot2",
        "html_url": "https://github.com/apollo-bot2",
        "followers_url": "https://api.github.com/users/apollo-bot2/followers",
        "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
        "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
        "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
        "repos_url": "https://api.github.com/users/apollo-bot2/repos",
        "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
        "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "text/plain; charset=utf-8",
      "state": "uploaded",
      "size": 362,
      "download_count": 0,
      "created_at": "2023-08-31T18:51:12Z",
      "updated_at": "2023-08-31T18:51:12Z",
      "browser_download_url": "https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.5.4/sha1sums.txt"
    },
    {
      "url": "https://api.github.com/repos/apollographql/federation-rs/releases/assets/123988014",
      "id": 123988014,
      "node_id": "RA_kwDOGit3qc4HY-gu",
      "name": "sha256sums.txt",
      "label": "",
      "uploader": {
        "login": "apollo-bot2",
        "id": 50160705,
        "node_id": "MDQ6VXNlcjUwMTYwNzA1",
        "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/apollo-bot2",
        "html_url": "https://github.com/apollo-bot2",
        "followers_url": "https://api.github.com/users/apollo-bot2/followers",
        "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
        "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
        "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
        "repos_url": "https://api.github.com/users/apollo-bot2/repos",
        "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
        "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "text/plain; charset=utf-8",
      "state": "uploaded",
      "size": 458,
      "download_count": 0,
      "created_at": "2023-08-31T18:51:12Z",
      "updated_at": "2023-08-31T18:51:12Z",
      "browser_download_url": "https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.5.4/sha256sums.txt"
    },
    {
      "url": "https://api.github.com/repos/apollographql/federation-rs/releases/assets/123988015",
      "id": 123988015,
      "node_id": "RA_kwDOGit3qc4HY-gv",
      "name": "supergraph-v2.5.4-aarch64-unknown-linux-gnu.tar.gz",
      "label": "",
      "uploader": {
        "login": "apollo-bot2",
        "id": 50160705,
        "node_id": "MDQ6VXNlcjUwMTYwNzA1",
        "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/apollo-bot2",
        "html_url": "https://github.com/apollo-bot2",
        "followers_url": "https://api.github.com/users/apollo-bot2/followers",
        "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
        "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
        "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
        "repos_url": "https://api.github.com/users/apollo-bot2/repos",
        "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
        "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-gtar",
      "state": "uploaded",
      "size": 20391287,
      "download_count": 0,
      "created_at": "2023-08-31T18:51:12Z",
      "updated_at": "2023-08-31T18:51:13Z",
      "browser_download_url": "https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.5.4/supergraph-v2.5.4-aarch64-unknown-linux-gnu.tar.gz"
    },
    {
      "url": "https://api.github.com/repos/apollographql/federation-rs/releases/assets/123988017",
      "id": 123988017,
      "node_id": "RA_kwDOGit3qc4HY-gx",
      "name": "supergraph-v2.5.4-x86_64-apple-darwin.tar.gz",
      "label": "",
      "uploader": {
        "login": "apollo-bot2",
        "id": 50160705,
        "node_id": "MDQ6VXNlcjUwMTYwNzA1",
        "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/apollo-bot2",
        "html_url": "https://github.com/apollo-bot2",
        "followers_url": "https://api.github.com/users/apollo-bot2/followers",
        "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
        "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
        "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
        "repos_url": "https://api.github.com/users/apollo-bot2/repos",
        "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
        "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-gtar",
      "state": "uploaded",
      "size": 16904088,
      "download_count": 2,
      "created_at": "2023-08-31T18:51:12Z",
      "updated_at": "2023-08-31T18:51:13Z",
      "browser_download_url": "https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.5.4/supergraph-v2.5.4-x86_64-apple-darwin.tar.gz"
    },
    {
      "url": "https://api.github.com/repos/apollographql/federation-rs/releases/assets/123988021",
      "id": 123988021,
      "node_id": "RA_kwDOGit3qc4HY-g1",
      "name": "supergraph-v2.5.4-x86_64-pc-windows-msvc.tar.gz",
      "label": "",
      "uploader": {
        "login": "apollo-bot2",
        "id": 50160705,
        "node_id": "MDQ6VXNlcjUwMTYwNzA1",
        "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/apollo-bot2",
        "html_url": "https://github.com/apollo-bot2",
        "followers_url": "https://api.github.com/users/apollo-bot2/followers",
        "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
        "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
        "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
        "repos_url": "https://api.github.com/users/apollo-bot2/repos",
        "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
        "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-gtar",
      "state": "uploaded",
      "size": 14005669,
      "download_count": 2,
      "created_at": "2023-08-31T18:51:13Z",
      "updated_at": "2023-08-31T18:51:14Z",
      "browser_download_url": "https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.5.4/supergraph-v2.5.4-x86_64-pc-windows-msvc.tar.gz"
    },
    {
      "url": "https://api.github.com/repos/apollographql/federation-rs/releases/assets/123988022",
      "id": 123988022,
      "node_id": "RA_kwDOGit3qc4HY-g2",
      "name": "supergraph-v2.5.4-x86_64-unknown-linux-gnu.tar.gz",
      "label": "",
      "uploader": {
        "login": "apollo-bot2",
        "id": 50160705,
        "node_id": "MDQ6VXNlcjUwMTYwNzA1",
        "avatar_url": "https://avatars.githubusercontent.com/u/50160705?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/apollo-bot2",
        "html_url": "https://github.com/apollo-bot2",
        "followers_url": "https://api.github.com/users/apollo-bot2/followers",
        "following_url": "https://api.github.com/users/apollo-bot2/following{/other_user}",
        "gists_url": "https://api.github.com/users/apollo-bot2/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/apollo-bot2/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/apollo-bot2/subscriptions",
        "organizations_url": "https://api.github.com/users/apollo-bot2/orgs",
        "repos_url": "https://api.github.com/users/apollo-bot2/repos",
        "events_url": "https://api.github.com/users/apollo-bot2/events{/privacy}",
        "received_events_url": "https://api.github.com/users/apollo-bot2/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-gtar",
      "state": "uploaded",
      "size": 20847509,
      "download_count": 16,
      "created_at": "2023-08-31T18:51:13Z",
      "updated_at": "2023-08-31T18:51:14Z",
      "browser_download_url": "https://github.com/apollographql/federation-rs/releases/download/supergraph%40v2.5.4/supergraph-v2.5.4-x86_64-unknown-linux-gnu.tar.gz"
    }
  ],
  "tarball_url": "https://api.github.com/repos/apollographql/federation-rs/tarball/supergraph@v2.5.4",
  "zipball_url": "https://api.github.com/repos/apollographql/federation-rs/zipball/supergraph@v2.5.4",
  "body": "This release was automatically created by [CircleCI](./.circleci/config.yml).\n\nIf you would like to verify that the binary you have downloaded was built from the source code in this repository, you can compute a checksum of the zipped tarball and compare it to the checksums that are included as release artifacts.\n\nBinaries built for MacOS are signed, notarized, and automatically verified with [Gatekeeper](https://support.apple.com/guide/deployment-reference-macos/using-gatekeeper-apd02b925e38/web)."
}

[EverlastingBugstopper]

No, we don't use the GitHub API from Rover directly so that we retain control over DNS should we ever need to migrate away from GitHub releases as the primary binary store. This isn't a future we anticipate but the flexibility provided by rover.apollo.dev is intentional and not something we plan to change.

[LongLiveCHIEF]

I meant use the github api via orbiter. A releases api that returns json with a release tag wouldn't be that big of a deal to stand up compatibility wise if you ever did need to migrate away, and it's vastly more flexible and extensible than HTTP header data.

I'm afraid of too much hidden functionality being depended on by the --skip-update flag.

[EverlastingBugstopper]

I see. I'll be honest - it's not likely that we'll prioritize adding that functionality to orbiter. The existing HEAD requests are nice and lightweight and it already works properly.

[LongLiveCHIEF]

it's not super useful when installing a pinned version. A change could be made to only execute that HEAD request when installing the latest plugin version rather than a pinned version

I guess this is the best path forward then.

In regards to the router plugin for rover, the "pinned version" is just using --skip-update while already having the plugin installed, right? Is there a configuration way to designate the plugin version for router?

If not, then it becomes a chicken-egg problem, where in order for the HEAD request to be avoided, you have to have installed the plugin using latest, which would require the use of the HEAD request.

Granted, I'm not terribly concerned about router, because that's easy enough to run with docker.

[EverlastingBugstopper]

Sure, so the router plugin is only used for rover dev, and that can be "pinned" with the APOLLO_ROVER_DEV_ROUTER_VERSION environment variable (docs here). (similarly, APOLLO_ROVER_DEV_COMPOSITION_VERSION will pin the composition version).

There's no need to use --skip-update when running rover dev if you have these environment variables set, rover will look on the file system for that plugin before doing any HTTP calls, only reaching out to orbiter if it's not already installed.

You could really skip the whole HEAD request thing completely when hitting the mirror if you don't use rover install --plugin {supergraph|router}@{version} and just download the binary to ~/.rover/bin/{supergraph|router}-v{version} directly, but also happy to make a change so that the plugin install for a specific version skips that HEAD request.

[VannAtApollo]

Is it safe to say that, considering all the context here, we are unblocked and okay to continue with the POC?

@LongLiveCHIEF Would you like this change to be implemented ?

[LongLiveCHIEF]

@VannAtApollo yes, that would be great.