Response from github is 403 when downloading a release file.

[praveenkumar]

Not able to download release artifact from GitHub.

package main

import (
    "fmt"
    "log"

    "github.com/cavaliercoder/grab"
)

func main() {
    client := grab.NewClient()
    req, err := grab.NewRequest("", "https://github.com/minishift/minishift-centos-iso/releases/download/v1.12.0/minishift-centos7.iso")
    if err != nil {
        log.Fatal(err)
    }
    resp := client.Do(req)
    fmt.Printf("Response is: %v\n", resp.HTTPResponse.Status)
}

Unexpected one.

==== Output ====
$ go run test.go
Response is: 403 Forbidden

[cavaliercoder]

I can replicate this issue in curl:

$ curl -I "https://github.com/minishift/minishift-centos-iso/releases/download/v1.12.0/minishift-centos7.iso"
HTTP/1.1 302 Found
Date: Wed, 19 Sep 2018 17:27:42 GMT
Content-Type: text/html; charset=utf-8
Server: GitHub.com
Status: 302 Found
Cache-Control: no-cache
Vary: X-PJAX
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/69853246/922b03d8-8452-11e8-9dcd-92e33542c976?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180919%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180919T172742Z&X-Amz-Expires=300&X-Amz-Signature=ad8d7a78a3d95ebded5cc46af3d4ec23485a4b1f33a045f24a25e5358bdf26fb&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dminishift-centos7.iso&response-content-type=application%2Foctet-stream
Set-Cookie: has_recent_activity=1; path=/; expires=Wed, 19 Sep 2018 18:27:42 -0000
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Sun, 19 Sep 2038 17:27:42 -0000; secure; HttpOnly
Set-Cookie: _gh_sess=UnU5c0hUbHdjZmZoRjE4OWlCeHF0VWY2VzVUWXhVdWFEK3VFK1ZTRkhBMDZNWllqMWFydHFaMW5MNXZHTzBOM3UvVmQ4RC9ybTlSWUZQeHNsdlZ1cUROVHNKbGZTbWZPd3R2S3BXNVduaXY5ZHZnNUhPaUQ1V1dMMTd4aVR3N2VZcVljVGoxWWsrbGZ4MWRxeWVQMDdaSXNTRUdGeGNtZFp4RTdncXFNTWdwRTZqZkVMdjV5ZXR6OHUrdUdGUXJ6N1gzWGpETFFDaTM5dWhlT3lUeXNadz09LS03VE84ZGQ1aEdseW5KTlhWNldhc2FRPT0%3D--dec4ca32f8128b22491f9d221904b9006d8d06fe; path=/; secure; HttpOnly
X-Request-Id: 0c4332a1-bc17-4780-88ff-b98db1b12b57
X-Runtime: 0.061618
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com
X-Runtime-rack: 0.068508
Vary: Accept-Encoding
X-GitHub-Request-Id: F6BD:0E72:8FB4FC:CB87F2:5BA2870E

# following the redirect:

$ curl "https://github-production-release-asset-2e65be.s3.amazonaws.com/69853246/922b03d8-8452-11e8-9dcd-92e33542c976?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180919%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180919T172545Z&X-Amz-Expires=300&X-Amz-Signature=741823d55e97c378a0bf5f8414c2856d57f4747198c5ce1f5aabfb5d906565e9&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dminishift-centos7.iso&response-content-type=application%2Foctet-stream"
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationQueryParametersError</Code><Message>Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.</Message><RequestId>8BE04D0D70220D4E</RequestId><HostId>tUUqB4EFLgK8yPMPRZg6oBBqHQ9lEd0DlZCt+b/dqhU2LyKPIuDEuicjFEZbsKyUqKQ4uwTc32Y=</HostId></Error>

However, trying again a few times, it sometimes works. Retrying with grab was unsuccessful.

I suspect this is an issue with the URL parameter encoding. I'll take a closer look.

[praveenkumar]

@cavaliercoder thanks, we are planning to consume this to our minishift project (https://github.com/minishift/minishift) to enable resume download functionality for artifacts which comes from GitHub.

[cmaglie]

It's not an encoding problem, it's GitHub's service on AWS that doesn't allow for HEAD requests:

$ curl -s -I https://github.com/igrr/esptool-ck/releases/download/0.4.13/esptool-0.4.13-linux64.tar.gz | grep Location
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/28329151/548ca91e-141d-11e8-9fdc-f4ecf656b364?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20181018%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181018T162441Z&X-Amz-Expires=300&X-Amz-Signature=493e902d0182277b83f6ae5b48e26f733de16b3325784336ff95ebf1107c5256&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Desptool-0.4.13-linux64.tar.gz&response-content-type=application%2Foctet-stream

$ curl -I "https://github-production-release-asset-2e65be.s3.amazonaws.com/28329151/548ca91e-141d-11e8-9fdc-f4ecf656b364?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20181018%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181018T162441Z&X-Amz-Expires=300&X-Amz-Signature=493e902d0182277b83f6ae5b48e26f733de16b3325784336ff95ebf1107c5256&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Desptool-0.4.13-linux64.tar.gz&response-content-type=application%2Foctet-stream"
HTTP/1.1 403 Forbidden
x-amz-request-id: 557B9E78EBFEC255
x-amz-id-2: /xreZi/OaeYFgfXvlEA+MCGzu9kaJlRtimj8j+IX7oULJys2ls0Drq1tkacK6O2ipM32nsim8ek=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Thu, 18 Oct 2018 16:25:54 GMT
Server: AmazonS3

$ curl -i "https://github-production-release-asset-2e65be.s3.amazonaws.com/28329151/548ca91e-141d-11e8-9fdc-f4ecf656b364?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20181018%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181018T162441Z&X-Amz-Expires=300&X-Amz-Signature=493e902d0182277b83f6ae5b48e26f733de16b3325784336ff95ebf1107c5256&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Desptool-0.4.13-linux64.tar.gz&response-content-type=application%2Foctet-stream" 
HTTP/1.1 200 OK
x-amz-id-2: 5PHul+TN8/85SKelKJCH83MY6nTmk02sm00UcK18VpiMTE/3V9mzqULHiapXXC8AbZwd1sNQuQY=
x-amz-request-id: C0B02D042431843D
Date: Thu, 18 Oct 2018 16:27:48 GMT
Last-Modified: Sat, 17 Feb 2018 12:01:25 GMT
ETag: "91b1af2f38d729ba4908b0551f2b99d9"
Content-Disposition: attachment; filename=esptool-0.4.13-linux64.tar.gz
Accept-Ranges: bytes
Content-Type: application/octet-stream
Content-Length: 15743
Server: AmazonS3

Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.

so on the same url a HEAD request is forbidden but the GET get through. I don't know if this is a bug on AWS or in this library. I see that this library do a HEAD request to get webserver capabilities, maybe errors on HEAD should be ignored and continue anyway?

[cavaliercoder]

I think this is a bug. Failed HEAD requests should be ignored but following the code, the 403 is returned to the caller.

This needs a patch and tests. Will try to address this soon, but PRs are welcomed!

[cavaliercoder]

This should be fixed now. Thanks @cmaglie for the winning tip! Please test and advise.

[praveenkumar]

Thanks @cmaglie , I will also give it another try and see if that works.