hashicorp / go-getter

Package for downloading things from a string URL using a variety of protocols.
Mozilla Public License 2.0
1.65k stars 232 forks source link

https download fails for a valid URL #438

Open Subhajit97 opened 1 year ago

Subhajit97 commented 1 year ago

I am trying to download a CSV file from a valid URL, i.e. https://cdn.wsform.com/wp-content/uploads/2020/06/industry.csv. Using the browser, I am able to download the file, but when I try using Go-getter, it returns no source URL was returned.

I'm using v1.7.1 version of the go-getter.

Below is the code snippet I'm using:

err := getter.Get("<dest>", "https://cdn.wsform.com/wp-content/uploads/2020/06/industry.csv")
if err != nil {
    log.Println(err)
    return
}

Error:

error downloading 'https://cdn.wsform.com/wp-content/uploads/2020/06/industry.csv': no source URL was returned

Is there anything I'm missing here, or it is a bug?

Subhajit97 commented 1 year ago

I tried the same using the GetAny method instead of Get, and I'm able to download the files using that URL. But, the GetAny method is not working for S3 URLs, whereas I was able to download files using the Get method.

Using the Get method (working)

err := getter.Get("<dest>", "s3::https://cloudformation-templates-ap-southeast-1.s3.ap-southeast-1.amazonaws.com/")
if err != nil {
    log.Println(err)
    return
}

Using the GetAny method (fails)

err := getter.GetAny("<dest>", "s3::https://cloudformation-templates-ap-southeast-1.s3.ap-southeast-1.amazonaws.com/")
if err != nil {
    log.Println(err)
    return
}

Error:

InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, GetObjectInput.Key.

Is there anything I'm missing here? Are there any workaround for this?