HTTP redirects breaks libvirt_volume.source

[fasmide]

System Information

Linux distribution

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

Terraform version

$ terraform -v
Terraform v1.3.7
on linux_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.7.1

Description of Issue/Question

I've been trying to have terraform create some archboxes using the qcow2 images provided by vagrant - these URLs redirect a few times and ultimately end up in some S3 bucket at AWS I think it was.

Setup

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
}

resource "libvirt_volume" "archlinux-qcow2" {
  name   = "archlinux-qcow2"
  pool   = "default"
  source = "https://app.vagrantup.com/archlinux/boxes/archlinux/versions/20230104.116125/providers/libvirt.box"
  format = "qcow2"
}

Steps to Reproduce Issue

$ terraform apply

Result

libvirt_volume.archlinux-qcow2: Creating...
╷
│ Error: error while getting Content-Length of "https://app.vagrantup.com/archlinux/boxes/archlinux/versions/20230104.116125/providers/libvirt.box": strconv.Atoi: parsing "": invalid syntax - got 
│ 
│   with libvirt_volume.archlinux-qcow2,
│   on main.tf line 15, in resource "libvirt_volume" "archlinux-qcow2":
│   15: resource "libvirt_volume" "archlinux-qcow2" {
│ 
╵

[fasmide]

Looking at libvirt/volume_image.go#L94

response, err := http.Head(i.url.String())
if err != nil {
    return 0, err
}

I think I got the title wrong, as the documentation clearly states that http.Head will follow up to 10 redirects.

As far as I can see, the mentioned URL does 4 redirects:

$ curl -v -L https://app.vagrantup.com/archlinux/boxes/archlinux/versions/20230104.116125/providers/libvirt.box 2> >(grep "HTTP/1.1")
> GET /archlinux/boxes/archlinux/versions/20230104.116125/providers/libvirt.box HTTP/1.1
< HTTP/1.1 302 Found
> GET /archlinux/boxes/archlinux/versions/20230104.116125/providers/download/libvirt.box HTTP/1.1
< HTTP/1.1 302 Found
> GET /v1/object/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJib3hlcy82NGNjZGNiMy01YzZkLTQ3YWUtOWNlOS1lNzgyMzcxZTgwM2QiLCJtb2RlIjoiciIsImV4cGlyZSI6MTY3MzQ3Nzc5NX0.z34E8jDTIKLZH7q8vjrAmKUoSXmKGs38Hdh90Tb4iGo HTTP/1.1
< HTTP/1.1 307 Temporary Redirect
> GET /archivist/boxes/64ccdcb3-5c6d-47ae-9ce9-e782371e803d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA6NDPRW4BQPUMO3GT%2F20230111%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230111T224136Z&X-Amz-Expires=900&X-Amz-Security-Token=FwoGZXIvYXdzEGgaDC0Fikpyw%2FSOkdJ5aSK3ARFecWXHJqqZ95ZgyOAIpFdwqoKIcqqlZms3kTIA5ehUdJGmIQ6O36Z2nhTKgbIWMIP9If2ZQl577ULIAjzWL3J4%2BJDK89JQL7hM%2Fve7F0r0z6we0l%2F3VCQdu51KCcNAXBTJ7DCXjSHsCJ%2FFHYWoLcGljkPR5bHuOj6xCZoikuNr2qICjGxqwwCOEtkHMQnMBSqn%2BG95VHMvEr%2FwR7rRgeJXjATg2NavYe2%2BAkqJZXISfYjiTet9Aijm9PydBjItrG2a5h916gwFhPvFi617WYQifF4XYAQSQTmfPRvsE%2BgZ570vCGvnhVaOn4uo&X-Amz-SignedHeaders=host&X-Amz-Signature=2893c009b97f9ce63ba02cd4f038b3397a9c06b0db0ca7969a3de55f08f17281 HTTP/1.1
< HTTP/1.1 200 OK

And this last 200 OK does indeed include a Content-Length header.

< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 495197110
< Connection: keep-alive
< x-amz-id-2: t0xymyPQVNGUC0lh0/Jc8cgKGNRLeKt7uuFB9kCa10hyDR1nOUcA5CGHojTATyBEK/lKaP7YBwQ=
< x-amz-request-id: JWTQNTXN4EXPNPSD
< Date: Wed, 11 Jan 2023 22:41:10 GMT
< Last-Modified: Wed, 04 Jan 2023 17:31:59 GMT
< ETag: "d1b84cae0adfe7f02930f0d6a1e90f4e"
< Accept-Ranges: bytes
< Server: AmazonS3
< X-Cache: Miss from cloudfront
< Via: 1.1 c2c75215aa2ab067e062055fa68a3fde.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: CPH50-C1
< X-Amz-Cf-Id: YxSxrJab2i2VDU3BDowXeBgii0zCDTYVA8VYMDA8EzFVbbORtHM0xw==

So I'm not quite sure whats going on

[n-able-consulting]

Think same issue here. Since yesterday evening, on multiple systems, without a local code change (i rebuild terraform and providers, with terraform init). My scripts keep hanging: libvirt_volume.w-base-vol[6]: Creating... libvirt_volume.second_disk[3]: Creating... libvirt_volume.second_disk[4]: Creating... libvirt_volume.second_disk[5]: Creating... libvirt_volume.second_disk[2]: Creating... libvirt_volume.second_disk[5]: Creation complete after 0s [id=/vm/w6.k106-second-disk] libvirt_volume.w-base-vol[0]: Creating... libvirt_volume.second_disk[6]: Still creating... [10s elapsed] libvirt_volume.w-base-vol[5]: Still creating... [10s elapsed] libvirt_volume.second_disk[0]: Still creating... [10s elapsed] libvirt_volume.w-base-vol[6]: Still creating... [10s elapsed] libvirt_volume.second_disk[3]: Still creating... [10s elapsed] libvirt_volume.second_disk[4]: Still creating... [10s elapsed] libvirt_volume.second_disk[1]: Still creating... [10s elapse ..... libvirt_volume.second_disk[3]: Still creating... [17m10s elapsed] libvirt_volume.w-base-vol[6]: Still creating... [17m10s elapsed] libvirt_volume.second_disk[2]: Still creating... [17m10s elapsed] libvirt_volume.second_disk[1]: Still creating... [17m10s elapsed] libvirt_volume.second_disk[6]: Still creating... [17m10s elapsed] libvirt_volume.w-base-vol[0]: Still creating... [17m10s elapsed]

Without succeeding

[fasmide]

@n-able-consulting, while annoying - i don't think its the same issue - when using the arch box vagrant images, terraform fails instantly;

Terraform will perform the following actions:

  # libvirt_volume.archlinux-qcow2 will be created
  + resource "libvirt_volume" "archlinux-qcow2" {
      + format = "qcow2"
      + id     = (known after apply)
      + name   = "archlinux-qcow2"
      + pool   = "default"
      + size   = (known after apply)
      + source = "https://app.vagrantup.com/archlinux/boxes/archlinux/versions/20230104.116125/providers/libvirt.box"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

libvirt_volume.archlinux-qcow2: Creating...
╷
│ Error: error while getting Content-Length of "https://app.vagrantup.com/archlinux/boxes/archlinux/versions/20230104.116125/providers/libvirt.box": strconv.Atoi: parsing "": invalid syntax - got 
│ 
│   with libvirt_volume.archlinux-qcow2,
│   on main.tf line 15, in resource "libvirt_volume" "archlinux-qcow2":
│   15: resource "libvirt_volume" "archlinux-qcow2" {
│ 
╵

[fasmide]

Digging around with mitmproxy, the problem lies in the vagrantup.com web service, which directs users to the s3 buckets. When presented with a HEAD request, it eventually sends a 200 OK and includes a "Location: " header. It should of course send a 307 TemporaryRedirect together with the "Location: " header - which it does when presented with a GET request.

I don't think there's much for this provider to do about that

Furthermore, the builds provided by vagrantup are actually gzipped tarballs and need to be extracted in order for libvirt to use so - this is going essentially nowhere the way I wanted it to :)

Feel free to close this issue