400 when downloading provider teleport on another registry

[seboudry]

Hi!

We use teleport provider, and it is hosted at their registry terraform.releases.teleport.dev.

Requests for available versions and version metadat are working. But the one to download the provider binary returned a 400 error.

Commands executed inside hermitcrab pod:

# curl -k https://localhost/v1/providers/terraform.releases.teleport.dev/gravitational/teleport/index.json
{"versions":{"0.0.0":{},"10.0.0":{},"10.0.2":{},"10.1.2":{},"10.1.4":{},"10.1.9":{},"10.2.1":{},"10.2.2":{},"10.2.4":{},"10.2.6":{},"10.2.7-roman.1":{},"10.3.1":{},"10.3.2":{},"11.0.0":{},"11.0.0-dev-walt.1":{},"11.0.0-dev-walt.2":{},"11.0.0-dev-walt.6":{},"11.0.0-dev-walt.7":{},"11.0.1":{},"11.0.3":{},"11.1.0":{},"11.1.1":{},"11.1.2":{},"11.1.4":{},"11.2.1":{},"11.2.2":{},"11.2.3":{},"11.3.1":{},"11.3.2":{},"12.0.1":{},"12.0.2":{},"12.0.4":{},"12.0.5":{},"12.1.0":{},"12.1.1":{},"12.1.5":{},"12.2.1":{},"12.2.3":{},"12.2.4":{},"12.3.1":{},"13.0.0":{},"13.0.2":{},"13.0.3":{},"13.0.4":{},"13.1.0":{},"13.1.1":{},"13.1.5":{},"13.2.0":{},"13.2.1":{},"13.2.2":{},"13.2.2-hugoslackloop":{},"13.2.3":{},"13.2.5":{},"13.3.0":{},"13.3.1":{},"13.3.2":{},"13.3.4":{},"13.3.5":{},"13.3.6":{},"13.3.7":{},"13.3.8":{},"14.0.0":{},"14.0.1":{},"14.0.3":{},"14.1.0":{},"14.1.5":{},"14.2.0":{},"14.2.1":{},"14.2.3":{},"14.3.0":{},"14.3.1":{},"14.3.2":{},"14.3.3":{},"15.0.0":{},"15.0.1":{},"15.0.2":{},"15.1.1":{},"15.1.10":{},"15.1.4":{},"15.1.6":{},"15.1.8":{},"15.1.9":{},"15.2.0":{},"15.2.1":{},"15.2.4":{},"15.3.0":{},"15.3.1":{},"15.3.6":{},"15.3.7":{},"15.4.0":{},"15.4.10":{},"15.4.11":{},"15.4.12":{},"15.4.14":{},"15.4.16":{},"15.4.17":{},"15.4.2":{},"15.4.3":{},"15.4.4":{},"15.4.5":{},"15.4.6":{},"15.4.7":{},"15.4.9":{},"16.0.0":{},"16.0.1":{},"16.0.4":{},"16.1.0":{},"16.1.1":{},"16.1.3":{},"16.1.4":{},"16.1.6":{},"16.1.7":{},"16.1.8":{},"16.2.0":{},"8.3.4":{},"9.0.0":{},"9.1.2":{},"9.1.3":{},"9.2.1":{},"9.2.3":{},"9.2.4":{},"9.3.0":{},"9.3.2":{},"9.3.4":{},"9.3.6":{},"9.3.7":{},"9.3.9":{}}}

# curl -k https://localhost/v1/providers/terraform.releases.teleport.dev/gravitational/teleport/14.3.3.json
{"archives":{"darwin_amd64":{"url":"download/terraform-provider-teleport-v14.3.3-darwin-amd64-bin.zip","hashes":["zh:e2c9cb2f08207e663b217e0469e61b1df1591d09b0304b1073bc62a2256cf42a"]},"darwin_arm64":{"url":"download/terraform-provider-teleport-v14.3.3-darwin-arm64-bin.zip","hashes":["zh:77f4d3427bc050c94fc73be16ba87c7a7d943bbfce6a4c92ec0502f0f7c429eb"]},"linux_amd64":{"url":"download/terraform-provider-teleport-v14.3.3-linux-amd64-bin.zip","hashes":["zh:bb13cf6e0634106ab59f86dee92c8d692d2f55ccb0192fd572587f3f7ef41855"]},"linux_arm64":{"url":"download/terraform-provider-teleport-v14.3.3-linux-arm64-bin.zip","hashes":["zh:be548ccac7668c4e389b07bfe764ff25e36f28daea7cf57db9bd87d7d87660d8"]}}}

# curl -k --noproxy '*' https://localhost/v1/providers/terraform.releases.teleport.dev/gravitational/teleport/download/terraform-provider-teleport-v14.3.3-linux-amd64-bin.zip
{"message":"invalid archive","status":400,"statusText":"Bad Request"}

BTW, we are behind a proxy. Maybe something related with this only for download?

[thxCode]

@seboudry can you try to rename the provider to follow ^terraform-provider-(?P<type>\w+)_(?P<version>[\w|\\.]+)_(?P<os>[a-z]+)_(?P<arch>[a-z0-9]+)\.zip$?

FYI, https://github.com/seal-io/hermitcrab/blob/4d1aafd475fea874d9d224a1d4fbd1238ad3693d/pkg/apis/provider/handler_view.go#L72-L92

[seboudry]

Unfortunatly, I think teleport put -bin as a suffix on all their providers archives.

# grep -e download_url.*teleport-v14.3 /var/run/hermitcrab/data/metadata.db 
  "download_url": "https://terraform.releases.teleport.dev/store/terraform-provider-teleport-v14.3.3-linux-arm64-bin.zip",
  "download_url": "https://terraform.releases.teleport.dev/store/terraform-provider-teleport-v14.3.3-darwin-arm64-bin.zip",
  "download_url": "https://terraform.releases.teleport.dev/store/terraform-provider-teleport-v14.3.3-darwin-amd64-bin.zip",
  "download_url": "https://terraform.releases.teleport.dev/store/terraform-provider-teleport-v14.3.3-linux-amd64-bin.zip"

If I try request hermitcrab API without -bin in archive name, I got also a 400 error:

# curl -k --noproxy '*' https://localhost/v1/providers/terraform.releases.teleport.dev/gravitational/teleport/download/terraform-provider-teleport-v14.3.3-linux-amd64.zip
{"message":"invalid archive","status":400,"statusText":"Bad Request"}

[seboudry]

I don't think Teleport team will change something on their provider registry so easily.

So, is it possible to accept a suffix and hyphens in archive name?

By modifying the regex like this for example: ^terraform-provider-(?P<type>\w+)[_-](?P<version>[\w|\\.]+)[_-](?P<os>[a-z]+)[_-](?P<arch>[a-z0-9]+)([_-].*)?\.zip$.

[thxCode]

@seboudry https://hub.docker.com/r/sealio/hermitcrab/tags v0.1.5

[seboudry]

Thx for your help ;)

But KO unfortunately.

On terraform plan logs:

33  Error while installing terraform.releases.teleport.dev/gravitational/teleport
34  v14.3.3: unsuccessful request to
35  https://burrito-hermitcrab.burrito-system.svc.cluster.local/v1/providers/terraform.releases.teleport.dev/gravitational/teleport/download/terraform-provider-teleport-v14.3.3-linux-amd64-bin.zip:
36  500 Internal Server Error

And on hermitcrab logs:

D   api 200 | HTTP/2.0 |        0 B |    1.6 KiB |     626.158µs |    10.171.7.186 | GET     /v1/providers/terraform.releases.teleport.dev/gravitational/teleport/index.json
D   api 200 | HTTP/2.0 |        0 B |      698 B |     193.974µs |    10.171.7.186 | GET     /v1/providers/terraform.releases.teleport.dev/gravitational/teleport/14.3.3.json
E   api error requesting GET /v1/providers/terraform.releases.teleport.dev/gravitational/teleport/download/terraform-provider-teleport-v14.3.3-linux-amd64-bin.zip: 1 error occurred:
* version not found
github.com/seal-io/hermitcrab/pkg/apis/runtime.erroring
github.com/seal-io/hermitcrab/pkg/apis/runtime/middleware_error.go:41
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/hermitcrab/pkg/apis/runtime.recovering
github.com/seal-io/hermitcrab/pkg/apis/runtime/middleware_recovery.go:39
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/hermitcrab/pkg/apis/runtime.observing
github.com/seal-io/hermitcrab/pkg/apis/runtime/middleware_observation.go:89
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
github.com/gin-gonic/gin@v1.9.1/gin.go:620
github.com/gin-gonic/gin.(*Engine).ServeHTTP
github.com/gin-gonic/gin@v1.9.1/gin.go:576
github.com/seal-io/hermitcrab/pkg/apis/runtime.(*Router).ServeHTTP
github.com/seal-io/hermitcrab/pkg/apis/runtime/router.go:191
net/http.serverHandler.ServeHTTP
net/http/server.go:2943
net/http.initALPNRequest.ServeHTTP
net/http/server.go:3551
net/http.(*http2serverConn).runHandler
net/http/h2_bundle.go:6232
D   api 500 | HTTP/2.0 |        0 B |       91 B |     179.094µs |    10.171.7.186 | GET     /v1/providers/terraform.releases.teleport.dev/gravitational/teleport/download/terraform-provider-teleport-v14.3.3-linux-amd64-bin.zip

[seboudry]

I tried to debug a little bit this behavior.

Something weird, is that on file system the providers directory contains only terraform one and not teleport one:

ls /run/hermitcrab/data/providers/
registry.terraform.io

Does some config is needed to define a new registry other than terraform/opentofu?

[seboudry]

hi @thxCode! Any insight to go further?

[thxCode]

the versioned archives list of teleport

the versioned archives list of a legal provider

The root cause is the version mark of the archive download URL doesn't match the requested version:14.3.3 -> v14.3.3.

I can fix this here, but I am confused about why teleport doesn't follow the Terraform Release Rules.

[thxCode]

@seboudry https://hub.docker.com/r/sealio/hermitcrab/tags v0.1.6

[seboudry]

Thanks a lot @thxCode !

It works well with this new release.

I raised an issue on Teleport side: https://github.com/gravitational/teleport/issues/46527.