Hashicorp providers appear to have moved to a different repo breaking codegen

[starhawking]

It appears that several hashicorp maintained providers have moved from https://github.com/terraform-providers into https://github.com/hashicorp .

This breaks the codegen for adding new providers, and the makecode.py script does not currently provide useful information when it fails to pull these repos.

I added some temporary logging to demonstrate the situation:

aus-tmay1:python-terrascript taylor.may$ make code
rm -f tests/*.pyc
rm -f terrascript/*.pyc
rm -f terrascript/*/*.pyc
rm -f .coverage
rm -f .noseid*
( cd tools && ./makecode.py )
INFO:root:alicloud
INFO:root:archive
INFO:root:arukas
INFO:root:aws
INFO:root:bitbucket
INFO:root:clc
INFO:root:chef
INFO:root:circonus
INFO:root:cloudflare
INFO:root:cloudscale
INFO:root:cloudstack
INFO:root:cobbler
INFO:root:datadog
ERROR:root:Missing 'provider.go' for repo 'archive' in '/var/folders/wr/4_8vkc1d7_x52gzplzw6bvc9l0j93f/T/tmpuq_v396t' resultcode [0]
archive
b"Cloning into '.'...\n"
INFO:root:digitalocean
INFO:root:dns
INFO:root:dme
INFO:root:dnsimple
INFO:root:docker
INFO:root:dyn
INFO:root:external
INFO:root:fastly
ERROR:root:Missing 'provider.go' for repo 'dns' in '/var/folders/wr/4_8vkc1d7_x52gzplzw6bvc9l0j93f/T/tmpjcrsxafq' resultcode [0]
dns
b"Cloning into '.'...\n"
INFO:root:github
INFO:root:gitlab
INFO:root:google
INFO:root:google-beta
ERROR:root:Missing 'provider.go' for repo 'external' in '/var/folders/wr/4_8vkc1d7_x52gzplzw6bvc9l0j93f/T/tmpwtqbe9f6' resultcode [0]
external
b"Cloning into '.'...\n"
INFO:root:grafana

[starhawking]

It looks like github is doing a redirect for several of these repos, however the provider.go is at a different path for the ones that are falling through. For example, instead of being in the root of the repo, the tls provider has it at internal/provider/provider.go

[mjuenema]

Hi @starhawking, thanks for reporting this. The logic in makecode.py really needs a lot of work. Have a look at #issue145 for another problem with makecode.py. I hope to find some time this weekend to look into these problems.

[starhawking]

Hi @starhawking, thanks for reporting this. The logic in makecode.py really needs a lot of work. Have a look at

issue145 for another problem with makecode.py. I hope to find some time this weekend to look into these problems.

Thanks for the super quick response!

I was starting to put together a PR for this if you'd like it. I found a lot of places where things had moved in the repos, and the clone depth of 0 was no longer catching the provider.go file, and some had moved the provider.go to another name entirely. I don't quite have all the edge cases captured yet tho.

[mjuenema]

FYI: I started a separate discussion (#149) on the issue of making makecode.py more robust.

[ilons]

I think some of the issues with moved directories (and moved providers.go) should be solved by https://github.com/mjuenema/python-terrascript/pull/150 (which I made while @mjuenema was investigating better alternatives)

[mjuenema]

@starhawking Feel free to add your name to CONTRIBUTORS.md and create a Pull Request for it. You deserve it.

[ilons]

Also, if you pull develop now, you'll be able to run make providers and it would actually handle some (all) of those cases, but the increased logging would be good non-the-less.

[ilons]

This should now be working again after https://github.com/mjuenema/python-terrascript/commit/3d8731059fa763b7e6d2104ffe18e97fb0ed1e37#diff-50718022ac695dec6197c6721dec23891b36fb7396b2a304363d2774226c51f7

[ilons]

I'll close this issue for now as it should be working, and hopefully @mjuenema will be able to finish the work on the updated tooling soon.