feat: add hetzner cloud provider

[Thunderbottom]

adds a provider for hetzner cloud that does:

• filter instances by label selector and datacenter location
• fetch public/private IP address based on provided address type
• checks for associated floating IPs in case of blocked public/private IP addresses
• accepts API token as environment variable: HCLOUD_TOKEN

Enhancement over: #159 Closes: #73

[hashicorp-cla]

All committers have signed the CLA.

[svenstaro]

Could we get this merged?

[laukaichung]

While waiting for it to be merged, is there any way for now to automatically get a list of private IPs for the retry_join field without hardcoding a list of IPs? I'm planning to run a consul cluster on Hetzner Cloud with private network. It seems that GetPrivateIPs from go-sockaddr, e,g retry_join={{ GetPrivateIPs }} would work but I''m not sure how to use it in the config.

[dnephin]

I don't think there is any support for that yet, but https://github.com/hashicorp/consul/issues/9100 is a proposal that should make it possible.

[vitamindit]

While waiting for it to be merged, is there any way for now to automatically get a list of private IPs for the retry_join field without hardcoding a list of IPs? ...

I'm waiting for it too :) ... here is my approach:

You get private IP addresses once the server is assigned to a network or subnet.

Then you should use e.g. the hcloud server-info API. This will return the available servers including their private IP addresses for available interfaces.

From this server-info response you can then construct the list of desired private cluster IP addresses.

[KevinGimbel]

What's blocking this PR from being merged? I'm currently evaluating Consul for work and I'd love to use/try the auto-discovery feature as part of this PoC.

[gardion]

Please merge, it would massively help adopting Consul for our platform.

[tomschlenkhoff]

Any help needed? Any blockers left? If not please (with sugar on top) merge, would be a boost for Consul/Hetzner adoption in Germany/EU.

[Thunderbottom]

I believe there's no blockers, although the repo hasn't been updated since last year. So I'm not sure if it'll be merged any time soon.

[ronaldburns]

This is a feature we need for our infrastructure. It doesn't appear like it's going to be committed anytime soon. We've forked this and made a fix or two.

go-discover consul

If you'd like to immediately use this implementation, binaries have been added for the latest releases.

Consul 1.11.3 (hcloud)

Consul 1.11.4 (hcloud)

Consul 1.12.0-beta1 (hcloud)

[devnet-io]

What are the blockers on getting this merged? This will be very useful for my org. Happy to assist with what is needed.

[KrakenMare]

Echoing the above. It would be fantastic to see this merged or insight into what is needed for a merge.

[pugnacity]

any news here?

[marco-m]

Hello @dnephin, @riddhi89, @alvin-huang, we were almost there with #159 :-), then after a while @Thunderbottom dared to summon the skeleton (just past-Halloween pun) and did this second attempt. I understand very well that you are fully loaded, but if you could spare a moment to review this PR many of us would be happy. Thanks again! @Thunderbottom would you have time to resolve the conflicts, to ease the work of the HashiCorp reviewers? Merci!

[acaloiaro]

FYI @Thunderbottom I can't imagine you're enthusiastic to do more work on this PR since it hasn't been re-reviewed yet, but in case you are -- a merge conflict has occurred.

[Thunderbottom]

Hi, haven't gotten the time to check GitHub that often anymore. I have fixed all the issues and rebased to the latest master. let me know if anything else needs to be done.

[svenstaro]

@riddhi89 Any chance we could get this merged?

[Neyury]

Any updates?

[Neyury]

Hello! @alvin-huang any chance you could review this PR? Or do you have somebody else from HashiCorp to suggest? I really want to see Hetzner Cloud in go-discovery so that Сonsul and Nomad could use auto-join. Thanks!

[acaloiaro]

@Neyury If you're desperate to use @Thunderbottom's patch, you can do what I did and fork nomad and add a replace to its go.mod as I've done here: https://github.com/hashicorp/nomad/commit/332c3e447d1d5fd55375b0bb28ce015a486d81e6

Then my server.server_join stanza looks like:

  server_join {
    retry_join = [ "provider=hcloud label_selector=nomad-server apiToken=${hetzner_compute_access_token}" ]
  }

Note: You'll want to label your nomad server compute nodes with nomad-server to use the above.

I've been running a hetzner nomad cluster this way for over 6 months. I do recommend pinning your fork to a tagged version of nomad, however.

I have a 1.4.x nomad build with patched go-discover here: https://github.com/acaloiaro/nomad/releases/tag/v1.4.x

You really should not use binaries from unknown strangers on the internet. But feel free to use it in a dev/test environment if you feel that it's sufficiently isolated from the rest of your infrastructure

[tagirb]

Hi Yuriy, thanks for taking care of this feature, also very interested in it. I'd suggest involving @LKaemmerling from Hetzner for this.

[acaloiaro]

Hashicorp are "semi-actively moving our repos to https://github.com/hashicorp/go-netaddr".

I emailed a member of their team to learn about the status of this repository. Those are the words of one individual, but you can see from the project activity that the project is not under heavy development.

Anyone waiting for this or other features to land in go-discover should probably not hold their breath about them being added to go-discover. It sounds like this project has become too brittle and difficult to maintain, motivating the migration to go-netaddr. An understandable move given this project's dependencies.

With that said, I don't see how go-netaddr replaces go-discover's cloud host-aware functionality. go-netaddr seems like a nice complement to, but not replacement for go-discover.

[kaspergrubbe]

Hello from 2024, I also hit this in my own setup, and would have loved for go-discover to support hcloud.