ionos-cloud / terraform-provider-ionoscloud

The IonosCloud Terraform provider gives the ability to deploy and configure resources using the IonosCloud APIs.
Mozilla Public License 2.0
34 stars 23 forks source link

IP failover group creation not behaving like when created in DCD #241

Closed marcelbrueckner closed 2 years ago

marcelbrueckner commented 2 years ago

Description

Not sure if a bug, but as discussions aren't enabled for this repo, I'm creating this issue. If what I want to achieve is generally not yet possible with the Terraform provider, consider this as a feature request.

I'm wrapping my head around IP failover configuration for my servers and can't find a way to get it working the way it does in DCD.

Expected behavior

Two servers that are part of the same public network should share a floating IP. For this, they should be member of the same IP failover group and the failover IP should appear on all servers as it appears when adding a failover group via UI.

Screenshots ![grafik](https://user-images.githubusercontent.com/3597890/158850195-d0175896-acb2-477a-9f6d-431f2f016b82.png) ![grafik](https://user-images.githubusercontent.com/3597890/158850274-69170624-8b3c-4c8a-89a3-7da8d83173fd.png) ![grafik](https://user-images.githubusercontent.com/3597890/158850324-206bfbff-c27b-4e00-bb4a-dcdab35fd883.png)

Environment

Terraform version:

Terraform v1.1.7

Tested with the following provider version:

registry.terraform.io/ionos-cloud/ionoscloud v6.1.5

OS:

macOS Big Sur 11.6.2
darwin_amd64

Configuration Files

I've only set IONOS_TOKEN.

How to Reproduce

Looking at the Data Center Designer UI, a failover group contains of a list of public NICs. The ionoscloud_ipfailover documentation however wants a string.

My first thought was to just add the failover twice via count = 2, using each servers nicuuid.

```hcl # Basic Terraform config omitted for readability resource "ionoscloud_ipblock" "floating_ip" { name = "Test DC public IP" location = ionoscloud_datacenter.my_datacenter.location size = 1 } resource "ionoscloud_server" "frontend" { # Actually, I wrapped this resource in a module doing common provisioning tasks etc. # But this should sufficiently illustrate what I'm doing count = 2 # ... server config according to docs here ... nic { lan = ionoscloud_lan.public_network.id dhcp = true ips = null firewall_active = false } } resource "ionoscloud_ipfailover" "frontend" { count = 2 datacenter_id = ionoscloud_datacenter.my_datacenter.id lan_id = ionoscloud_lan.public_network.id ip = ionoscloud_ipblock.floating_ip.ips[0] nicuuid = ionoscloud_server.frontend[count.index].primary_nic } ``` Planning goes well, but applying the configuration fails. ```bash $ terraform plan -out=tfplan Terraform will perform the following actions: # ionoscloud_ipfailover.frontend[0] will be created + resource "ionoscloud_ipfailover" "frontend" { + datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" + id = (known after apply) + ip = "85.215.221.214" + lan_id = "2" + nicuuid = "7f3073fc-b20b-4f3a-9bd3-8620be608df5" } # ionoscloud_ipfailover.frontend[1] will be created + resource "ionoscloud_ipfailover" "frontend" { + datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" + id = (known after apply) + ip = "85.215.221.214" + lan_id = "2" + nicuuid = "a6d40e0c-faee-4c82-9399-a87193d707da" } Plan: 2 to add, 0 to change, 0 to destroy. $ terraform apply tfplan Acquiring state lock. This may take a few moments... ionoscloud_ipfailover.frontend[0]: Creating... ionoscloud_ipfailover.frontend[1]: Creating... ╷ │ Error: an error occured while patching a lans failover group 2 422 Unprocessable Entity { │ "httpStatus" : 422, │ "messages" : [ { │ "errorCode" : "183", │ "message" : "[(root).properties.ipFailover] Failover IP '85.215.221.214' not found in Master NIC '7f3073fc-b20b-4f3a-9bd3-8620be608df5'. IP failover settings cannot be applied." │ } ] │ } │ │ │ with ionoscloud_ipfailover.frontend[0], │ on main.tf line 60, in resource "ionoscloud_ipfailover" "frontend": │ 60: resource "ionoscloud_ipfailover" "frontend" { │ ╵ ╷ │ Error: an error occured while patching a lans failover group 2 422 Unprocessable Entity { │ "httpStatus" : 422, │ "messages" : [ { │ "errorCode" : "183", │ "message" : "[(root).properties.ipFailover] Failover IP '85.215.221.214' not found in Master NIC 'a6d40e0c-faee-4c82-9399-a87193d707da'. IP failover settings cannot be applied." │ } ] │ } │ │ │ with ionoscloud_ipfailover.frontend[1], │ on main.tf line 60, in resource "ionoscloud_ipfailover" "frontend": │ 60: resource "ionoscloud_ipfailover" "frontend" { │ ```

Maybe I need to add the floating IP to the servers upfront. Seems reasonable. Nothing easier than that I thought (Hint: not easy).

```hcl # Basic Terraform config omitted for readability resource "ionoscloud_ipblock" "floating_ip" { name = "Test DC public IP" location = ionoscloud_datacenter.my_datacenter.location size = 1 } resource "ionoscloud_ipblock" "frontend" { name = "Test DC frontend server IPs" location = ionoscloud_datacenter.my_datacenter.location size = 2 } resource "ionoscloud_server" "frontend" { # Actually, I wrapped this resource in a module doing common provisioning tasks etc. # But this should sufficiently illustrate what I'm doing count = 2 # ... server config according to docs here ... nic { lan = ionoscloud_lan.public_network.id dhcp = true ips = [ionoscloud_ipblock.frontend.ips[count.index], ionoscloud_ipblock.cloud_public_ip.ips[0]] firewall_active = false } } resource "ionoscloud_ipfailover" "frontend" { count = 2 datacenter_id = ionoscloud_datacenter.my_datacenter.id lan_id = ionoscloud_lan.public_network.id ip = ionoscloud_ipblock.floating_ip.ips[0] nicuuid = ionoscloud_server.frontend[count.index].primary_nic } ``` Planning goes well as it often does, but applying the configuration fails. ```bash $ terraform plan -out=tfplan Terraform will perform the following actions: # ionoscloud_server.frontend[0] will be updated in-place ~ resource "ionoscloud_server" "frontend" { # (15 unchanged attributes hidden) ~ nic { ~ ips = [ - "85.215.206.89", + "85.215.222.107", + "85.215.221.214", ] # (7 unchanged attributes hidden) } # (1 unchanged block hidden) } # ionoscloud_server.frontend[1] will be updated in-place ~ resource "ionoscloud_server" "frontend" { # (15 unchanged attributes hidden) ~ nic { ~ ips = [ - "85.215.242.177", + "85.215.222.94", + "85.215.221.214", ] # (7 unchanged attributes hidden) } # (1 unchanged block hidden) } Plan: 0 to add, 2 to change, 0 to destroy. $ terraform apply tfplan │ Error: error getting state change for nics patch Request failed with following error: [VDC-5-1108] The public IP address 85.215.221.214 is already in use. ```

Next, I assigned the floating IP only to the first of my frontend servers. This lets Terraform complete successfully. But it doesn't look like I'm used to in the DCD WebUI.

```hcl # Basic Terraform config omitted for readability resource "ionoscloud_ipblock" "floating_ip" { name = "Test DC public IP" location = ionoscloud_datacenter.my_datacenter.location size = 1 } resource "ionoscloud_ipblock" "frontend" { name = "Test DC frontend server IPs" location = ionoscloud_datacenter.my_datacenter.location size = 2 } resource "ionoscloud_server" "frontend" { # Actually, I wrapped this resource in a module doing common provisioning tasks etc. # But this should sufficiently illustrate what I'm doing count = 2 # ... server config according to docs here ... nic { lan = ionoscloud_lan.public_network.id dhcp = true ips = concat([ionoscloud_ipblock.frontend.ips[count.index]], count.index == 0 ? [ionoscloud_ipblock.floating_ip.ips[0]] : []) firewall_active = false } } resource "ionoscloud_ipfailover" "frontend" { datacenter_id = ionoscloud_datacenter.my_datacenter.id lan_id = ionoscloud_lan.public_network.id ip = ionoscloud_ipblock.floating_ip.ips[0] nicuuid = ionoscloud_server.frontend[0].primary_nic } ``` ```bash $ terraform plan -out=tfplan Terraform will perform the following actions: # ionoscloud_ipfailover.frontend will be created + resource "ionoscloud_ipfailover" "frontend" { + datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" + id = (known after apply) + ip = "85.215.221.214" + lan_id = "1" + nicuuid = "7f3073fc-b20b-4f3a-9bd3-8620be608df5" } Plan: 1 to add, 0 to change, 0 to destroy. $ terraform apply tfplan ionoscloud_ipfailover.frontend: Creating... ionoscloud_ipfailover.frontend: Still creating... [10s elapsed] ionoscloud_ipfailover.frontend: Creation complete after 16s [id=1] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. ``` Result ![grafik](https://user-images.githubusercontent.com/3597890/158899415-4e8cf37d-0457-433c-8df2-ce79103dc1d5.png)

Lastly, I created the failover IP group manually in the web UI and imported it into Terraform to see how it should look like.

```hcl resource "ionoscloud_ipfailover" "frontend" { datacenter_id = ionoscloud_datacenter.my_datacenter.id lan_id = ionoscloud_lan.public_network.id ip = ionoscloud_ipblock.cloud_public_ip.ips[0] nicuuid = ionoscloud_server.frontend[0].primary_nic } ``` ```bash $ terraform import ionoscloud_ipfailover.frontend 73738a73-b7fa-426e-aa0f-e1b666d795d9/1 ionoscloud_ipfailover.frontend: Importing from ID "73738a73-b7fa-426e-aa0f-e1b666d795d9/1"... ionoscloud_ipfailover.frontend: Import prepared! Prepared ionoscloud_ipfailover for import ionoscloud_ipfailover.frontend: Refreshing state... [id=1] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $ terraform state show ionoscloud_ipfailover.frontend # ionoscloud_ipfailover.frontend: resource "ionoscloud_ipfailover" "frontend" { datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" id = "1" ip = "85.215.221.214" lan_id = "1" nicuuid = "7f3073fc-b20b-4f3a-9bd3-8620be608df5" timeouts {} } $ terraform plan -out=tfplan # omitted ionoscloud_ipfailover.frontend: Refreshing state... [id=1] Note: Objects have changed outside of Terraform Terraform detected the following changes made outside of Terraform since the last "terraform apply": # ionoscloud_datacenter.my_datacenter has changed ~ resource "ionoscloud_datacenter" "my_datacenter" { id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" name = "terraform-test" ~ version = 2 -> 5 # (5 unchanged attributes hidden) } # ionoscloud_ipblock.frontend has changed ~ resource "ionoscloud_ipblock" "frontend" { id = "4914d3ba-bb05-4943-9f8f-c9100d3fdac1" name = "Test DC frontend server IPs" # (3 unchanged attributes hidden) + ip_consumers { + datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" + datacenter_name = "terraform-test" + ip = "85.215.222.94" + mac = "02:01:38:de:ca:c1" + nic_id = "7f3073fc-b20b-4f3a-9bd3-8620be608df5" + server_id = "c5dfc231-d0a2-49a0-9eef-5f48c44b27fa" + server_name = "fe01" } + ip_consumers { + datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" + datacenter_name = "terraform-test" + ip = "85.215.222.107" + mac = "02:01:a3:f7:dd:da" + nic_id = "a6d40e0c-faee-4c82-9399-a87193d707da" + server_id = "9d854297-8a9d-4840-a5b1-931f169b4254" + server_name = "fe02" } } # ionoscloud_ipblock.floating_ip has changed ~ resource "ionoscloud_ipblock" "floating_ip" { id = "27833a5b-46db-42ec-8eb7-e0fd55bcd2c8" name = "Test DC public IP" # (3 unchanged attributes hidden) + ip_consumers { + datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" + datacenter_name = "terraform-test" + ip = "85.215.221.214" + mac = "02:01:38:de:ca:c1" + nic_id = "7f3073fc-b20b-4f3a-9bd3-8620be608df5" + server_id = "c5dfc231-d0a2-49a0-9eef-5f48c44b27fa" + server_name = "fe01" } + ip_consumers { + datacenter_id = "73738a73-b7fa-426e-aa0f-e1b666d795d9" + datacenter_name = "terraform-test" + ip = "85.215.221.214" + mac = "02:01:a3:f7:dd:da" + nic_id = "a6d40e0c-faee-4c82-9399-a87193d707da" + server_id = "9d854297-8a9d-4840-a5b1-931f169b4254" + server_name = "fe02" } } # ionoscloud_lan.public_network has changed ~ resource "ionoscloud_lan" "public_network" { id = "1" name = "public" # (2 unchanged attributes hidden) + ip_failover { + ip = "85.215.221.214" + nic_uuid = "7f3073fc-b20b-4f3a-9bd3-8620be608df5" } } # ionoscloud_server.frontend[1] has changed ~ resource "ionoscloud_server" "frontend" { id = "9d854297-8a9d-4840-a5b1-931f169b4254" name = "fe02" # (13 unchanged attributes hidden) ~ nic { ~ ips = [ "85.215.222.107", + "85.215.221.214", ] # (7 unchanged attributes hidden) } # (1 unchanged block hidden) } Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes. ────────────────────────────── Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # ionoscloud_server.server[1] will be updated in-place ~ resource "ionoscloud_server" "frontend" { id = "9d854297-8a9d-4840-a5b1-931f169b4254" name = "fe02" # (13 unchanged attributes hidden) ~ nic { ~ ips = [ "85.215.222.107", - "85.215.221.214", ] # (7 unchanged attributes hidden) } # (1 unchanged block hidden) } Plan: 0 to add, 1 to change, 0 to destroy. ```

When looking at the plan, you can see that both of my frontend servers have the floating IP assigned (something that's not possible using Terraform looking at my second attempt). Of course terraform plan also tries to remove that IP from the second server. When assigning the floating IP to both servers in my config, I can refresh my state without any changes.

```hcl resource "ionoscloud_server" "frontend" { # Actually, I wrapped this resource in a module doing common provisioning tasks etc. # But this should sufficiently illustrate what I'm doing count = 2 # ... server config according to docs here ... nic { lan = ionoscloud_lan.public_network.id dhcp = true ips = [ionoscloud_ipblock.frontend.ips[count.index], ionoscloud_ipblock.floating_ip.ips[0]] firewall_active = false } } resource "ionoscloud_ipfailover" "frontend" { datacenter_id = ionoscloud_datacenter.my_datacenter.id lan_id = ionoscloud_lan.public_network.id ip = ionoscloud_ipblock.floating_ip.ips[0] nicuuid = ionoscloud_server.frontend[0].primary_nic } ``` ```bash $ terraform plan -out=tfplan # Same output as previous plan, plus the below No changes. Your infrastructure matches the configuration. Your configuration already matches the changes detected above. If you would like to update the Terraform state to match, create and apply a refresh-only plan: terraform apply -refresh-only $ terraform apply -refresh-only This is a refresh-only plan, so Terraform will not take any actions to undo these. If you were expecting these changes then you can apply this plan to record the updated values in the Terraform state without changing any remote objects. Would you like to update the Terraform state to reflect these detected changes? Terraform will write these changes to the state without modifying any real infrastructure. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes Apply complete! Resources: 0 added, 0 changed, 0 destroyed. $ terraform state show 'ionoscloud_server.frontend[0]' | grep -A3 ips ips = [ "85.215.222.94", "85.215.221.214", ] $ terraform state show 'ionoscloud_server.frontend[1]' | grep -A3 ips ips = [ "85.215.222.107", "85.215.221.214", ] ```
iblindu commented 2 years ago

Hi @marcelbrueckner,

Thank you a lot for opening this topic since this should be documented properly. The strategy would be the following (here being the docs examples changed according to your needs):

1. First create NIC A and NIC B under LAN1 with different IPs. Create the ip_failover on LAN 1 with NIC A and failover IP of NIC A

``` hcl ... datacenter here ... resource "ionoscloud_ipblock" "example" { location = "us/las" size = 2 name = "IP Block Example" } resource "ionoscloud_lan" "example" { datacenter_id = ionoscloud_datacenter.example.id public = true name = "Lan Example" } resource "ionoscloud_server" "example" { count = 2 ... server config ... nic { lan = ionoscloud_lan.example.id dhcp = true firewall_active = true ips = [ ionoscloud_ipblock.example.ips[count.index] ] } } resource "ionoscloud_ipfailover" "example" { depends_on = [ ionoscloud_lan.example ] datacenter_id = ionoscloud_datacenter.example.id lan_id = ionoscloud_lan.example.id ip = ionoscloud_ipblock.example.ips[0] nicuuid = ionoscloud_server.example[0].primary_nic } ```

2. Update NIC B IP to be failover IP

``` hcl ... datacenter here ... resource "ionoscloud_ipblock" "example" { location = "us/las" size = 2 name = "IP Block Example" } resource "ionoscloud_lan" "example" { datacenter_id = ionoscloud_datacenter.example.id public = true name = "Lan Example" } resource "ionoscloud_server" "example" { count = 2 ... server config ... nic { lan = ionoscloud_lan.example.id dhcp = true firewall_active = true ips = [ ionoscloud_ipblock.example.ips[0] ] } } resource "ionoscloud_ipfailover" "example" { depends_on = [ ionoscloud_lan.example ] datacenter_id = ionoscloud_datacenter.example.id lan_id = ionoscloud_lan.example.id ip = ionoscloud_ipblock.example.ips[0] nicuuid = ionoscloud_server.example[0].primary_nic } ```

3. After this, you can create NIC C, NIC D and so on, with the ip being the failover ip, with no problems!

We will add this steps in our documentation. Thanks again for raising this issue and please let me know if it works for you!

P.S. We will also enable discussions on our repositories for this kind of topics.

marcelbrueckner commented 2 years ago

Thanks @IuliaBlindu

I have adapted your example to my use-case and it works as you described. I've omitted the depends_on attribute from the IP failover group as Terraform seems clever enough to determine the dependencies on it's own.

```hcl resource "ionoscloud_datacenter" "my_datacenter" { name = "terraform-test" location = "de/txl" description = "Test ionoscloud_provider" } resource "ionoscloud_lan" "public_network" { datacenter_id = ionoscloud_datacenter.my_datacenter.id public = true name = "public" } resource "ionoscloud_ipblock" "public_ip" { name = "Test DC public IP" location = ionoscloud_datacenter.my_datacenter.location size = 1 } resource "ionoscloud_ipblock" "frontend" { name = "Test DC frontend server IPs" location = ionoscloud_datacenter.my_datacenter.location size = 2 } resource "ionoscloud_server" "frontend" { count = 2 ... server config ... nic { lan = ionoscloud_lan.public_network.id dhcp = true firewall_active = true # Initial creation ips = concat([ionoscloud_ipblock.frontend.ips[count.index]], count.index == 0 ? [ionoscloud_ipblock.public_ip.ips[0]] : []) # After IP failover group has been created, change ips to this # Or use it right from the beginning, but start with count = 1 and increase it afterwards ips = [ionoscloud_ipblock.frontend.ips[count.index], ionoscloud_ipblock.public_ip.ips[0]] } } resource "ionoscloud_ipfailover" "frontend" { datacenter_id = ionoscloud_datacenter.my_datacenter.id lan_id = ionoscloud_lan.public_network.id ip = ionoscloud_ipblock.public_ip.ips[0] nicuuid = ionoscloud_server.frontend[0].primary_nic } ```

But while this works, it's not really the path I would like to go with my infrastructure code. Because the code can't be used to re-create all infrastructure without changes.

Imagine a disaster recovery scenario, where I want to rebuild everything from scratch. I would use the latest commit of my infrastructure code to create everything, but having the same IP for all servers will initially fail as in my 2nd attempt, described in my initial post.

Of course I could wrap a Makefile or similar around Terraform to do the IP replacement, but it's just an unnecessary overhead.

Passing a list instead of a single UUID to ionoscloud_ipfailover's nicuuid seem more logical to me. This might be done in conjunction of allowing the same IP being attached to different NICs from the very beginning (I now see that this is only possible if an IP failover group already exists - which isn't the case initially).

However, playing with it a little longer today I found a compromise I guess. Not very elegant code-wise (because WET), but this will allow me setting up everything even from scratch without changing code. I simply create the first server independently from others.

```hcl resource "ionoscloud_datacenter" "my_datacenter" { name = "terraform-test" location = "de/txl" description = "Test ionoscloud_provider" } resource "ionoscloud_lan" "public_network" { datacenter_id = ionoscloud_datacenter.my_datacenter.id public = true name = "public" } resource "ionoscloud_ipblock" "public_ip" { name = "Test DC public IP" location = ionoscloud_datacenter.my_datacenter.location size = 1 } resource "ionoscloud_ipblock" "frontend" { name = "Test DC frontend server IPs" location = ionoscloud_datacenter.my_datacenter.location size = 2 } resource "ionoscloud_server" "frontend_first" { # Notice: No count attribute here ... server config ... nic { lan = ionoscloud_lan.public_network.id dhcp = true firewall_active = true ips = [ionoscloud_ipblock.frontend.ips[0], ionoscloud_ipblock.public_ip.ips[0]] } } resource "ionoscloud_ipfailover" "frontend" { datacenter_id = ionoscloud_datacenter.my_datacenter.id lan_id = ionoscloud_lan.public_network.id ip = ionoscloud_ipblock.public_ip.ips[0] nicuuid = ionoscloud_server.frontend_first.primary_nic } # Now, as the failover group exists, I can add all other servers resource "ionoscloud_server" "frontend_others" { # Little benefit writing count = 1, but I only need two servers in total (for now) :) count = 1 # Add dependency, otherwise the server would be created before the IP failover group is in place depends_on = [ ionoscloud_ipfailover.frontend ] ... server config ... nic { lan = ionoscloud_lan.public_network.id dhcp = true firewall_active = true ips = [ionoscloud_ipblock.frontend.ips[0], ionoscloud_ipblock.public_ip.ips[0]] } } ```
iblindu commented 2 years ago

Hi @marcelbrueckner,

Yes, I totally understand. We will come with a more elegant solution, involving a list instead of a single UUID for the ionoscloud_ipfailover, as you mentioned. Since you have a workaround for the moment, how urgent is it for you?

marcelbrueckner commented 2 years ago

The solution I came up with thanks to your example is sufficient for my use-case. Everything else I've mentioned is purely cosmetic.

So, not urgent at all.

iblindu commented 2 years ago

Thank you again for opening this issue and for all the details provided! I will close the issue for now and will include an improvement on this side in our backlog .

jbuchhammer commented 2 years ago

Hi @marcelbrueckner , @IuliaBlindu that's an interesting discussion. I'll try it out next week if I find the time.

If I got it right, Iulia, you have to perform two times an apply, right?

I chose a different approach by not using count but creating the two servers individually. The order is then to create the first server with the failover-IP, create the ipfailover for that server and finally create the second server with the failover-IP. (That's the way how it has to be done on the API). However that needs an explicit depends_on = [ionoscloud_ipfailover.example] for the second server.