shebang42 commented 7 months ago

Hello,

I have added /router/bgp/as/vrfs/vrf/neighbors/neighbor/use/neighbor-group . See https://github.com/CiscoDevNet/terraform-provider-iosxr/compare/main...shebang42:terraform-provider-iosxr:feature/router-bgp-wip#diff-8b07de0d9611ac267890e0f303cb77e3cc0bb818f062500377db9d5f74623fef

This introduces an issue where neighbors/local_as_dual_as and neighbors/local_as_replace_as appear to be mutually exclusive with neighbors/use_neighbor_group in the configuration. However, I retrieve all of them in the data source. When configuring a new resource with the data from the data source, this results in an "Invalid configuration item." error.

So: I want the "local_as_dual_as" and "local_as_replace_as" from the data source to return "null" and not some boolean value when "use_neighbor_group" has a value. Is this possible, or is this a restriction of the YANG model?

Details:

Workflow:

Retrieve config

1) Retrieve router config with the iosxr_router_bgp_vrf data source 2) Convert to YAML and write to file

Configure device

3) Read configuration file and decode YAML 4) configure device with iosxr_router_bgp_vrf resource.

This results in a "BGP' detected the 'warning' condition 'Invalid configuration item." error.

Cisco configuration

Note that this configuration contains a use neighbor-group.

RP/0/RP0/CPU0:xr9kv-0#sh run router bgp 65002 vrf Azure_small router bgp 65002 vrf Azure_small address-family ipv4 unicast <..> ! neighbor 10. use neighbor-group Azure-group bfd fast-detect address-family ipv4 unicast route-policy rAzure-out(50) out !

Retrieve data

Output of `terraform apply`

Note that this output contains use_neighbor_group, as well as local_as_dual_as and local_as_replace_as

> terraform apply
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - ciscodevnet/iosxr in /Users/paul/go/bin
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with
│ published releases.
╵
data.iosxr_router_bgp_vrf.from_device: Reading...
data.iosxr_router_bgp_vrf.from_device: Read complete after 0s [id=Cisco-IOS-XR-um-router-bgp-cfg:/router/bgp/as[as-number=65002]/vrfs/vrf[vrf-name=Azure_small]]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_file.bgp_vrf_yaml will be created
  + resource "local_file" "bgp_vrf_yaml" {
      + content              = <<-EOT
            "as_number": "65002"
            "bfd_minimum_interval": 2000
            "bfd_multiplier": 3
            "default_information_originate": false
            "default_metric": null
            "device": null
            "id": "Cisco-IOS-XR-um-router-bgp-cfg:/router/bgp/as[as-number=65002]/vrfs/vrf[vrf-name=Azure_small]"
            "neighbors":
            - "advertisement_interval_milliseconds": null
              "advertisement_interval_seconds": null
              "bfd_fast_detect": true
              "bfd_fast_detect_disable": false
              "bfd_fast_detect_strict_mode": false
              "bfd_minimum_interval": null
              "bfd_multiplier": null
              "description": null
              "ebgp_multihop_maximum_hop_count": null
              "ignore_connected_check": false
              "local_as": null
              "local_as_dual_as": false
              "local_as_no_prepend": false
              "local_as_replace_as": false
              "neighbor_address": "10.<xx>"
              "password": null
              "remote_as": null
              "shutdown": false
              "timers_holdtime": null
              "timers_keepalive_interval": null
              "ttl_security": false
              "update_source": null
              "use_neighbor_group": "Azure-group"
            "rd_auto": false
            "rd_four_byte_as_as_number": null
            "rd_four_byte_as_index": null
            "rd_ip_address_index": null
            "rd_ip_address_ipv4_address": null
            "rd_two_byte_as_as_number": null
            "rd_two_byte_as_index": null
            "timers_bgp_holdtime": "30"
            "timers_bgp_keepalive_interval": 10
            "vrf_name": "Azure_small"
        EOT
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "router-bgp-vrf.yaml"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + bgp_vrf_data = {
      + as_number                     = "65002"
      + bfd_minimum_interval          = 2000
      + bfd_multiplier                = 3
      + default_information_originate = false
      + default_metric                = null
      + device                        = null
      + id                            = "Cisco-IOS-XR-um-router-bgp-cfg:/router/bgp/as[as-number=65002]/vrfs/vrf[vrf-name=Azure_small]"
      + neighbors                     = [
          + {
              + advertisement_interval_milliseconds = null
              + advertisement_interval_seconds      = null
              + bfd_fast_detect                     = true
              + bfd_fast_detect_disable             = false
              + bfd_fast_detect_strict_mode         = false
              + bfd_minimum_interval                = null
              + bfd_multiplier                      = null
              + description                         = null
              + ebgp_multihop_maximum_hop_count     = null
              + ignore_connected_check              = false
              + local_as                            = null
              + local_as_dual_as                    = false
              + local_as_no_prepend                 = false
              + local_as_replace_as                 = false
              + neighbor_address                    = "10.<xx>"
              + password                            = null
              + remote_as                           = null
              + shutdown                            = false
              + timers_holdtime                     = null
              + timers_keepalive_interval           = null
              + ttl_security                        = false
              + update_source                       = null
              + use_neighbor_group                  = "Azure-group"
            },
        ]
      + rd_auto                       = false
      + rd_four_byte_as_as_number     = null
      + rd_four_byte_as_index         = null
      + rd_ip_address_index           = null
      + rd_ip_address_ipv4_address    = null
      + rd_two_byte_as_as_number      = null
      + rd_two_byte_as_index          = null
      + timers_bgp_holdtime           = "30"
      + timers_bgp_keepalive_interval = 10
      + vrf_name                      = "Azure_small"
    }

Configure device:

> terraform apply
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - ciscodevnet/iosxr in /Users/paul/go/bin
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with
│ published releases.
╵
data.local_file.router-bgp-vrf-config: Reading...
data.local_file.router-bgp-vrf-config: Read complete after 0s [id=ee394476ef478508d6d23814008f794a084a57e2]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # iosxr_router_bgp_vrf.to_device will be created
  + resource "iosxr_router_bgp_vrf" "to_device" {
      + as_number                     = "65002"
      + bfd_minimum_interval          = 2000
      + bfd_multiplier                = 3
      + default_information_originate = false
      + id                            = (known after apply)
      + neighbors                     = [
          + {
              + bfd_fast_detect             = false
              + bfd_fast_detect_disable     = false
              + bfd_fast_detect_strict_mode = false
              + ignore_connected_check      = false
              + local_as_dual_as            = false
              + local_as_no_prepend         = false
              + local_as_replace_as         = false
              + neighbor_address            = "10.<xx>"
              + shutdown                    = false
              + ttl_security                = false
              + use_neighbor_group          = "Azure-group"
            },
        ]
      + rd_auto                       = false
      + timers_bgp_holdtime           = "30"
      + timers_bgp_keepalive_interval = 10
      + vrf_name                      = "Azure_small"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

iosxr_router_bgp_vrf.to_device: Creating...
iosxr_router_bgp_vrf.to_device: Still creating... [10s elapsed]
╷
│ Error: Client Error
│ 
│   with iosxr_router_bgp_vrf.to_device,
│   on export_bgp_vrf.tf line 11, in resource "iosxr_router_bgp_vrf" "to_device":
│   11: resource "iosxr_router_bgp_vrf" "to_device" {
│ 
│ Set request failed, got error: rpc error: code = Internal desc = {
│  "cisco-grpc:errors": {
│   "error": [
│    {
│     "error-type": "application",
│     "error-tag": "operation-failed",
│     "error-severity": "error",
│     "error-path": "Cisco-IOS-XR-um-router-bgp-cfg:router/bgp/as[as-number = '65002']/vrfs/vrf[vrf-name = 'Azure_small']/neighbors/neighbor[neighbor-address = '10.<xx>']/local-as",
│     "error-message": "'BGP' detected the 'warning' condition 'Invalid configuration item.'"
│    }
│   ]
│  }
│ }
│

However, when local_as_dual_as and local_as_replace_as are removed from the yaml file, the configuration is oke:

<..>
"neighbors":
  - "advertisement_interval_milliseconds": null
    <..>
    "local_as": null
    # "local_as_dual_as": false
    "local_as_no_prepend": false
    # "local_as_replace_as": false

Terraform will perform the following actions:

  # iosxr_router_bgp_vrf.to_device will be created
  + resource "iosxr_router_bgp_vrf" "to_device" {
      + as_number                     = "65002"
      + bfd_minimum_interval          = 2000
      + bfd_multiplier                = 3
      + default_information_originate = false
      + id                            = (known after apply)
      + neighbors                     = [
          + {
              + bfd_fast_detect             = false
              + bfd_fast_detect_disable     = false
              + bfd_fast_detect_strict_mode = false
              + ignore_connected_check      = false
              + local_as_no_prepend         = false
              + neighbor_address            = "10.<xx>"
              + shutdown                    = false
              + ttl_security                = false
              + use_neighbor_group          = "Azure-group"
            },
        ]
      + rd_auto                       = false
      + timers_bgp_holdtime           = "30"
      + timers_bgp_keepalive_interval = 10
      + vrf_name                      = "Azure_small"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

iosxr_router_bgp_vrf.to_device: Creating...
iosxr_router_bgp_vrf.to_device: Creation complete after 2s [id=Cisco-IOS-XR-um-router-bgp-cfg:/router/bgp/as[as-number=65002]/vrfs/vrf[vrf-name=Azure_small]]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

So, to summarize: I want the "local_as_dual_as" and "local_as_replace_as" from the data source to return "null" and not some boolean value when "use_neighbor_group" has a value.

Do you have any suggestions how to fix this? Do I overlook something?

I would be able to fix this in the TF code with a local variable. Something like below, but this would be a workaround for a limitation in the provider, so this is not the preferred solution.

    local_as_dual_as                    = neighbor.use_neighbor_group != null ? null : neighbor.local_as_dual_as
    local_as_no_prepend                 = neighbor.local_as_no_prepend
    local_as_replace_as                 = neighbor.use_neighbor_group != null ? null : neighbor.local_as_replace_as

danischm commented 7 months ago

I would say having both attributes set to false is correct, I am wondering why it is trying to configure both when both are set to false. Could you please share the following information:

Which version of the provider are you using?
Share the output of show run bgp ... | json unified
Enable debug logging by configuring an env var TF_LOG=Trace, which should include the gNMI call issued to push the config and share this as well

shebang42 commented 7 months ago

Thanks for your support Daniel,

Which version of the provider are you using?

Version 0.5.0 with the following patch https://github.com/CiscoDevNet/terraform-provider-iosxr/commit/8cb79b65910ea61140bca204481ba88f3edf87ad (Add to router_bgp_vrf.yaml):

      - yang_name: use/neighbor-group
        example: GROUP1

Please note that I later noticed that some local_as stuff is missing from gen/definitions/router_bgp_neighbor_group.yaml. I have added them later in this patch: https://github.com/CiscoDevNet/terraform-provider-iosxr/commit/8aca4261d241f8248806eed774523391f402ac28

This patch is NOT included in the troubleshooting logs below. And although I think it's unrelated, it's worth mentioning, since it also configures local_as/*.

Using Cisco IOS-XR Version 7.11.1 on CML.

Share the output of show run bgp ... | json unified

RP/0/RP0/CPU0:xr9kv-0#sh run router bgp 65002 vrf Azure_small | json unified
Tue Apr  9 10:18:30.094 UTC
{
 "data": {
  "Cisco-IOS-XR-um-router-bgp-cfg:router": {
   "bgp": {
    "as": [
     {
      "as-number": 65002,
      "vrfs": {
       "vrf": [
        {
         "vrf-name": "Azure_small",
         "bfd": {
          "minimum-interval": 2000,
          "multiplier": 3
         },
         "timers": {
          "bgp": {
           "keepalive-interval": 10,
           "holdtime": 30
          }
         },
         "bgp": {
          "router-id": "10.240.255.84"
         },
         "address-families": {
          "address-family": [
           {
            "af-name": "ipv4-unicast",
            "aggregate-addresses": {
             "aggregate-address": [
              {
               "address": "10.52.0.0",
               "masklength": 14,
               "summary-only": {

               }
              },
              {
               "address": "10.68.0.0",
               "masklength": 14,
               "summary-only": {

               }
              },
              {
               "address": "10.224.0.0",
               "masklength": 14,
               "summary-only": {

               }
              },
              {
               "address": "10.232.0.0",
               "masklength": 15,
               "summary-only": {

               }
              },
              {
               "address": "10.235.0.0",
               "masklength": 16,
               "as-set": {

               },
               "summary-only": {

               }
              },
              {
               "address": "10.236.0.0",
               "masklength": 16,
               "as-set": {

               },
               "summary-only": {

               }
              }
             ]
            }
           }
          ]
         },
         "neighbors": {
          "neighbor": [
           {
            "neighbor-address": "10.240.38.137",
            "use": {
             "neighbor-group": "Azure-to-Fastpath-BGP-group"
            },
            "bfd": {
             "fast-detect": {

             }
            },
            "address-families": {
             "address-family": [
              {
               "af-name": "ipv4-unicast",
               "route-policy": {
                "out": "rAzure-FastPath-out(50)"
               }
              }
             ]
            }
           }
          ]
         }
        }
       ]
      }
     }
    ]
   }
  }
 }
}

Enable debug logging by configuring an env var TF_LOG=Trace, which should include the gNMI call issued to push the config and share this as well

https://gist.github.com/shebang42/85a3fb03d1836fc487690edf8e719676

shebang42 commented 7 months ago

Update: I assumed that I introduced this behaviour with the yang_name: use/neighbor-group, but I was able to reproduce this with the vanilla 0.5.0 provider.

Also noticed similar behaviour, where the TF data-source seems to 'hallucinate' configuration. Will create an issue for this shortly.

CiscoDevNet / terraform-provider-iosxr

Question: Help for configuring mutually exclusive elements in a data source #242

Workflow:

Retrieve config

Configure device

Cisco configuration

Retrieve data

Output of `terraform apply`

Configure device:

CiscoDevNet / terraform-provider-iosxr

Question: Help for configuring mutually exclusive elements in a data source #242

Workflow:

Retrieve config

Configure device

Cisco configuration

Retrieve data

Output of terraform apply

Configure device:

Output of `terraform apply`