search

F5Networks / f5-azure-arm-templates-v2

Azure Resource Manager Templates for quickly deploying BIG-IP services in Azure
22 stars 45 forks source link

Trying to re-fetch interface data. Left attempts: onboarder extension #17

Closed de1chk1nd closed 2 years ago

de1chk1nd commented 2 years ago

Describe the bug

The quickstart template fails on the child bigipTemplate at the onboarder extension.

Expected behavior

Successful deployment and onboarding of BIG-IP instance

Current behavior

The child bigipTemplate fails during onboarder extension and therefore the BIG-IP fails to deploy.

error - will be pasted in comments

Possible solution

?

Steps to reproduce

deploy failover/3 NIC deployment template as-is by launching blue button fill in parameters, launch

Screenshots

n/a

Context

n/a

Your Environment

n/a

de1chk1nd commented 2 years ago

error:

{
    "status": "Failed",
    "error": {
        "code": "VMExtensionProvisioningError",
        "message": "VM has reported a failure when processing extension 'onboarder'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\n[23272]: info: Local interface 0 MAC address 0022485bc04e matches Azure network interface 0 MAC address 0022485bc04e\n2022-07-19T08:20:35.382Z [23272]: info: Interface:1.2\n2022-07-19T08:20:35.383Z [23272]: info: MAC adddress is not populated on 1.2 BIGIP interface. Trying to re-fetch interface data. Left attempts: 99\n2022-07-19T08:20:35.392Z [23272]: info: Interface:1.1\n2022-07-19T08:20:35.392Z [23272]: info: MAC adddress is not populated on 1.1 BIGIP interface. Trying to re-fetch interface data. Left attempts: 99\n2022-07-19T08:20:45.383Z [23272]: info: Interface:1.1\n2022-07-19T08:20:45.385Z [23272]: info: MAC adddress is not populated on 1.1 BIGIP interface. Trying to re-fetch interface data. Left attempts: 98\n2022-07-19T08:20:45.398Z [23272]: info: Interface:1.2\n2022-07-19T08:20:45.399Z [23272]: info: MAC adddress is not populated on 1.2 BIGIP interface. Trying to re-fetch interface data. Left attempts: 98\n2022-07-19T08:20:45.409Z [23272]: info: Interface:1.1\n2022-07-19T08:20:45.410Z [23272]: info: MAC adddress is not populated on 1.1 BIGIP interface. Trying to re-fetch interface data. Left attempts: 98\n2022-07-19T08:20:55.417Z [23272]: info: Interface:1.1\n2022-07-19T08:20:55.418Z [23272]: info: MAC address found for 1.1: 00:22:48:5b:ce:03\n2022-07-19T08:20:55.419Z [23272]: info: Local interface 1 MAC address 0022485bce03 matches Azure network interface 1 MAC address 0022485bce03\n2022-07-19T08:20:55.428Z [23272]: info: Interface:1.2\n2022-07-19T08:20:55.430Z [23272]: info: MAC address found for 1.2: 00:22:48:5b:ca:2c\n2022-07-19T08:20:55.431Z [23272]: info: Local interface 2 MAC address 0022485bca2c matches Azure network interface 2 MAC address 0022485bca2c\n2022-07-19T08:20:55.443Z [23272]: info: Interface:1.1\n2022-07-19T08:20:55.445Z [23272]: info: MAC address found for 1.1: 00:22:48:5b:ce:03\n2022-07-19T08:20:55.447Z [23272]: info: Local interface 1 MAC address 0022485bce03 matches Azure network interface 1 MAC address 0022485bce03\n2022-07-19T08:20:55.573Z [23272]: error: <!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>\r\n<title>404 - File or directory not found.</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}\r\nfieldset{padding:0 15px 10px 15px;} \r\nh1{font-size:2.4em;margin:0;color:#FFF;}\r\nh2{font-size:1.7em;margin:0;color:#CC0000;} \r\nh3{font-size:1.2em;margin:10px 0 0 0;color:#000000;} \r\n#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:\"trebuchet MS\", Verdana, sans-serif;color:#FFF;\r\nbackground-color:#555555;}\r\n#content{margin:0 0 0 2%;position:relative;}\r\n.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}\r\n-->\r\n</style>\r\n</head>\r\n<body>\r\n<div id=\"header\"><h1>Server Error</h1></div>\r\n<div id=\"content\">\r\n <div class=\"content-container\"><fieldset>\r\n  <h2>404 - File or directory not found.</h2>\r\n  <h3>The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.</h3>\r\n </fieldset></div>\r\n</div>\r\n</body>\r\n</html>\r\n\n2022-07-19T08:20:55.576Z [23272]: info: Sending F5 Teem report for failure case.\n2022-07-19T08:20:56.189Z [23272]: warn: Problem with getting data from /mgmt/tm/sys/license endpoint. Leaving regKey with default value\n2022-07-19T08:20:56.190Z [23272]: info: {\"id\":\"593fb1fd-39ea-fe4c-d2260fd758c5\",\"product\":\"BIG-IP\",\"cpuCount\":8,\"diskSize\":56320,\"memoryInMb\":32176,\"version\":\"16.1.2.1\",\"nicCount\":3,\"platformId\":\"Z100\",\"hostname\":\"bigip1\",\"management\":\"10.1.1.6/24\",\"provisionedModules\":{\"ltm\":\"nominal\"},\"installedPackages\":{},\"environment\":{\"pythonVersion\":\"Python 2.7.5\",\"pythonVersionDetailed\":\"2.7.5 (default, Dec 22 2021, 16:19:59) \\n[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]\",\"nodeVersion\":\"v6.9.1\",\"libraries\":{\"ssh\":\"OpenSSH_7.4p1, OpenSSL 1.0.2u-fips  20 Dec 2019\"}}}\n2022-07-19T08:20:56.441Z [23272]: error: Device is not licensed yet\n\n[stderr]\n\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "
    }
}
de1chk1nd commented 2 years ago 
controls:
  logLevel: info
  logFilename: /var/log/cloud/bigIpRuntimeInit.log
extension_packages:
  install_operations:
    - extensionType: do
      extensionVersion: 1.28.0
      extensionHash: fef1f81c259d1ead19c256a67b8dad492f2d70b1019ff520e16172a93e8c17c9
    - extensionType: as3
      extensionVersion: 3.36.1
      extensionHash: 48876a92d3d8fe7da70310882dc9fd1499d209579d798394715e18c12138daf3
    - extensionType: cf
      extensionVersion: 1.11.0
      extensionHash: f3c7aca8a19d0dc01e529d38aa0d235b4dfad34beec2584f8402aa9a92c34699
extension_services:
  service_operations:
    - extensionType: do
      type: inline
      value:
        schemaVersion: 1.0.0
        class: Device
        async: true
        label: Standalone 3NIC BIG-IP declaration for Declarative Onboarding with
          BYOL license
        Common:
          class: Tenant
          My_DbVariables:
            class: DbVariables
            provision.extramb: 1000
            restjavad.useextramb: true
            dhclient.mgmt: disable
            config.allow.rfc3927: enable
            tm.tcpudptxchecksum: Software-only
          My_Provisioning:
            class: Provision
            ltm: nominal
          My_Ntp:
            class: NTP
            servers:
              - 0.pool.ntp.org
              - 1.pool.ntp.org
            timezone: UTC
          My_Dns:
            class: DNS
            nameServers:
              - 168.63.129.16
          My_System:
            autoPhonehome: true
            class: System
            hostname: 'failover0.local'
          My_License:
            class: License
            licenseType: regKey
            regKey: !!!!!!REMOVED!!!!!!
          admin:
            class: User
            userType: regular
            password: '{{{BIGIP_PASSWORD}}}'
            shell: bash
          default:
            class: ManagementRoute
            gw: '{{{MGMT_GW}}}'
            network: default
          dhclient_route1:
            class: ManagementRoute
            gw: '{{{MGMT_GW}}}'
            network: 168.63.129.16/32
          azureMetadata:
            class: ManagementRoute
            gw: '{{{MGMT_GW}}}'
            network: 169.254.169.254/32
          defaultRoute:
            class: Route
            gw: '{{{DEFAULT_GW}}}'
            network: default
            mtu: 1500
          external:
            class: VLAN
            tag: 4094
            mtu: 1500
            interfaces:
              - name: '1.1'
                tagged: false
          external-self:
            class: SelfIp
            address: '{{{SELF_IP_EXTERNAL}}}'
            vlan: external
            allowService: default
            trafficGroup: traffic-group-local-only
          internal:
            class: VLAN
            interfaces:
              - name: '1.2'
                tagged: false
            mtu: 1500
            tag: 4093
          internal-self:
            class: SelfIp
            address: '{{{SELF_IP_INTERNAL}}}'
            vlan: internal
            allowService: default
            trafficGroup: traffic-group-local-only
    - extensionType: cf
      type: inline
      value:
        schemaVersion: 1.0.0
        class: Cloud_Failover
        environment: azure
        controls:
          class: Controls
          logLevel: silly
        externalStorage:
          scopingTags:
            f5_cloud_failover_label: bigip_high_availability_solution
        failoverAddresses:
          enabled: true
          scopingTags:
            f5_cloud_failover_label: bigip_high_availability_solution
          requireScopingTags: false
    - extensionType: do
      type: inline
      value:
        schemaVersion: 1.0.0
        class: Device
        async: true
        label: Standalone 3NIC BIG-IP declaration for Declarative Onboarding with
          BYOL license
        Common:
          class: Tenant
          My_DbVariables:
            class: DbVariables
            provision.extramb: 1000
            restjavad.useextramb: true
            dhclient.mgmt: disable
            config.allow.rfc3927: enable
            tm.tcpudptxchecksum: Software-only
          My_Provisioning:
            class: Provision
            ltm: nominal
          My_Ntp:
            class: NTP
            servers:
              - 0.pool.ntp.org
              - 1.pool.ntp.org
            timezone: UTC
          My_Dns:
            class: DNS
            nameServers:
              - 168.63.129.16
          My_System:
            autoPhonehome: true
            class: System
            hostname: 'failover0.local'
          admin:
            class: User
            userType: regular
            password: '{{{BIGIP_PASSWORD}}}'
            shell: bash
          default:
            class: ManagementRoute
            gw: '{{{MGMT_GW}}}'
            network: default
          dhclient_route1:
            class: ManagementRoute
            gw: '{{{MGMT_GW}}}'
            network: 168.63.129.16/32
          azureMetadata:
            class: ManagementRoute
            gw: '{{{MGMT_GW}}}'
            network: 169.254.169.254/32
          defaultRoute:
            class: Route
            gw: '{{{DEFAULT_GW}}}'
            network: default
            mtu: 1500
          external:
            class: VLAN
            tag: 4094
            mtu: 1500
            interfaces:
              - name: '1.1'
                tagged: false
          external-self:
            class: SelfIp
            address: '{{{SELF_IP_EXTERNAL}}}'
            vlan: external
            allowService: default
            trafficGroup: traffic-group-local-only
          internal:
            class: VLAN
            interfaces:
              - name: '1.2'
                tagged: false
            mtu: 1500
            tag: 4093
          internal-self:
            class: SelfIp
            address: '{{{SELF_IP_INTERNAL}}}'
            vlan: internal
            allowService: default
            trafficGroup: traffic-group-local-only
          configSync:
            class: ConfigSync
            configsyncIp: /Common/external-self/address
          failoverAddress:
            class: FailoverUnicast
            address: /Common/external-self/address
          failoverGroup:
            class: DeviceGroup
            type: sync-failover
            members:
              - failover0.local
              - failover1.local
            owner: /Common/failoverGroup/members/0
            autoSync: true
            saveOnAutoSync: false
            networkFailover: true
            fullLoadOnSync: false
            asmSync: false
          trust:
            class: DeviceTrust
            localUsername: admin
            localPassword: '{{{BIGIP_PASSWORD}}}'
            remoteHost: /Common/failoverGroup/members/0
            remoteUsername: admin
            remotePassword: '{{{BIGIP_PASSWORD}}}'
post_onboard_enabled: []
pre_onboard_enabled:
  - name: provision_rest
    type: inline
    commands:
      - /usr/bin/setdb provision.extramb 1000
      - /usr/bin/setdb restjavad.useextramb true
runtime_parameters:
  - name: VAULT_URL
    type: url
    value: file:///config/cloud/vault_url
  - name: SECRET_ID
    type: url
    value: file:///config/cloud/secret_id
  - name: BIGIP_PASSWORD
    type: secret
    secretProvider:
      type: KeyVault
      environment: azure
      vaultUrl: '{{{VAULT_URL}}}'
      secretId: '{{{SECRET_ID}}}'
  - name: SELF_IP_EXTERNAL
    type: metadata
    metadataProvider:
      type: network
      environment: azure
      field: ipv4
      index: 1
  - name: SELF_IP_INTERNAL
    type: metadata
    metadataProvider:
      type: network
      environment: azure
      field: ipv4
      index: 2
  - name: DEFAULT_GW
    type: metadata
    metadataProvider:
      environment: azure
      type: network
      field: ipv4
      index: 1
      ipcalc: first
  - name: MGMT_GW
    type: metadata
    metadataProvider:
      environment: azure
      type: network
      field: ipv4
      index: 0
      ipcalc: first
mikeshimkus commented 2 years ago

@de1chk1nd On the BIG-IP, can you verify that the /config/cloud/vault_url and /config/cloud/secret_id files are present and contain the correct values (should be the url of your Key Vault and ID of the secret)?

mikeshimkus commented 2 years ago

@de1chk1nd I just realized that runtime init config file is for our failover template, but you said you are deploying the quickstart, which explains why you wouldn't have the vault and secret files (quickstart doesn't use them). I verified that the Deploy button links to the quickstart template and uses the correct runtime init config URL.

Can you send me the link to the page where you are clicking the button?

de1chk1nd commented 2 years ago

hi, it is based on the failover template. this is a c&p error (quickstart thing). https://github.com/F5Networks/f5-azure-arm-templates-v2/tree/main/examples/failover

to get the logs, I need to re-deploy. will let you know when its doner (today/tomorrow)

de1chk1nd commented 2 years ago

environment was still deployed - see info below:

[azureuser@localhost:NO LICENSE:Standalone] ~ # cat /config/cloud/vault_url https://de1chk1nd-ts-vault.vault.azure.net/F5-BigIPPW

[azureuser@localhost:NO LICENSE:Standalone] ~ # cat /config/cloud/secret_id https://de1chk1nd-ts-vault.vault.azure.net/F5-BigIPPW

hmmm - both contain url/secred. theoretically he should be able to craft correct uri - but do not know if he expects https://de1chk1nd-ts-vault.vault.azure.net/ only

mikeshimkus commented 2 years ago

@de1chk1nd The BIG-IP module expects the secretId to contain "/secrets/" in the path: https://github.com/F5Networks/f5-azure-arm-templates-v2/blob/39390740cd45cae04ed2c7514282930f4bcd3560/examples/modules/bigip-standalone/bigip.json#L227

So the secretId passed to bigip.json should be something like: https://de1chk1nd-ts-vault.vault.azure.net/secrets/F5-BigIPPW

de1chk1nd commented 2 years ago

oh - f*** u r right. missed that. will test later - but guess ...yes...my bad... :( thx for pointing that out.

shyawnkarim commented 2 years ago

Closing. Please reopen if you still need assistance.