runtime-init not discovering MAC addresses, DO failing.

Summary

I am trying to use runtime init with DO to onboard my device. I can set the hostname and device password with DO, but when I add selfIP and vlan configurations to my DO declaration, runtime init starts to fail. (In my example below I am passing hostname to the DO declaration from Terraform, but using the variable from runtime init also works).

I am following runtime init example_6 from the documentation, where AWS is the cloud provider. I'm using a DO declaration based on the DO example #6 and I've shared my DO declaration below.

If I remove the selfIP and vlans from DO, then runtime init and DO work great. Once I add selfIP and vlans to the DO declaration, runtime init fails. The DO RPM is not downloaded or installed, and obviously the device is not configured with my desired password and selfIPs. I am using silly logging but my bigIpRuntimeInit.log does not appear to show what's failing. It does send a TEEM report indicating there was a failure.

I need help with finding why a DO declaration using selfIP addresses from runtime-init is failing in AWS. If I send the DO declaration via Postman with my desired SelfIP's with /32 notation

Environment

BIGIP version: 15.1.1 DO: 1.15 runtime init version: 1.0.0

cloud_config.yaml

---
runtime_parameters:
  - name: HOST_NAME
    type: metadata
    metadataProvider:
        environment: aws
        type: compute
        field: hostname
  - name: ADMIN_PASS
    type: secret
    secretProvider:
      environment: aws
      type: SecretsManager
      version: AWSCURRENT
      secretId: ${secret_id}
  - name: SELF_IP_INTERNAL
    type: metadata
    metadataProvider:
      environment: aws
      type: network
      field: local-ipv4s
      index: 2
  - name: SELF_IP_EXTERNAL
    type: metadata
    metadataProvider:
      environment: aws
      type: network
      field: local-ipv4s
      index: 1
pre_onboard_enabled:
  - name: provision_rest
    type: inline
    commands:
      - /usr/bin/setdb provision.extramb 500
      - /usr/bin/setdb restjavad.useextramb true
  - name: provision_modules
    type: inline
    commands:
      - echo 'sys provision asm { level nominal }' >> bigip_base.conf
extension_packages:
  install_operations:
    - extensionType: do
      extensionVersion: 1.15.0
    - extensionType: as3
      extensionVersion: 3.20.0
    - extensionType: ts
      extensionVersion: 1.14.0
    - extensionType: cf
      extensionVersion: 1.5.0
    - extensionType: ilx
      extensionUrl: https://github.com/F5Networks/f5-appsvcs-templates/releases/download/v1.3.0/f5-appsvcs-templates-1.3.0-1.noarch.rpm
      extensionVersion: 1.3.0
      extensionVerificationEndpoint: /mgmt/shared/fast/info
extension_services:
  service_operations:
    - extensionType: do
      type: url
      value: file:///var/tmp/do_config.json

do_config.json

{
    "schemaVersion": "1.15.0",
    "class": "Device",
    "async": true,
    "label": "my BIG-IP declaration for declarative onboarding",
    "Common": {
        "class": "Tenant",
        "hostname": "${hostname}",
        "myDns": {
            "class": "DNS",
            "nameServers": [
                "8.8.8.8"
            ]
        },
        "myNtp": {
            "class": "NTP",
            "servers": [
                "0.pool.ntp.org"
            ],
            "timezone": "UTC"
        },
        "admin": {
            "class": "User",
            "userType": "regular",
            "password": "{{{ ADMIN_PASS }}}",
            "shell": "bash"
        },
        "internal": {
            "class": "VLAN",
            "tag": 4093,
            "mtu": 1500,
            "interfaces": [
                {
                    "name": "1.2",
                    "tagged": false
                }
            ]
        },
        "internal-self": {
            "class": "SelfIp",
            "address": "{{{ SELF_IP_INTERNAL }}}",
            "vlan": "internal",
            "allowService": "default",
            "trafficGroup": "traffic-group-local-only"
        },
        "external": {
            "class": "VLAN",
            "tag": 4094,
            "mtu": 1500,
            "interfaces": [
                {
                    "name": "1.1",
                    "tagged": false
                }
            ]
        },
        "external-self": {
            "class": "SelfIp",
            "address": "{{{ SELF_IP_EXTERNAL }}}",
            "vlan": "external",
            "allowService": "none",
            "trafficGroup": "traffic-group-local-only"
        }
    }
}

Update and request

I have a workaround now. I have a sleep 60 command in my user_data before running f5-bigip-runtime-init --config-file /config/cloud/cloud_config.yaml

If I don't do this, runtime-init seems to run too early, and the MAC addresses are not found for interfaces 1.1 and 1.2, and then runtime init does not run the DO declaration, or even download and install the RPM files specified. Debug-level log excerpt when I run user_data without sleeping before runtime init:

[admin@ip-192-168-1-246:Active:Standalone] ~ # more /var/log/cloud/bigIpRuntimeInit.log
2020-11-17T02:09:14.628Z [19745]: info: Configuration file: /config/cloud/cloud_config.yaml
2020-11-17T02:09:14.638Z [19745]: info: Validating provided declaration
2020-11-17T02:09:14.719Z [19745]: info: Successfully validated declaration
2020-11-17T02:09:14.781Z [19745]: info: Resolving parameters
2020-11-17T02:09:15.100Z [19745]: debug: Interface Response:[object Object]
2020-11-17T02:09:15.100Z [19745]: info: Interface:1.1
2020-11-17T02:09:15.101Z [19745]: debug: filtered:[object Object]
2020-11-17T02:09:15.101Z [19745]: info: MAC address found for 1.1: none
2020-11-17T02:09:15.103Z [19745]: debug: Interface Response:[object Object]
2020-11-17T02:09:15.103Z [19745]: info: Interface:1.2
2020-11-17T02:09:15.103Z [19745]: debug: filtered:[object Object]
2020-11-17T02:09:15.103Z [19745]: info: MAC address found for 1.2: none
2020-11-17T02:09:15.107Z [19745]: info:
2020-11-17T02:09:15.108Z [19745]: info: Sending F5 Teem report for failure case.

Would it be possible to have logic built into runtime init so that it waits for these MAC addresses to be discoverable? I'm not sure if it's the AWS metadata service that isn't ready, or BIG-IP that is not ready (from my basic testing, the AWS metadata service is able to list the 3 MAC addresses of the VM before runtime init fails with the logs above).

Can I ask this as a feature request?

F5Networks / f5-bigip-runtime-init