F5Networks / f5-bigip-runtime-init

Apache License 2.0
14 stars 15 forks source link

Error There is an active asynchronous task executing #24

Closed JeffGiroux closed 2 years ago

JeffGiroux commented 3 years ago

I have a runtime init yaml file that includes DO, AS3, TS, CFE. The onboarding appears to have applied the declarations successfully. The as3 VIP is there, the DO made the self IPs, etc. The HA pair formed. However, I receive async task executing errors in the startup-log.

Should I make the DO ask async=false instead of true?

2021-08-15T20:08:50.764Z [9845]: info: Configuration file: /config/cloud/runtime-init-conf.yaml
2021-08-15T20:08:50.776Z [9845]: info: Validating provided declaration
2021-08-15T20:08:50.831Z [9845]: info: Successfully validated declaration
2021-08-15T20:08:50.832Z [9845]: info: Executing custom pre_onboard_enabled commands
2021-08-15T20:08:50.833Z [9845]: info: Executing inline shell command: /usr/bin/setdb provision.extramb 500
2021-08-15T20:08:54.610Z [9845]: info: Shell command: /usr/bin/setdb provision.extramb 500 execution completed; response: 
2021-08-15T20:08:54.620Z [9845]: info: Executing inline shell command: /usr/bin/setdb restjavad.useextramb true
2021-08-15T20:08:54.813Z [9845]: info: Shell command: /usr/bin/setdb restjavad.useextramb true execution completed; response: 
2021-08-15T20:08:54.813Z [9845]: info: Executing inline shell command: /usr/bin/setdb setup.run false
2021-08-15T20:08:55.000Z [9845]: info: Shell command: /usr/bin/setdb setup.run false execution completed; response: 
2021-08-15T20:09:45.284Z [9845]: info: Resolving parameters
2021-08-15T20:09:45.648Z [9845]: info: Executing install operations.
2021-08-15T20:09:45.658Z [9845]: info: Installing - do v1.23.0
2021-08-15T20:09:48.249Z [9845]: info: Validating - do extension is available.
2021-08-15T20:09:58.297Z [9845]: info: Installing - as3 v3.30.0
2021-08-15T20:10:04.540Z [9845]: info: Validating - as3 extension is available.
2021-08-15T20:10:26.623Z [9845]: info: Installing - ts v1.22.0
2021-08-15T20:10:32.309Z [9845]: info: Validating - ts extension is available.
2021-08-15T20:10:42.325Z [9845]: info: Installing - cf v1.9.0
2021-08-15T20:10:53.746Z [9845]: info: Validating - cf extension is available.
2021-08-15T20:11:24.818Z [9845]: info: Executing service operations.
2021-08-15T20:11:24.834Z [9845]: info: Creating - do v1.23.0 {"schemaVersion":"1.0.0","class":"Device","async":true,"label":"Onboard BIG-IP into an HA Pair","Common":{"class":"Tenant","dbVars":{"class":"DbVariables","restjavad.useextramb":true,"provision.extramb":500,"config.allow.rfc3927":"enable","ui.advisory.enabled":true,"ui.advisory.color":"blue","ui.advisory.text":"giroux15-f5vm01.example.com"},"mySystem":{"autoPhonehome":true,"class":"System","hostname":"giroux15-f5vm01.example.com"},"azureuser":{"class":"User","partitionAccess":{"all-partitions":{"role":"admin"}},"password":"********","shell":"bash","userType":"regular","keys":["ssh-rsa AAA"]},"myDns":{"class":"DNS","nameServers":["8.8.8.8","2001:4860:4860::8844"],"search":["f5.com"]},"myNtp":{"class":"NTP","servers":["0.us.pool.ntp.org","1.pool.ntp.org","2.pool.ntp.org"],"timezone":"UTC"},"myProvisioning":{"class":"Provision","ltm":"nominal"},"external":{"class":"VLAN","tag":4094,"mtu":1500,"interfaces":[{"name":1.1,"tagged":false}]},"external-localself":{"class":"SelfIp","address":"10.90.2.4/24","vlan":"external","allowService":"default","trafficGroup":"traffic-group-local-only"},"internal":{"class":"VLAN","tag":4093,"mtu":1500,"interfaces":[{"name":1.2,"tagged":false}]},"internal-localself":{"class":"SelfIp","address":"10.90.3.4/24","vlan":"internal","allowService":"default","trafficGroup":"traffic-group-local-only"},"default":{"class":"Route","gw":"10.90.2.1","network":"default","mtu":1500},"configsync":{"class":"ConfigSync","configsyncIp":"/Common/internal-localself/address"},"failoverAddress":{"class":"FailoverUnicast","address":"/Common/internal-localself/address"},"failoverGroup":{"class":"DeviceGroup","type":"sync-failover","members":["giroux15-f5vm01.example.com","giroux15-f5vm02.example.com"],"owner":"/Common/failoverGroup/members/0","autoSync":true,"saveOnAutoSync":false,"networkFailover":true,"fullLoadOnSync":false,"asmSync":false},"trust":{"class":"DeviceTrust","localUsername":"azureuser","localPassword":"********","remoteHost":"10.90.3.5","remoteUsername":"azureuser","remotePassword":"********"}}}
2021-08-15T20:12:15.793Z [9845]: info: Creating - as3 v3.30.0 {"class":"AS3","action":"deploy","persist":true,"declaration":{"class":"ADC","schemaVersion":"3.0.0","label":"Sample 1","remark":"An HTTP sample application","Sample_01":{"class":"Tenant","A1":{"class":"Application","template":"http","serviceMain":{"class":"Service_HTTP","virtualAddresses":["10.90.2.11"],"pool":"web_pool"},"web_pool":{"class":"Pool","monitors":["tcp"],"members":[{"servicePort":80,"addressDiscovery":"fqdn","autoPopulate":true,"hostname":"httpbin.org"}]}}}}}
2021-08-15T20:12:32.692Z [9845]: info: Creating - cf v1.9.0 {"class":"Cloud_Failover","environment":"azure","externalStorage":{"scopingTags":{"f5_cloud_failover_label":"mydeployment"}},"failoverAddresses":{"enabled":true,"scopingTags":{"f5_cloud_failover_label":"mydeployment"}},"failoverRoutes":{"enabled":true,"scopingTags":{"f5_cloud_failover_label":"mydeployment"},"scopingAddressRanges":[{"range":"0.0.0.0/0"}],"defaultNextHopAddresses":{"discoveryType":"static","items":["10.90.2.4","10.90.2.5"]}}}
2021-08-15T20:12:32.715Z [9845]: warn: Task creation failed; response code: 500
2021-08-15T20:12:32.716Z [9845]: error: {"message":"tryUntil: max tries reached: There is an active asynchronous task executing. -> Also see cloud docs link for more help: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/troubleshooting.html"}
2021-08-15T20:12:32.716Z [9845]: info: Sending F5 Teem report for failure case.
2021-08-15T20:12:44.884Z [9845]: info: {"id":"7c0e334c-7c2f-584b-1d54b0755e5a","product":"BIG-IP","cpuCount":8,"diskSize":137216,"memoryInMb":28144,"version":"15.1.2.1","nicCount":3,"regKey":"AFZRN-ZTWUD-FMJJJ-STDBL-MKDQKMW","platformId":"Z100","hostname":"giroux15-f5vm01.example.com","management":"10.90.1.4/24","provisionedModules":{"ltm":"nominal"},"installedPackages":{"f5-service-discovery-1.9.2-1.noarch":"1.9.2","f5-declarative-onboarding-1.23.0-4.noarch":"1.23.0","f5-cloud-failover-1.9.0-0.noarch":"1.9.0","f5-appsvcs-3.30.0-5.noarch":"3.30.0","f5-telemetry-1.22.0-1.noarch":"1.22.0"},"environment":{"pythonVersion":"Python 2.7.5","pythonVersionDetailed":"2.7.5 (default, Jan 15 2021, 12:52:53) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]","nodeVersion":"v6.9.1","libraries":{"ssh":"OpenSSH_7.4p1, OpenSSL 1.0.2s-fips  28 May 2019"}}}
2021-08-15T20:12:48.696Z [9845]: info: F5 Teem report was successfully sent for failure case.
2021-08-15T20:12:48.703Z [9845]: info: {"message":"tryUntil: max tries reached: There is an active asynchronous task executing. -> Also see cloud docs link for more help: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/troubleshooting.html"}
+ rm -f /tmp/5501.tmp
JeffGiroux commented 3 years ago

Using my repo here... https://github.com/JeffGiroux/f5_terraform/tree/main/Azure/HA_via_api

This onboarding file which contains the runtime init yaml https://github.com/JeffGiroux/f5_terraform/blob/main/Azure/HA_via_api/f5_onboard.tmpl

yaml snippet...

runtime_parameters:
  - name: HOST_NAME
    type: metadata
    metadataProvider:
      environment: azure
      type: compute
      field: name
  - name: REGION
    type: url
    value: http://169.254.169.254/metadata/instance/compute/location?api-version=2021-05-01&format=text
    headers:
      - name: Metadata
        value: true
  - name: USER_NAME
    type: static
    value: ${f5_username}
  - name: ADMIN_PASS
    type: static
    value: ${f5_password}
  - name: SSH_KEYS
    type: static
    value: ${ssh_keypair}
  - name: LAW_ID
    type: static
    value: ${law_id}
  - name: LAW_PRIMKEY
    type: static
    value: ${law_primkey}
  - name: FAILOVER_LABEL
    type: static
    value: ${f5_cloud_failover_label}
  - name: MANAGED_ROUTE
    type: static
    value: ${managed_route}
  - name: LOCAL_SELFIP_EXT
    type: static
    value: ${local_selfip_ext}
  - name: REMOTE_SELFIP_EXT
    type: static
    value: ${remote_selfip_ext}
  - name: LOCAL_SELFIP_INT
    type: static
    value: ${local_selfip_int}
  - name: REMOTE_SELFIP_INT
    type: static
    value: ${remote_selfip_int}
  - name: DNS_SERVER
    type: static
    value: ${dns_server}
  - name: NTP_SERVER
    type: static
    value: ${ntp_server}
  - name: TIMEZONE
    type: static
    value: ${timezone}
  - name: GATEWAY
    type: static
    value: ${gateway}
  - name: HOST1
    type: static
    value: ${host1}
  - name: HOST2
    type: static
    value: ${host2}
  - name: PUBLIC_VIP
    type: static
    value: ${public_vip}
pre_onboard_enabled:
  - name: provision_rest
    type: inline
    commands:
      - /usr/bin/setdb provision.extramb 500
      - /usr/bin/setdb restjavad.useextramb true
      - /usr/bin/setdb setup.run false
extension_packages:
  install_operations:
    - extensionType: do
      extensionVersion: ${DO_VER}
      extensionUrl: ${DO_URL}
    - extensionType: as3
      extensionVersion: ${AS3_VER}
      extensionUrl: ${AS3_URL}
    - extensionType: ts
      extensionVersion: ${TS_VER}
      extensionUrl: ${TS_URL}
    - extensionType: cf
      extensionVersion: ${CFE_VER}
      extensionUrl: ${CFE_URL}
extension_services:
  service_operations:
    - extensionType: do
      type: inline
      value:
        schemaVersion: 1.0.0
        class: Device
        async: true
        label: Onboard BIG-IP into an HA Pair
        Common:
          class: Tenant
          dbVars:
            class: DbVariables
            restjavad.useextramb: true
            provision.extramb: 500
            config.allow.rfc3927: enable
            ui.advisory.enabled: true
            ui.advisory.color: blue
            ui.advisory.text: '{{{ HOST_NAME }}}.example.com'
          mySystem:
            autoPhonehome: true
            class: System
            hostname: '{{{ HOST_NAME }}}.example.com'
          '{{{ USER_NAME }}}':
            class: User
            partitionAccess:
              all-partitions:
                role: admin
            password: '{{{ ADMIN_PASS }}}'
            shell: bash
            userType: regular
            keys:
              - '{{{ SSH_KEYS }}}'
          myDns:
            class: DNS
            nameServers:
              - '{{{ DNS_SERVER }}}'
              - 2001:4860:4860::8844
            search:
              - f5.com
          myNtp:
            class: NTP
            servers:
              - '{{{ NTP_SERVER }}}'
              - 1.pool.ntp.org
              - 2.pool.ntp.org
            timezone: '{{{ TIMEZONE }}}'
          myProvisioning:
            class: Provision
            ltm: nominal
          external:
            class: VLAN
            tag: 4094
            mtu: 1500
            interfaces:
              - name: 1.1
                tagged: false
          external-localself:
            class: SelfIp
            address: '{{{ LOCAL_SELFIP_EXT }}}/24'
            vlan: external
            allowService: default
            trafficGroup: traffic-group-local-only
          internal:
            class: VLAN
            tag: 4093
            mtu: 1500
            interfaces:
              - name: 1.2
                tagged: false
          internal-localself:
            class: SelfIp
            address: '{{{ LOCAL_SELFIP_INT }}}/24'
            vlan: internal
            allowService: default
            trafficGroup: traffic-group-local-only
          default:
            class: Route
            gw: '{{{ GATEWAY }}}'
            network: default
            mtu: 1500
          configsync:
            class: ConfigSync
            configsyncIp: /Common/internal-localself/address
          failoverAddress:
            class: FailoverUnicast
            address: /Common/internal-localself/address
          failoverGroup:
            class: DeviceGroup
            type: sync-failover
            members:
              - '{{{ HOST1 }}}.example.com'
              - '{{{ HOST2 }}}.example.com'
            owner: /Common/failoverGroup/members/0
            autoSync: true
            saveOnAutoSync: false
            networkFailover: true
            fullLoadOnSync: false
            asmSync: false
          trust:
            class: DeviceTrust
            localUsername: '{{{ USER_NAME }}}'
            localPassword: '{{{ ADMIN_PASS }}}'
            remoteHost: '{{{ REMOTE_SELFIP_INT }}}'
            remoteUsername: '{{{ USER_NAME }}}'
            remotePassword: '{{{ ADMIN_PASS }}}'
    - extensionType: as3
      type: inline
      value:
        class: AS3
        action: deploy
        persist: true
        declaration:
          class: ADC
          schemaVersion: 3.0.0
          label: Sample 1
          remark: An HTTP sample application
          Sample_01:
            class: Tenant
            A1:
              class: Application
              template: http
              serviceMain:
                class: Service_HTTP
                virtualAddresses:
                  - '{{{ PUBLIC_VIP }}}'
                pool: web_pool
              web_pool:
                class: Pool
                monitors:
                  - tcp
                members:
                  - servicePort: 80
                    addressDiscovery: fqdn
                    autoPopulate: true
                    hostname: httpbin.org
    - extensionType: cf
      type: inline
      value:
        class: Cloud_Failover
        environment: azure
        externalStorage:
          scopingTags:
            f5_cloud_failover_label: '{{{ FAILOVER_LABEL }}}'
        failoverAddresses:
          enabled: true
          scopingTags:
            f5_cloud_failover_label: '{{{ FAILOVER_LABEL }}}'
        failoverRoutes:
          enabled: true
          scopingTags:
            f5_cloud_failover_label: '{{{ FAILOVER_LABEL }}}'
          scopingAddressRanges:
            - range: '{{{ MANAGED_ROUTE }}}'
          defaultNextHopAddresses:
            discoveryType: static
            items:
              - '{{{ LOCAL_SELFIP_EXT }}}'
              - '{{{ REMOTE_SELFIP_EXT }}}'
    - extensionType: ts
      type: inline
      value:
        class: Telemetry
        My_System:
          class: Telemetry_System
          systemPoller:
            interval: 60
        My_Listener:
          class: Telemetry_Listener
          port: 6514
        My_Consumer:
          class: Telemetry_Consumer
          type: Azure_Log_Analytics
          workspaceId: '{{{ LAW_ID }}}'
          passphrase:
            cipherText: '{{{ LAW_PRIMKEY }}}'
          useManagedIdentity: false
          region: '{{{ REGION }}}'
post_onboard_enabled:
  - name: create_misc_routes
    type: inline
    commands:
    - tmsh save sys config
JeffGiroux commented 2 years ago

Need better logging in runtime init. Simply showing task async error and 500 code doesn't help much. Are there any other more detailed logging steps you can enable to see more than a 500 error?

shyawnkarim commented 2 years ago

Instructions for how to configure the logging level can be seen here.

Were you using 'silly' when you saw the 500 error?

JeffGiroux commented 2 years ago

It was a slew of problems when I was initially going through my HA BIG-IP builds. I eventually had storage bucket issues or permissions and few other syntax things wrong in the runtime yaml file. I can't say for sure if I did silly logging or not, but I remember trying everything to figure out why it wasn't working.

I can't reproduce the problem now...most likely because of fixed yaml/declaration. Good to know on the logging level documentation. thx...closing. If I get task 500 error again i'll reopen.

shyawnkarim commented 2 years ago

Sounds good. Thanks.