extensionUrl not working with remote URLs with BIG-IP 15.1.2

chen23 commented 3 years ago

When you specify an external URL for extensionUrl the process hangs on installing the extension. This occurs on 15.1.2, but does not occur on 15.1.0.4

Example that fails

runtime_parameters: []
pre_onboard_enabled:
  - name: provision_rest
    type: inline
    commands:
      - /usr/bin/setdb provision.extramb 500
      - /usr/bin/setdb restjavad.useextramb true
      - /usr/bin/setdb setup.run false
extension_packages:
  install_operations:
    - extensionType: as3
      extensionVersion: 3.24.0
      extensionUrl: https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.24.0/f5-appsvcs-3.24.0-5.noarch.rpm
extension_services:
    service_operations: []

example that works

---
runtime_parameters: []
pre_onboard_enabled:
  - name: provision_rest
    type: inline
    commands:
      - /usr/bin/setdb provision.extramb 500
      - /usr/bin/setdb restjavad.useextramb true
      - /usr/bin/setdb setup.run false
extension_packages:
  install_operations:
    - extensionType: as3
      extensionVersion: 3.24.0
      extensionUrl: file:///var/config/rest/downloads/f5-appsvcs-3.24.0-5.noarch.rpm
extension_services:
    service_operations: []

andreykashcheev commented 3 years ago

Thanks for reporting this issue!

I have created an internal ticket to investigate this: ESECLDTPLT-2398

chen23 commented 3 years ago

I'm not sure whether the root cause is remote URLs. Here's an example where it failed because the backend BIG-IP was having restjavad errors. Having additional logging enabled/options would be helpful to understand when/where a problem is happening with the process.

shyawnkarim commented 3 years ago

Thanks for providing this screenshot. I have attached the pic and comment to the ticket.

mikeshimkus commented 3 years ago

Hi @chen23, I'm unable to repro this error on v15.1.2. If using the default log level, have you tried bumping it up to debug or silly?

https://github.com/F5Networks/f5-bigip-runtime-init#logging

thanks

chen23 commented 3 years ago

RPM installation is completed.
2020-12-19T02:54:37.250Z [20731]: info: Configuration file: /config/cloud/runtime-init-conf.yaml
2020-12-19T02:54:37.257Z [20731]: silly: F5 Telemetry is enabled.
2020-12-19T02:54:37.257Z [20731]: info: Validating provided declaration
2020-12-19T02:54:37.306Z [20731]: info: Successfully validated declaration
2020-12-19T02:54:37.308Z [20731]: silly: Making request: GET http://localhost:8100/mgmt/tm/sys/ready verifyTls: true
2020-12-19T02:54:37.360Z [20731]: silly: Request response: 200 {"kind":"tm:sys:ready:readystats","selfLink":"https://localhost/mgmt/tm/sys/ready?ver=15.1.2","entries":{"https://localhost/mgmt/tm/sys/ready/0":{"nestedStats":{"entries":{"configReady":{"description":"yes"},"licenseReady":{"description":"yes"},"provisionReady":{"description":"yes"}}}}}}
2020-12-19T02:54:37.360Z [20731]: info: Resolving parameters
2020-12-19T02:54:37.361Z [20731]: info: Executing custom pre-onboard commands
2020-12-19T02:54:37.363Z [20731]: debug: Executing inline shell command: /usr/bin/setdb provision.extramb 500
2020-12-19T02:54:39.147Z [20731]: info: Shell command: /usr/bin/setdb provision.extramb 500 execution completed; response:
2020-12-19T02:54:39.148Z [20731]: debug: Executing inline shell command: /usr/bin/setdb restjavad.useextramb true
2020-12-19T02:54:39.324Z [20731]: info: Shell command: /usr/bin/setdb restjavad.useextramb true execution completed; response:
2020-12-19T02:54:39.324Z [20731]: debug: Executing inline shell command: /usr/bin/setdb setup.run false
2020-12-19T02:54:39.670Z [20731]: info: Shell command: /usr/bin/setdb setup.run false execution completed; response:
2020-12-19T02:54:39.670Z [20731]: info: Executing install operations.
2020-12-19T02:54:39.672Z [20731]: silly: Making request: GET http://localhost:8100/mgmt/shared/iapp/global-installed-packages verifyTls: true
2020-12-19T02:54:39.702Z [20731]: silly: Request response: 200 {"items":[],"generation":0,"kind":"shared:iapp:global-installed-packages:installedpackagecollectionstate","lastUpdateMicros":0,"selfLink":"https://localhost/mgmt/shared/iapp/global-installed-packages"}
2020-12-19T02:54:39.703Z [20731]: info: Installing - as3 3.24.0
2020-12-19T02:54:39.706Z [20731]: silly: Downloading File: https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.24.0/f5-appsvcs-3.24.0-5.noarch.rpm
2020-12-19T02:54:39.706Z [20731]: silly: Options: {"verifyTls":true}

mikeshimkus commented 3 years ago

Looks like it got past the first error, but then logging stopped dead when downloading AS3?

Can you tell me:

Which cloud this is running in?
If running in Azure, are you using the waagent provisioner or customData?
Can you share the full script that installs and executes f5-bigip-runtime-init?

thanks

chen23 commented 3 years ago

This was running in AWS with a single NIC. Script below (you can omit the S3 parts)

#!/bin/bash

mkdir -p /config/cloud
cat << 'EOF' > /config/cloud/runtime-init-conf.yaml
---
runtime_parameters: []
pre_onboard_enabled:
  - name: provision_rest
    type: inline
    commands:
      - /usr/bin/setdb provision.extramb 500
      - /usr/bin/setdb restjavad.useextramb true
      - /usr/bin/setdb setup.run false
extension_packages:
  install_operations:
    - extensionType: as3
      extensionVersion: 3.24.0
      extensionUrl: https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.24.0/f5-appsvcs-3.24.0-5.noarch.rpm
extension_services:
    service_operations: []
post_onboard_enabled:
    - name: custom-config
      type: inline
      commands:
        - bash /config/custom-config.sh

EOF
cat << 'EOF' > /config/custom-config.sh
#!/bin/bash
sleep 60
source /usr/lib/bigstart/bigip-ready-functions
wait_bigip_ready

PYTHONPATH=/opt/aws/awscli-1.10.26/lib/python2.7/site-packages/ /opt/aws/awscli-1.10.26/bin/aws s3 cp s3://${s3_bucket}/admin.shadow /config/admin.shadow --region ${region}
tmsh modify /auth user admin encrypted-password $(cat /config/admin.shadow)
tmsh modify auth user admin shell bash

tmsh save sys config
rm -f /config/admin.shadow
PYTHONPATH=/opt/aws/awscli-1.10.26/lib/python2.7/site-packages/ /opt/aws/awscli-1.10.26/bin/aws s3 rm s3://${s3_bucket}/admin.shadow --region ${region}

EOF

source /usr/lib/bigstart/bigip-ready-functions
wait_bigip_ready

for i in {1..30}; do
    curl -fv --retry 1 --connect-timeout 5 -L "https://cdn.f5.com/product/cloudsolutions/f5-bigip-runtime-init/v1.1.0/dist/f5-bigip-runtime-init-1.1.0-1.gz.run" -o "/var/config/rest/downloads/f5-bigip-runtime-init-1.1.0-1.gz.run" && break || sleep 10
done
bash /var/config/rest/downloads/f5-bigip-runtime-init-1.1.0-1.gz.run -- '--cloud aws'

F5_BIGIP_RUNTIME_INIT_LOG_LEVEL=silly f5-bigip-runtime-init --config-file /config/cloud/runtime-init-conf.yaml

F5Networks / f5-bigip-runtime-init

extensionUrl not working with remote URLs with BIG-IP 15.1.2 #5