search

F5Networks / f5-appsvcs-extension

F5 BIG-IP Application Services 3 Extension
Apache License 2.0
164 stars 53 forks source link

Command failed: tmsh -a list ltm data-group internal /Common/appsvcs/settings #565

Closed Otimun closed 8 months ago

Otimun commented 2 years ago

Environment

Summary

When trying to do an initial as3 call to create partitions, the following error is encountered:

│ Error: Getting AS3 Version failed with HTTP 500 :: {"code":500,"message":"Command failed: tmsh -a list ltm data-group internal /Common/appsvcs/settings\n01020036:3: The requested value list (/Common/appsvcs/settings) was not found.\n"} │ │ with module.f5.bigip_as3.partition_creation, │ on ../modules/f5/bigip.tf line 16, in resource "bigip_as3" "partition_creation": │ 16: resource "bigip_as3" "partition_creation" {

Unable to continue with the automatic deployment of a new F5.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Submit the following declaration:

    {
"class": "AS3",
"action": "deploy",
"persist": true,
"declaration": {
    "class": "ADC",
    "schemaVersion": "3.34.0",
    "id": "initial-as3-config",
    "label": "Initial",
    "remark": "Initial as3 config to create partitions"
    "Test" : {
        "class" : "Tenant",
        "defaultRouteDomain" : 1,
        "initial" : {
            "class" : "Application",
            "VS-initial" : {
                "class" : "Service_HTTP",
                "virtualAddresses" : [
                    "1.2.3.4"
                ]
            }
        }
    },
    "Test2" : {
        "class" : "Tenant",
        "defaultRouteDomain" : 2,
        "initial" : {
            "class" : "Application",
            "VS-initial" : {
                "class" : "Service_HTTP",
                "virtualAddresses" : [
                    "1.2.3.4"
                ]
            }
        }
    }
}
}

  2. Observe the following error response: Mon, 21 Feb 2022 16:41:57 GMT - severe: [appsvcs] {"message":"Error encountered while building requestContext. Command failed: tmsh -a list ltm data-group internal /Common/appsvcs/settings\n01020036:3: The requested value list (/Common/appsvcs/settings) was not found.\n","level":"error"}

Expected Behavior

This initial call is made to create partitions so that the vlans and selfips can be created in the partition.

Expected behavior is for AS3 not to call on a command that cannot be found.

Actual Behavior

It seems AS3 is calling a tmsh command that cannot be found in this standalone device.: tmsh -a list ltm data-group internal /Common/appsvcs/settings 01020036:3: The requested value list (/Common/appsvcs/settings) was not found.

the command without /Common/appsvcs/settings does give a normal response.

Due to this error the terraform AS3 call fails, stopping the deployment.

Otimun commented 2 years ago

Perhaps associated with the fix of #525 ?

Otimun commented 2 years ago

After testing the same code, I can confirm that the issue does not present itself when using 3.33.0

dstokesf5 commented 2 years ago

Thank you for your feedback. I have added this issue to our internal product backlog as AUTOTOOL-3027.

dstokesf5 commented 2 years ago

@Otimun Is this device in an HA cluster?

Otimun commented 2 years ago

No, this device is configured standalone.

dstokesf5 commented 2 years ago

Thank you. We are having a difficult time reproducing the issue. We suspect that a restart of restnoded could fix the problem. We believe that might be why going back to a previous version appeared to fix the issue. Installing AS3 causes a restnoded restart.

Could you try the latest version again to see if the issue returns for you?

Otimun commented 2 years ago

Hi @dstokesf5, I've just rebuild the same environment installing 3.34.0 and I get the same error again.

These are the logs for the install:

null_resource.install_as3[0] (local-exec): Uploading RPM to https://x.x.x.x/mgmt/shared/file-transfer/uploads/f5-appsvcs-3.34.0-4.noarch.rpm
null_resource.install_as3[0]: Still creating... [10s elapsed]
null_resource.install_as3[0]: Still creating... [20s elapsed]
null_resource.install_as3[0]: Still creating... [30s elapsed]
null_resource.install_as3[0]: Still creating... [40s elapsed]
null_resource.install_as3[0] (local-exec): Installing f5-appsvcs-3.34.0-4.noarch.rpm on x.x.x.x
null_resource.install_as3[0]: Still creating... [50s elapsed]
null_resource.install_as3[0] (local-exec): Waiting for /info endpoint to be available
null_resource.install_as3[0]: Still creating... [1m0s elapsed]
null_resource.install_as3[0]: Still creating... [1m10s elapsed]
null_resource.install_as3[0] (local-exec): Installed f5-appsvcs-3.34.0-4.noarch.rpm on x.x.x.x
null_resource.install_as3[0]: Creation complete after 1m16s [id=6843809222254658644]

and the logs when trying to create the partitions using as3:

module.f5.bigip_as3.partition_creation: Creating...
╷
│ Error: Getting AS3 Version failed with HTTP 500 :: {"code":500,"message":"Command failed: tmsh -a list ltm data-group internal /Common/appsvcs/settings\n01020036:3: The requested value list (/Common/appsvcs/settings) was not found.\n"}
│ 
│   with module.f5.bigip_as3.partition_creation,
│   on ../modules/f5/bigip.tf line 16, in resource "bigip_as3" "partition_creation":
│   16: resource "bigip_as3" "partition_creation" {
│ 
╵

in restnode.log I find the same log:

Wed, 23 Feb 2022 14:48:11 GMT - severe: [appsvcs] {"message":"Error encountered while building requestContext. Command failed: tmsh -a list ltm data-group internal /Common/appsvcs/settings\n01020036:3: The requested value list (/Common/appsvcs/settings) was not found.\n","level":"error"}

After restarting restnoded as you mentioned, the problem disappears. This leaves me with the question: Does this mean with this version that I need to restart the restnoded before I can continue with the rest of my deployment code?

Otimun commented 2 years ago

Just to make sure, I have executed another test this time installing 3.33.0 directly after building the environment, however with 3.33.0 I do not need to do a restart of restnoded before it starts working. It seems that something changed between the two versions that requires this restart of restnoded.

dstokesf5 commented 2 years ago

Thank you for all of this additional information. We are still working on reproducing the issue and this is helpful in narrowing down the root cause. Can you also provide the full BIG-IP version you are using so we can rule out any BIG-IP version issues?

Otimun commented 2 years ago

The current F5 BIG-IP version I am using is BIG-IP 15.1.4 Build 0.0.47 Final. Just built a cluster (this time) using 3.33.0 and not experiencing any issues.

dstokesf5 commented 2 years ago

Thank you. I have another question from the developer looking into this:

How are you setting up the route-domains on the BIG-IP? Are you using DO or doing it manually?

Otimun commented 2 years ago

The route domains are set up via DO, I use Terraform to concatenate a static do json file with a for loop over a template file to add the route domains dynamically, depending on how many I need based on the partitions declared.

So the order of execution is this: terraform to create the VMs, using cloud-init and F5 disk image download the latest DO and AS3 images upload DO image Install DO upload AS3 image install AS3 terraform to install license with Big IQ DO to install basic config, route domains etc AS3 to create the partitions terraform to create the rest of Layer 2 such as VLANs and Self IPs

dstokesf5 commented 9 months ago

It looks like there have been improvements in this part of the project. Are you still experiencing this issue with the latest release?

github-actions[bot] commented 8 months ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.