External data group cannot be updated after creation

megamattzilla commented 2 years ago

Environment

Application Services Version: 3.36
BIG-IP Version: 15.1.5.1

Summary

Initial creation of an external data group works as expected. Subsequent updates to the data group through AS3 does not update TMM with the changes.

Steps To Reproduce

Steps to reproduce the behavior:

Submit the following declaration:

{
"class": "AS3",
"action": "deploy",
"persist": true,
"loglevel": "debug",
"declaration": {
"class": "ADC",
"controls": {
  "trace": true,
  "logLevel": "debug"
   },
"schemaVersion": "3.36.0",
  "Common": {
    "class": "Tenant",
    "Shared": {
      "class": "Application",
      "template": "shared",
      "header_to_pool": {
                    "class": "Data_Group",
                    "storageType": "external",
                    "keyDataType": "string",
                    "externalFilePath": "https://raw.githubusercontent.com/megamattzilla/f5PublicConfigCommon/1.0.4/dataGroups/header_to_pool.txt",
                    "ignoreChanges": false,
                    "separator": ":=",
                    "label": "header_to_pool"
                }
      }
    }
  }
}

The URL contains only one entry "192.168.1.1" := "demo_pool",

AS3 is successful

POST change to the declaration

{
"class": "AS3",
"action": "deploy",
"persist": true,
"loglevel": "debug",
"declaration": {
"class": "ADC",
"controls": {
  "trace": true,
  "logLevel": "debug"
   },
"schemaVersion": "3.36.0",
  "Common": {
    "class": "Tenant",
    "Shared": {
      "class": "Application",
      "template": "shared",
      "header_to_pool": {
                    "class": "Data_Group",
                    "storageType": "external",
                    "keyDataType": "string",
                    "externalFilePath": "https://raw.githubusercontent.com/megamattzilla/f5PublicConfigCommon/1.0.5/dataGroups/header_to_pool.txt",
                    "ignoreChanges": false,
                    "separator": ":=",
                    "label": "header_to_pool"
                }
      }
    }
  }
}

The URL contains two entries:

"example.com" := "demo_pool10",
"example2.com" := "demo_pool11",

AS3 is successful.

Configure this iRule to log the data group values in TMM
```
when HTTP_REQUEST {
log local0. "[class get Shared/header_to_pool]"
}
```
Apply iRule to VS and send a HTTP request.

Expected Behavior

Datagroup should contain two entries per the log.

Actual Behavior

Datagroup contains only one entry per the log: May 25 18:52:01 big-ip.local info tmm7[13399]: Rule /Common/test <HTTP_REQUEST>: 192.168.1.1 demo_pool

In addition to the log, you will also find class match against the data group does not work as expected do you to the missing data. If you open the management console and go to System ›› File Management : Data Group File List ›› header_to_pool and click update- TMM is able to see the entire data group entries.

Example log only after clicking update manually:


May 25 19:14:19 big-ip.local info tmm7[13399]: Rule /Common/test <HTTP_REQUEST>: {example2.com demo_pool11} {example.com demo_pool10}

dstokesf5 commented 2 years ago

@megamattzilla thank you for your feedback. Do you happen to know a TMSH equivalent for this action:

If you open the management console and go to System ›› File Management : Data Group File List ›› header_to_pool and click update- TMM is able to see the entire data group entries.

megamattzilla commented 2 years ago

I havent been able to find an exact tmsh command. I see this when I enable debugging and click the button:

May 26 17:49:15 big-ip.local notice mcpd[8090]: 01070417:5: AUDIT - client tmui, user admin - transaction #1234817-2 - object 0 - modify { data_group_file_object { data_group_file_object_name "/Common/Shared/header_to_pool" data_group_file_object_local_path "/tmp/header_to_pool" } } [Status=Command OK]
May 26 17:49:21 big-ip.local notice tmsh[3168]: 01420002:5: AUDIT - pid=3168 user=root folder=/Common module=(tmos)# status=[Command OK] cmd_data=save / sys config partitions all

I did find this command that sort of does what AS3 is doing in terms of updating the URL: tmsh modify /sys file data-group Shared/header_to_pool file source-path https://raw.githubusercontent.com/megamattzilla/f5PublicConfigCommon/1.0.5/dataGroups/header_to_pool.txt

That does trigger TMM/iRules to read the new values. We are evaluating using this as a workaround until a permanent fix is in place.

dstokesf5 commented 2 years ago

Ah, I see. That is just doing a simple modify. I was thinking it might have been similar to some profiles that have an explicit apply step. The root cause here appears to be a tmsh transaction bug. I believe your work-around works because it is doing the update outside of a transaction.

dstokesf5 commented 2 years ago

I have added this issue to our internal product backlog as AUTOTOOL-3326.

dstokesf5 commented 2 years ago

This has been resolved in the 3.38.0 release of AS3.

F5Networks / f5-appsvcs-extension