zscaler / zscaler-sdk-go

Zscaler SDK for Golang (ZPA/ZIA/ZDX/ZCC APIs)
https://zscaler.com
MIT License
10 stars 1 forks source link

GetAllByType return "appConnectorGroup" = nil instead the values #255

Closed chenmal closed 1 month ago

chenmal commented 3 months ago

Confirmation

zscaler-sdk-go version

v2.5.23 - latest

Go environment

1.22

Expected output

The func GetAllByType called with the parameter "GLOBAL_POLICY" should return also the "appConnectorGroup" of the relevant policies rules. ( (service Service) GetAllByType(policyType string) ([]PolicyRuleResource, http.Response, error) )

Actual output

The field "appConnectorGroups" of policy rule that actually contains appp connector groups is null.

image

Code demonstrating the issue

zc.PolicyServiceV2.GetAllByType("GLOBAL_POLICY")

The actual http call, called from GetAllByType. GET https://config.zpabeta.net/mgmtconfig/v1/admin/customers//policySet/rules/policyType/GLOBAL_POLICY

response:

[...
    {
      "id": "***",
      "name": "app connector server group conditions",
      "action": "ALLOW",
      "creationTime": "1717657044",
      "modifiedBy": "****",
      "modifiedTime": "1717657044",
      "operator": "AND",
      "policyType": "1",
      "priority": "1",
      "ruleOrder": "27",
      "conditions": [
        {
          "id": "1418023",
          "creationTime": "1717657044",
          "modifiedBy": "***",
          "modifiedTime": "1717657044",
          "negated": false,
          "operator": "OR",
          "operands": [
            {
              "id": "1418024",
              "creationTime": "1717657044",
              "modifiedBy": "***",
              "modifiedTime": "1717657044",
              "objectType": "CLIENT_TYPE",
              "lhs": "id",
              "rhs": "zpn_client_type_ip_anchoring"
            },
            {
              "id": "1418025",
              "creationTime": "1717657044",
              "modifiedBy": "***",
              "modifiedTime": "1717657044",
              "objectType": "CLIENT_TYPE",
              "lhs": "id",
              "rhs": "zpn_client_type_machine_tunnel"
            }
          ]
        }
      ],
      "appServerGroups": [
        {
          "id": "72059024262037506",
          "name": "Server Group"
        },
        {
          "id": "72059024262038023",
          "name": "Sarit1Server Group"
        }
      ],
      "privilegedCapabilities": {
        "id": ""
      }
    }
...]

The data is missing in the response , so it missing also in PolicyRuleResource return from the function.

UI Http call: Request URL: https://api.zpabeta.net/zpn/api/v1/admin/customers//policySet/rules/policyType/GLOBAL_POLICY/?scopeId=0&page=1&pagesize=100 Request Method: GET

UI http response: assistantGroups contains the values of appConnectorGroups


{
            "id": "72059024262038192",
            "modifiedTime": "1717657044",
            "creationTime": "1717657044",
            "modifiedBy": "***",
            "name": "app connector server group conditions",
            "ruleOrder": "27",
            "priority": "1",
            "policyType": "1",
            "operator": "AND",
            "conditions": [
                {
                    "id": "1418023",
                    "modifiedTime": "1717657044",
                    "creationTime": "1717657044",
                    "modifiedBy": "***",
                    "operator": "OR",
                    "negated": false,
                    "operands": [
                        {
                            "id": "1418024",
                            "modifiedTime": "1717657044",
                            "creationTime": "1717657044",
                            "modifiedBy": "***",
                            "objectType": "CLIENT_TYPE",
                            "lhs": "id",
                            "rhs": "zpn_client_type_ip_anchoring",
                            "name": "zpn_client_type_ip_anchoring"
                        },
                        {
                            "id": "1418025",
                            "modifiedTime": "1717657044",
                            "creationTime": "1717657044",
                            "modifiedBy": "***",
                            "objectType": "CLIENT_TYPE",
                            "lhs": "id",
                            "rhs": "zpn_client_type_machine_tunnel",
                            "name": "zpn_client_type_machine_tunnel"
                        }
                    ]
                }
            ],
            "action": "ALLOW",
            "disabled": "0",
            "appServerGroups": [
                {
                    "id": "72059024262037506",
                    "modifiedTime": "1696620196",
                    "creationTime": "1696545576",
                    "modifiedBy": "***",
                    "name": "Server Group",
                    "enabled": true,
                    "learningEnabled": true,
                    "configSpace": "DEFAULT"
                },
                {
                    "id": "72059024262038023",
                    "modifiedTime": "1714472974",
                    "creationTime": "1714472974",
                    "modifiedBy": "***",
                    "name": "Sarit1Server Group",
                    "enabled": true,
                    "learningEnabled": true,
                    "description": "test",
                    "configSpace": "DEFAULT"
                }
            ],
            "assistantGroups": [
                {
                    "id": "72059024262037540",
                    "modifiedTime": "1696608098",
                    "creationTime": "1696608098",
                    "modifiedBy": "***",
                    "name": "First App Connector Group",
                    "enabled": true,
                    "versionProfileId": "0",
                    "overrideVersionProfile": false,
                    "location": "Boulder, CO, USA",
                    "dnsQueryType": "IPV4_IPV6",
                    "cityCountry": "Boulder, US",
                    "countryCode": "US",
                    "tcpQuickAckApp": false,
                    "tcpQuickAckAssistant": false,
                    "tcpQuickAckReadAssistant": false,
                    "praEnabled": false,
                    "useInDrMode": false,
                    "wafDisabled": false,
                    "siemAssistantGroup": false
                },
                {
                    "id": "72059024262037963",
                    "modifiedTime": "1713095542",
                    "creationTime": "1713095542",
                    "modifiedBy": "***",
                    "name": "app connector group 1",
                    "enabled": true,
                    "versionProfileId": "0",
                    "overrideVersionProfile": false,
                    "location": "Israel",
                    "dnsQueryType": "IPV4_IPV6",
                    "cityCountry": "בית רימון, IL",
                    "countryCode": "IL",
                    "tcpQuickAckApp": false,
                    "tcpQuickAckAssistant": false,
                    "tcpQuickAckReadAssistant": false,
                    "praEnabled": false,
                    "useInDrMode": false,
                    "wafDisabled": false,
                    "siemAssistantGroup": false
                }
            ],
            "defaultRule": false,
            "restrictedEntity": false
        }```

### Steps to reproduce

1.Add Access Policy
2.Choose "App Connector Selection Method" -> "Specific App Connector groups or Server groups for the application
.."
3.Choose "App Connector Groups" -> few values 
4. Use the latest version of the sdk - policysetcontrollerv2 -> zc.PolicyServiceV2.GetAllByType("GLOBAL_POLICY")
5. Search for the new policy rule -> get the  PolicyRuleResource ->AppConnectorGroups value

The result is nil instead the app connector groups    

### References

No
willguibr commented 3 months ago

@chenmal You have not shared the code you are using for this test. Please share relevant parts of your code and we will investigate further. Also, please set the following environment variables before executing your code, in order to capture the debug logs with the request and response.

export ZSCALER_SDK_LOG=true
export ZSCALER_SDK_VERBOSE=true

In our tests we do not see such issue based on the simple script and the output below. Script:

func main() {
    policyType := "GLOBAL_POLICY"

    client, err := tests.NewZpaClient()
    if err != nil {
        log.Fatalf("Error creating client: %v", err)
    }

    service := policysetcontrollerv2.New(client)

    policies, _, err := service.GetAllByType(policyType)
    if err != nil {
        log.Fatalf("Error retrieving policies: %v", err)
    }

    fmt.Printf("Retrieved %d policies of type %s:\n", len(policies), policyType)
    for _, policy := range policies {
        fmt.Printf("ID: %s, Name: %s\n", policy.ID, policy.Name)
    }
}

Output:

{"totalPages":"1","totalCount":"0","list":[{"id":"72059124120061469","modifiedTime":"1717693640","creationTime":"1717693640","modifiedBy":"72059124120027137","name":"Example100","description":"Example100","ruleOrder":"1","priority":"2","policyType":"1","operator":"AND","action":"ALLOW","disabled":"0","defaultRule":false,"appConnectorGroups":[{"id":"72059124120050866","modifiedTime":"1714967837","creationTime":"1714967837","modifiedBy":"72059124120027170","name":"test_zpa_app_connector_group_FT5aY","enabled":true,"description":"test_zpa_app_connector_group_FT5aY","versionProfileId":"0","overrideVersionProfile":true,"location":"San Jose, CA, USA","dnsQueryType":"IPV4_IPV6","cityCountry":"San Jose, US","countryCode":"US","tcpQuickAckApp":true,"tcpQuickAckAssistant":true,"tcpQuickAckReadAssistant":true,"praEnabled":true,"useInDrMode":true,"wafDisabled":false,"lssAppConnectorGroup":false}]},{"id":"72059124120061470","modifiedTime":"1717693649","creationTime":"1717693649","modifiedBy":"72059124120027137","name":"Example200","description":"Example200","ruleOrder":"2","priority":"1","policyType":"1","operator":"AND","action":"ALLOW","disabled":"0","defaultRule":false,"appConnectorGroups":[{"id":"72059124120050866","modifiedTime":"1714967837","creationTime":"1714967837","modifiedBy":"72059124120027170","name":"test_zpa_app_connector_group_FT5aY","enabled":true,"description":"test_zpa_app_connector_group_FT5aY","versionProfileId":"0","overrideVersionProfile":true,"location":"San Jose, CA, USA","dnsQueryType":"IPV4_IPV6","cityCountry":"San Jose, US","countryCode":"US","tcpQuickAckApp":true,"tcpQuickAckAssistant":true,"tcpQuickAckReadAssistant":true,"praEnabled":true,"useInDrMode":true,"wafDisabled":false,"lssAppConnectorGroup":false}]}]}{"totalPages":"1","totalCount":"0","list":[{"id":"72059124120061469","modifiedTime":"1717693640","creationTime":"1717693640","modifiedBy":"72059124120027137","name":"Example100","description":"Example100","ruleOrder":"1","priority":"2","policyType":"1","operator":"AND","action":"ALLOW","disabled":"0","defaultRule":false,"appConnectorGroups":[{"id":"72059124120050866","modifiedTime":"1714967837","creationTime":"1714967837","modifiedBy":"72059124120027170","name":"test_zpa_app_connector_group_FT5aY","enabled":true,"description":"test_zpa_app_connector_group_FT5aY","versionProfileId":"0","overrideVersionProfile":true,"location":"San Jose, CA, USA","dnsQueryType":"IPV4_IPV6","cityCountry":"San Jose, US","countryCode":"US","tcpQuickAckApp":true,"tcpQuickAckAssistant":true,"tcpQuickAckReadAssistant":true,"praEnabled":true,"useInDrMode":true,"wafDisabled":false,"lssAppConnectorGroup":false}]},{"id":"72059124120061470","modifiedTime":"1717693649","creationTime":"1717693649","modifiedBy":"72059124120027137","name":"Example200","description":"Example200","ruleOrder":"2","priority":"1","policyType":"1","operator":"AND","action":"ALLOW","disabled":"0","defaultRule":false,"appConnectorGroups":[{"id":"72059124120050866","modifiedTime":"1714967837","creationTime":"1714967837","modifiedBy":"72059124120027170","name":"test_zpa_app_connector_group_FT5aY","enabled":true,"description":"test_zpa_app_connector_group_FT5aY","versionProfileId":"0","overrideVersionProfile":true,"location":"San Jose, CA, USA","dnsQueryType":"IPV4_IPV6","cityCountry":"San Jose, US","countryCode":"US","tcpQuickAckApp":true,"tcpQuickAckAssistant":true,"tcpQuickAckReadAssistant":true,"praEnabled":true,"useInDrMode":true,"wafDisabled":false,"lssAppConnectorGroup":false}]}]}
chenmal commented 3 months ago

The "policies" output you display is not in "PolicyRuleResource" -> "AppConnectorGroups" (The structure I see contains only id and name).

Can you show me the implementation of the function you used? I used this function from policysetconttollerv2


func (service *Service) GetAllByType(policyType string) ([]PolicyRuleResource, *http.Response, error) {
    relativeURL := fmt.Sprintf(mgmtConfigV1+service.Client.Config.CustomerID+"/policySet/rules/policyType/%s", policyType)
    list, resp, err := common.GetAllPagesGenericWithCustomFilters[PolicyRuleResource](service.Client, relativeURL, common.Filter{MicroTenantID: service.microTenantID})
    if err != nil {
        return nil, nil, err
    }
    return list, resp, nil
}
type PolicyRuleResource struct {
      ID                       string                         `json:"id,omitempty"`
      Name                     string                         `json:"name,omitempty"`
      Description              string                         `json:"description,omitempty"`
      Action                   string                         `json:"action,omitempty"`
      ActionID                 string                         `json:"actionId,omitempty"`
      CreationTime             string                         `json:"creationTime,omitempty"`
      ModifiedBy               string                         `json:"modifiedBy,omitempty"`
      ModifiedTime             string                         `json:"modifiedTime,omitempty"`
      AuditMessage             string                         `json:"auditMessage,omitempty"`
      CustomMsg                string                         `json:"customMsg,omitempty"`
      Operator                 string                         `json:"operator,omitempty"`
      PolicySetID              string                         `json:"policySetId,omitempty"`
      PolicyType               string                         `json:"policyType,omitempty"`
      Priority                 string                         `json:"priority,omitempty"`
      ReauthIdleTimeout        string                         `json:"reauthIdleTimeout,omitempty"`
      ReauthTimeout            string                         `json:"reauthTimeout,omitempty"`
      RuleOrder                string                         `json:"ruleOrder,omitempty"`
      ZpnCbiProfileID          string                         `json:"zpnCbiProfileId,omitempty"`
      ZpnIsolationProfileID    string                         `json:"zpnIsolationProfileId,omitempty"`
      ZpnInspectionProfileID   string                         `json:"zpnInspectionProfileId,omitempty"`
      ZpnInspectionProfileName string                         `json:"zpnInspectionProfileName,omitempty"`
      MicroTenantID            string                         `json:"microtenantId,omitempty"`
      MicroTenantName          string                         `json:"microtenantName,omitempty"`
      Conditions               []PolicyRuleResourceConditions `json:"conditions,omitempty"`
      AppConnectorGroups       []AppConnectorGroups           `json:"connectorGroups,omitempty"`
      AppServerGroups          []AppServerGroups              `json:"appServerGroups,omitempty"`
      ServiceEdgeGroups        []ServiceEdgeGroups            `json:"serviceEdgeGroups,omitempty"`
      Credential               *Credential                    `json:"credential,omitempty"`
      PrivilegedCapabilities   PrivilegedCapabilities         `json:"privilegedCapabilities,omitempty"`
}
type AppConnectorGroups struct {
    ID   string `json:"id,omitempty"`
    Name string `json:"name,omitempty"`
}
chenmal commented 3 months ago

The issue is :

In GetAllByType PolicyRuleResource is the return value of the policies list,

list, resp, err := common.GetAllPagesGenericWithCustomFilters[**PolicyRuleResource**](service.Client, relativeURL, common.Filter{MicroTenantID: service.microTenantID})...

in the json response - the data of app connector group key is "appConnectorGroups" , and the PolicyRuleResource-> AppConnectorGroups json key is : connectorGroups

AppConnectorGroups []AppConnectorGroups `json:"connectorGroups,omitempty"`

So PolicyRuleResource->AppConnectorGroups return as null.

chenmal commented 3 months ago

Another issue : when using func (service *Service) UpdateRule(policySetID, ruleId string, policySetRule *PolicyRule) (*http.Response, error) and sending as param policysetcontrollerv2.PolicyRule with app connector group, it not updated.

The ignorance from app connector group (and server group) happen also in ConvertV1ResponseToV2Request.

willguibr commented 3 months ago

There's a lot to unpack here.

  1. Bear in mind that there were several API upstream changes recently, where we have introduced the policy-set-controller v2, which has a slightly difference in the payload structure during POST/PUT calls; however, the response (GET) is still returned in V1 format.
  2. So, if you are using the policysetcontrollerv2, we are accounting for the response processing to be returned in v1 instead.
  3. The attributes such as: connectorGroups has also changed in the upstream API even though it's still backwards compatible with both v1 and v2 policy-set-controller APIs. So, nothing wrong with the SDK structure in this case. Either way, we'll update to reflect the current attribute.
  4. I am not entirely sure what you mean by "PolicyRuleResource->AppConnectorGroups return as null." Based on our tests the entire payload for the AppConnectorGroups block is being returned. Notice that in the struct we purposefully added only the attributes ID and Name, for simplicity, since the SDK was designed originally to support our Terraform providers and only the ID attribute is required during CRUD operations. So, I am not not entirely sure why is is being pointed as a potential problem.
  5. When updating the appConnectorGroups using the policysetcontrollerv2 the endpoints require that you pass both the ID and Name of the app connectors. This is a change in behavior in the new v2 api and not applicable when using policysetcontroller (v1)
  6. ConvertV1ResponseToV2Request: This is a dedicated function specifically designed to convert the response from V2 into V1 format, and is not locally used in the SDK functions in anyway. This function was put in place to support the ZPA Terraform provider, to simplify the way we process the READ function into the Terraform state. This function may be removed eventually or moved to the actual Terraform provider itself. Either way, this is function does not influence anything related to your original question. Below is a sample program you can use for your tests:

    func main() {
    client, err := tests.NewZpaClient()
    if err != nil {
        log.Fatalf("Error creating client: %v", err)
    }
    
    service := policysetcontrollerv2.New(client)
    
    // Create a new policy rule
    newRule := &policysetcontrollerv2.PolicyRule{
        Name:        "Example255",
        Description: "Example255",
        Operator:    "AND",
        Action:      "ALLOW",
        PolicySetID: "", // Provide your policySetId
    }
    
    createdRule, _, err := service.CreateRule(newRule)
    if err != nil {
        log.Fatalf("Error creating policy rule: %v", err)
    }
    
    fmt.Printf("Created policy rule with ID: %s, Name: %s\n", createdRule.ID, createdRule.Name)
    
    // Update the policy rule with the AppConnectorGroups attribute block
    createdRule.AppConnectorGroups = []policysetcontrollerv2.AppConnectorGroups{
        {
            ID: "123456789", // V2 API now requires both the ID and Name. A 500 error will be returned if only one attribute is passed.
            Name: "Example255",
        },
    }
    
    _, err = service.UpdateRule(createdRule.PolicySetID, createdRule.ID, createdRule)
    if err != nil {
        log.Fatalf("Error updating policy rule: %v", err)
    }
    
    fmt.Printf("Updated policy rule with ID: %s, Name: %s\n", createdRule.ID, createdRule.Name)
    }

    To test v1 simply change the package to policysetcontroller instead. If you have any further questions, please raise a support case with our support team, and ask it to re-routed to the appropriate team.

chenmal commented 2 months ago

Hi, thanks for your answer.

The issue I am dealing with is related to the policySetControllerV2's functions. When using the policyRule and policyRuleResource objects provided in this controller, the appConnectorGroup value is not populated in policyRule and policyRuleResource.

(Ignore the value if it is part of the request, and do not return it as part of the response.)

I found the reason: it is because the response of the HTTP call returns the value under the json key "appConnectorGroups", but the objects in policySetControllerV2 are defined like this: AppConnectorGroups []AppConnectorGroups `json:"connectorGroups,omitempty"`

So, when I use functions like GetAllByType, the HTTP call response correctly returns the "appConnectorGroups" value, but when converting the response to the object, it ignores the "appConnectorGroups" value because it does not match the JSON key of any field in the object.

If the field were defined like this:

AppConnectorGroups []AppConnectorGroups `json:"appConnectorGroups,omitempty"` it would work correctly.

willguibr commented 2 months ago

@chenmal Please take a look at the latest version of the SDK v2.61.3 where we have fixed the attribute AppConnectorGroups. This attribute named has been changed over the last few months in the upstream API gateway, which then caused the problem.

chenmal commented 2 months ago

Its seems that it solve the issue partially ,Thank you very much for it.

The issue I found is: If you will update a PolicyRule, and delete all the AppConnectorGroups/AppServerGroups/ServiceEdgeGroups/Conditions, The update action will not update those fields, it will happen because the fields are tagged as "omitempty", and when we will pass empty slice as parameter, the Encode action (encode the object to json body) will ignore those fields and those fields will not be updated. @willguibr

func (client *Client) getRequest(method, urlPath string, options, body interface{}) (*http.Request, error) { var buf io.ReadWriter if body != nil { buf = new(bytes.Buffer) err := json.NewEncoder(buf).Encode(body) if err != nil { return nil, err } } .....

AppConnectorGroups       []AppConnectorGroups           `json:"appConnectorGroups,omitempty"`
AppServerGroups          []AppServerGroups              `json:"appServerGroups,omitempty"`
ServiceEdgeGroups        []ServiceEdgeGroups            `json:"serviceEdgeGroups,omitempty"`
Conditions               []PolicyRuleResourceConditions `json:"conditions,omitempty"`
willguibr commented 2 months ago

@chenmal Please check version 2.61.5 and let us know if it addresses the issue you're stating. Regards

github-actions[bot] commented 1 month ago

This issue has been marked stale because there has been no activity within the last 14 days. To keep this issue active, remove the stale label.