Policies and opensearchserverless

[ljordan-pea]

First off, thanks the amazing library, this is a major dependency in a number of our projects.

Dependencies

com.cognitect.aws/opensearchserverless {:mvn/version "825.2.1266.0"}

Description

The goal, get the policy documents attached to opensearch-serverless security-policy objects.

Setup, using the AWS console I created a new collection, called "deleteme-07" of type SEARCH. Following the wizard, this creates several ancillary objects, including two 'security-policy' objects, one of type 'network' and one of type 'encryption'. These are populated with a policy document (i.e. json) based on your selection during the creation process.

running:

aws opensearchserverless --profile ZZZ get-security-policy  --name auto-deleteme-07 --type "network" --region us-east-1

gets you something like:

{
    "securityPolicyDetail": {
        "createdDate": 1677523259855,
        "description": "deleteme-07",
        "lastModifiedDate": 1677523259855,
        "name": "auto-deleteme-07",
        "policy": [
            {
                "Rules": [
                    {
                        "Resource": [
                            "collection/deleteme-07"
                        ],
                        "ResourceType": "dashboard"
                    },
                    {
                        "Resource": [
                            "collection/deleteme-07"
                        ],
                        "ResourceType": "collection"
                    }
                ],
                "AllowFromPublic": true
            }
        ],
        "policyVersion": "MTY3NzUyMzI1OTg1NV8x",
        "type": "network"
    }
}

Things to note, the policy field contains an array of objects. Those objects have a 'Rules' field, which contain the meat of the matter.

When I execute a similar command via aws-api:

(aws/invoke _client {:op :GetSecurityPolicy :request {:name "auto-deleteme-07" :type "network"}})

I get back:

{:securityPolicyDetail {:createdDate 1677523259855,
                        :description "deleteme-07",
                        :lastModifiedDate 1677523259855,
                        :name "auto-deleteme-07",
                        :policy {},
                        :policyVersion "MTY3NzUyMzI1OTg1NV8x",
                        :type "network"}}

Note that the 'policy' field is an empty map '{}'

I would expect to find either:

1. The json of the policy document as a String, or
2. An array of maps describing the policy (better option?)

When I run:

(aws/doc _client :GetSecurityPolicy)

I get

-------------------------
GetSecurityPolicy

<p>Returns information about a configured OpenSearch Serverless security policy. For more information, see <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-network.html">Network access for Amazon OpenSearch Serverless</a> and <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-encryption.html">Encryption at rest for Amazon OpenSearch Serverless</a>.</p>

-------------------------
Request

{:name string, :type [:one-of ["encryption" "network"]]}

Required

[:name :type]

-------------------------
Response

{:securityPolicyDetail
 {:createdDate long,
  :description string,
  :lastModifiedDate long,
  :name string,
  :policy {},
  :policyVersion string,
  :type [:one-of ["encryption" "network"]]}}

Which I think, wrongly, shows :policy as a map in the response. It should be [] I think (or just the json string)

Conclusion

Primary issue, not seeing the policy content at all. Secondly issue or maybe cause, the type of policy is an array not a map.

Thanks! -Lucas

[dchelimsky]

aws-api builds these structures based on specs we read from AWS, and, in this case, what you're seeing is accurately reflecting what we're getting from AWS. I'm not saying this is not a bug, or even that we can't fix it (though I have no idea how we might yet), but just documenting for you and whomever on our team might try to tackle this.

Bear with me because this is a bit convoluted:

The spec is defined in opensearchserverless-2021-11-01.min.json.

The GetSecurityPolicy operation specifies an output type of securityPolicyDetail, which has the shape "S1r", which is a "structure" (i.e. a map) with a "policy" key, whose type is "S12", which is another "structure" defined like this:

    "S12": {
      "type": "structure",
      "members": {},
      "document": true
    },

So AWS is telling us that "policy" is a map, but that it has no specified keys.

[dchelimsky]

So AWS is telling us that "policy" is a map, but that it has no specified keys.

Clarification: since it doesn't tell aws-api anything about what's in the map, aws-api has no way to convert that into a payload for you.

[dchelimsky]

One thing that might help us here is that there is a key in that description "document" true. I'm pretty sure that's the key to solving this, since aws-api doesn't look for that at the moment.

[ljordan-pea]

Thanks for the update. That matches my expectations. I suspect there might be a similar issue for the command :CreateSecurityPolicy, which also states a {} for the policy... that might be the right type of thing in the end, but it is different from the cli and CloudFormation which have a [] for the policy.

-Lucas

[dchelimsky]

Here's all of the output shapes that have :document true (after keywordizing the maps):

({:service "gamesparks",
  :document-shapes {:Document {:type "structure", :members {}, :documentation "<p/>", :document true}}}
 {:service "omics", :document-shapes {:RunParameters {:type "structure", :members {}, :document true}}}
 {:service "iottwinmaker", :document-shapes {:QueryResultValue {:type "structure", :members {}, :document true}}}
 {:service "identitystore",
  :document-shapes
  {:AttributeValue
   {:type "structure",
    :members {},
    :documentation
    "<p>The value of the attribute. This is a <code>Document</code> type. This type is not supported by Java V1, Go V1, and older versions of the AWS CLI.</p>",
    :document true}}}
 {:service "resource-explorer-2", :document-shapes {:Document {:type "structure", :members {}, :document true}}}
 {:service "tnb", :document-shapes {:Document {:type "structure", :members {}, :document true}}}
 {:service "opensearchserverless", :document-shapes {:Document {:type "structure", :members {}, :document true}}}
 {:service "kendra",
  :document-shapes
  {:Template
   {:type "structure",
    :members {},
    :documentation
    "<p>The template schema used for the data source, where templates schemas are supported.</p> <p>See <a href=\"https://docs.aws.amazon.com/kendra/latest/dg/ds-schemas.html\">Data source template schemas</a>.</p>",
    :document true}}})

[dchelimsky]

Fixed in a532fe1c14794dd8e4f9fd55750d4519ad218b8a. I'll leave this open until it's released.

[dchelimsky]

Released in v 0.8.652

[dchelimsky]

FYI, I just dropped artifact releases for all the services that have document in their output payloads. With opensearchserverless-845.2.1326.0:

> (require '[cognitect.aws.client.api :as aws])
> (def oss (aws/client {:api :opensearchserverless}))
> (aws/doc opensearchserverless :GetSecurityPolicy)
-------------------------
GetSecurityPolicy

<p>Returns information about a configured OpenSearch Serverless security policy. For more information, see <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-network.html">Network access for Amazon OpenSearch Serverless</a> and <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-encryption.html">Encryption at rest for Amazon OpenSearch Serverless</a>.</p>

-------------------------
Request

{:name string, :type [:one-of ["encryption" "network"]]}

Required

[:name :type]

-------------------------
Response

{:securityPolicyDetail
 {:createdDate long,
  :description string,
  :lastModifiedDate long,
  :name string,
  :policy document,
  :policyVersion string,
  :type [:one-of ["encryption" "network"]]}}