Determine Handling for API Pagination Properties

[bflad]

Some APIs utilize pagination or markers for handling operations which can return a lot of data.

For example in the Scaleway OpenAPI specification there is:

paths:
  /instance/v1/zones/{zone}/servers:
    get:
      tags:
      - Servers
      operationId: ListServers
      summary: List all servers
      parameters:
      - in: path
        name: zone
        description: The zone you want to target
        required: true
        schema:
          type: string
          description: The zone you want to target
          enum:
          - fr-par-1
          - fr-par-2
          - fr-par-3
          - nl-ams-1
          - nl-ams-2
          - pl-waw-1
          - pl-waw-2
      - in: query
        name: per_page
        description: A positive integer lower or equal to 100 to select the number
          of items to return.
        schema:
          type: integer
          description: A positive integer lower or equal to 100 to select the number
            of items to return.
          default: "50"
      - in: query
        name: page
        description: A positive integer to choose the page to return.
        schema:
          type: integer
          description: A positive integer to choose the page to return.
          default: 1

Which currently results in the associated code generation specification:

    "datasources": [
        {
            "name": "instance_servers",
            "schema": {
                "attributes": [
                    {
                        "name": "per_page",
                        "int64": {
                            "computed_optional_required": "computed_optional",
                            "description": "A positive integer lower or equal to 100 to select the number of items to return."
                        }
                    },
                    {
                        "name": "page",
                        "int64": {
                            "computed_optional_required": "computed_optional",
                            "description": "A positive integer to choose the page to return."
                        }
                    },

These low-level API details are never intended to be surfaced in Terraform, however they are quite critical to the operation of the API in the provider logic.

In the schema-only generation use case:

• Provider developers will need a way to omit these properties from the Terraform schema

In the full generation use case:

• The code generation specification will need a way to surface this API operation handling information for provider code
• This converter will need a way to omit these from the Terraform schema while filling in the API operation handling details

[bflad]

One quick proposal in this space could be introducing the following converter configuration:

data_sources:
  TYPE:
    read:
      path: /instance/v1/zones/{zone}/servers
      method: GET
      # New!
      pagination:
       selection_property: page
       size_property: per_page

Initially, the combination of those property names would be used to omit them from the Terraform schema. If/When the code generation specification supports passing API operation pagination information, then the converter could use these values to fill in that information.

There may also be merit to also introducing a separate "just omit these properties" converter configuration, but that feels like it could be addressed separately if/when the use case exists.

[austinvalle]

Note for future readers: If you want to ignore these properties from the Terraform schema, you can ignore them with the recently introduced schema.ignores functionality in v0.2.0, documented here.

Example ignoring the properties listed in this issue:

data_sources:
  TYPE:
    read:
      path: /instance/v1/zones/{zone}/servers
      method: GET
      schema:
       ignores:
         - page
         - per_page

This issue is still a valid future enhancement, as whenever a provider code spec can described operations for code generation, this pagination information will likely become relevant again.