List Identity Schemas needs pagination memory

shipperizer commented 1 year ago

kratos public endpoints /schema and /schema/{id} return successfully:

shipperizer in ~/shipperizer/identity-platform-admin-ui on IAM-366 ● ● λ http :4433/schemas page==0 page_size==2 
HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Length: 1751
Content-Type: application/json; charset=utf-8
Date: Thu, 24 Aug 2023 13:09:38 GMT
Link: <https://10.64.140.43/iam-kratos/schemas?page=0&per_page=2>; rel="first"
Set-Cookie: csrf_token_c172ca5a6cb7f832a9b8b4e40baf9b6219f6491e8bcb19122e8e335690c83c10=3P70sWIWF3DSCF6NgVdJlCXhvakFWyiI6TVeYyR2RH0=; Path=/; Max-Age=31536000; HttpOnly; Secure; SameSite=Lax
Vary: Origin
Vary: Cookie
X-Total-Count: 2

[
    {
        "id": "social_user_v0",
        "schema": {
            "$id": "https://schemas.canonical.com/presets/kratos/user_v0.json",
            "$schema": "http://json-schema.org/draft-07/schema#",
            "properties": {
                "additionalProperties": true,
                "traits": {
                    "properties": {
                        "address": {
                            "title": "Address",
                            "type": "string"
                        },
                        "birthdate": {
                            "title": "Birthdate",
                            "type": "string"
                        },
                        "email": {
                            "format": "email",
                            "title": "E-Mail",
                            "type": "string"
                        },
                        "family_name": {
                            "title": "Family Name",
                            "type": "string"
                        },
                        "gender": {
                            "title": "Gender",
                            "type": "string"
                        },
                        "given_name": {
                            "title": "Given Name",
                            "type": "string"
                        },
                        "last_name": {
                            "title": "Last Name",
                            "type": "string"
                        },
                        "locale": {
                            "title": "Locale",
                            "type": "string"
                        },
                        "middle_name": {
                            "title": "Middle Name",
                            "type": "string"
                        },
                        "name": {
                            "title": "Name",
                            "type": "string"
                        },
                        "nickname": {
                            "title": "Nickname",
                            "type": "string"
                        },
                        "phone_number": {
                            "title": "Phone Number",
                            "type": "string"
                        },
                        "picture": {
                            "title": "Picture",
                            "type": "string"
                        },
                        "preferred_username": {
                            "title": "Preferred Username",
                            "type": "string"
                        },
                        "profile": {
                            "title": "Profile",
                            "type": "string"
                        },
                        "website": {
                            "title": "Website",
                            "type": "string"
                        },
                        "zoneinfo": {
                            "title": "Zoneinfo",
                            "type": "string"
                        }
                    },
                    "required": [
                        "email"
                    ],
                    "type": "object"
                }
            },
            "title": "Person",
            "type": "object"
        }
    },
    {
        "id": "admin_v0",
        "schema": {
            "$id": "https://schemas.canonical.com/presets/kratos/admin_v0.json",
            "$schema": "http://json-schema.org/draft-07/schema#",
            "properties": {
                "additionalProperties": true,
                "traits": {
                    "properties": {
                        "email": {
                            "format": "email",
                            "minLength": 3,
                            "ory.sh/kratos": {
                                "verification": {
                                    "via": "email"
                                }
                            },
                            "title": "E-Mail",
                            "type": "string"
                        },
                        "name": {
                            "title": "Name",
                            "type": "string"
                        },
                        "phone_number": {
                            "title": "Phone Number",
                            "type": "string"
                        },
                        "username": {
                            "ory.sh/kratos": {
                                "credentials": {
                                    "password": {
                                        "identifier": true
                                    }
                                }
                            },
                            "title": "Username",
                            "type": "string"
                        }
                    },
                    "type": "object"
                }
            },
            "title": "Admin Account",
            "type": "object"
        }
    }
]


shipperizer in ~/shipperizer/identity-platform-admin-ui on IAM-366 ● λ http :4433/schemas/admin_v0
HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Length: 1003
Content-Type: application/json
Date: Thu, 24 Aug 2023 11:02:55 GMT
Vary: Origin
Vary: Cookie

{
    "$id": "https://schemas.canonical.com/presets/kratos/admin_v0.json",
    "$schema": "http://json-schema.org/draft-07/schema#",
    "properties": {
        "additionalProperties": true,
        "traits": {
            "properties": {
                "email": {
                    "format": "email",
                    "minLength": 3,
                    "ory.sh/kratos": {
                        "verification": {
                            "via": "email"
                        }
                    },
                    "title": "E-Mail",
                    "type": "string"
                },
                "name": {
                    "title": "Name",
                    "type": "string"
                },
                "phone_number": {
                    "title": "Phone Number",
                    "type": "string"
                },
                "username": {
                    "ory.sh/kratos": {
                        "credentials": {
                            "password": {
                                "identifier": true
                            }
                        }
                    },
                    "title": "Username",
                    "type": "string"
                }
            },
            "type": "object"
        }
    },
    "title": "Admin Account",
    "type": "object"
}

the proxied /api/v0/schema is always empty unless you pass the size==1:

shipperizer in ~/shipperizer/identity-platform-admin-ui on IAM-366 ● λ http :8000/api/v0/schemas
HTTP/1.1 200 OK
Content-Length: 75
Content-Type: application/json
Date: Thu, 24 Aug 2023 11:12:26 GMT
Vary: Origin

{
    "_meta": null,
    "data": [],
    "message": "List of Identity Schemas",
    "status": 200
}

shipperizer in ~/shipperizer/kratos-operator on IAM-362-add-configmap λ http :8000/api/v0/schemas page==1 size==1
HTTP/1.1 200 OK
Content-Length: 679
Content-Type: application/json
Date: Thu, 24 Aug 2023 13:11:17 GMT
Vary: Origin

{
    "_meta": null,
    "data": [
        {
            "id": "admin_v0",
            "schema": {
                "$id": "https://schemas.canonical.com/presets/kratos/admin_v0.json",
                "$schema": "http://json-schema.org/draft-07/schema#",
                "properties": {
                    "additionalProperties": true,
                    "traits": {
                        "properties": {
                            "email": {
                                "format": "email",
                                "minLength": 3,
                                "ory.sh/kratos": {
                                    "verification": {
                                        "via": "email"
                                    }
                                },
                                "title": "E-Mail",
                                "type": "string"
                            },
                            "name": {
                                "title": "Name",
                                "type": "string"
                            },
                            "phone_number": {
                                "title": "Phone Number",
                                "type": "string"
                            },
                            "username": {
                                "ory.sh/kratos": {
                                    "credentials": {
                                        "password": {
                                            "identifier": true
                                        }
                                    }
                                },
                                "title": "Username",
                                "type": "string"
                            }
                        },
                        "type": "object"
                    }
                },
                "title": "Admin Account",
                "type": "object"
            }
        }
    ],
    "message": "List of Identity Schemas",
    "status": 200
}

shipperizer in ~/shipperizer/kratos-operator on IAM-362-add-configmap λ http :8000/api/v0/schemas page==2 size==1
HTTP/1.1 200 OK
Content-Length: 75
Content-Type: application/json
Date: Thu, 24 Aug 2023 13:11:20 GMT
Vary: Origin

{
    "_meta": null,
    "data": [],
    "message": "List of Identity Schemas",
    "status": 200
}

the detailed view works all the time for all the schemas

shipperizer in ~/shipperizer/kratos-operator on IAM-362-add-configmap λ http :8000/api/v0/schemas/admin_v0         
HTTP/1.1 200 OK
Content-Length: 681
Content-Type: application/json
Date: Thu, 24 Aug 2023 13:18:01 GMT
Vary: Origin

{
    "_meta": null,
    "data": [
        {
            "id": "admin_v0",
            "schema": {
                "$id": "https://schemas.canonical.com/presets/kratos/admin_v0.json",
                "$schema": "http://json-schema.org/draft-07/schema#",
                "properties": {
                    "additionalProperties": true,
                    "traits": {
                        "properties": {
                            "email": {
                                "format": "email",
                                "minLength": 3,
                                "ory.sh/kratos": {
                                    "verification": {
                                        "via": "email"
                                    }
                                },
                                "title": "E-Mail",
                                "type": "string"
                            },
                            "name": {
                                "title": "Name",
                                "type": "string"
                            },
                            "phone_number": {
                                "title": "Phone Number",
                                "type": "string"
                            },
                            "username": {
                                "ory.sh/kratos": {
                                    "credentials": {
                                        "password": {
                                            "identifier": true
                                        }
                                    }
                                },
                                "title": "Username",
                                "type": "string"
                            }
                        },
                        "type": "object"
                    }
                },
                "title": "Admin Account",
                "type": "object"
            }
        }
    ],
    "message": "Detail of Identity Schemas",
    "status": 200
}

shipperizer in ~/shipperizer/kratos-operator on IAM-362-add-configmap λ http :8000/api/v0/schemas/social_user_v0
HTTP/1.1 200 OK
Content-Length: 1220
Content-Type: application/json
Date: Thu, 24 Aug 2023 13:18:07 GMT
Vary: Origin

{
    "_meta": null,
    "data": [
        {
            "id": "social_user_v0",
            "schema": {
                "$id": "https://schemas.canonical.com/presets/kratos/user_v0.json",
                "$schema": "http://json-schema.org/draft-07/schema#",
                "properties": {
                    "additionalProperties": true,
                    "traits": {
                        "properties": {
                            "address": {
                                "title": "Address",
                                "type": "string"
                            },
                            "birthdate": {
                                "title": "Birthdate",
                                "type": "string"
                            },
                            "email": {
                                "format": "email",
                                "title": "E-Mail",
                                "type": "string"
                            },
                            "family_name": {
                                "title": "Family Name",
                                "type": "string"
                            },
                            "gender": {
                                "title": "Gender",
                                "type": "string"
                            },
                            "given_name": {
                                "title": "Given Name",
                                "type": "string"
                            },
                            "last_name": {
                                "title": "Last Name",
                                "type": "string"
                            },
                            "locale": {
                                "title": "Locale",
                                "type": "string"
                            },
                            "middle_name": {
                                "title": "Middle Name",
                                "type": "string"
                            },
                            "name": {
                                "title": "Name",
                                "type": "string"
                            },
                            "nickname": {
                                "title": "Nickname",
                                "type": "string"
                            },
                            "phone_number": {
                                "title": "Phone Number",
                                "type": "string"
                            },
                            "picture": {
                                "title": "Picture",
                                "type": "string"
                            },
                            "preferred_username": {
                                "title": "Preferred Username",
                                "type": "string"
                            },
                            "profile": {
                                "title": "Profile",
                                "type": "string"
                            },
                            "website": {
                                "title": "Website",
                                "type": "string"
                            },
                            "zoneinfo": {
                                "title": "Zoneinfo",
                                "type": "string"
                            }
                        },
                        "required": [
                            "email"
                        ],
                        "type": "object"
                    }
                },
                "title": "Person",
                "type": "object"
            }
        }
    ],
    "message": "Detail of Identity Schemas",
    "status": 200
}

as docs go https://github.com/ory/kratos-client-go/blob/v0.13.1/api_identity.go#L2188 pages don't necessarily follow each and i was not able to fund the second schema

investigate a solution on how to get a reliable behaviour

nsklikas commented 1 year ago

After trying it for a little, I found the following: Even though the API docs state that page must be >1, the value can be <1 and if we set page=0 in our requests as default value then we get the expected behavior.

TBH their pagination logic seems really weird. Additionally, the Links header that is returned does not contain the link to the first page if you are on the first page and it does not contain the link to the last page if you are on the last page.

We will need to set the default page to 0 for this endpoint. I will have to look if this can work for the other endpoints or if we will need special logic for this one.

nsklikas commented 1 year ago

To add on this, Kratos and Hydra have different pagination logic.

On Kratos if we request for a page that is out of bounds, we will get no entries back. E.g. if Kratos has 3 identities and we request for page=10, size=2 then we will get an empty response.
In Hydra if we request for a page that is out of bounds, we will get the last page back. E.g. if Hydra has 3 clients and we request for page=10, size=2 then we will get the last 2 entries (the final page).

IMHO the hydra behavior is better. The page parameter should limit the returned values, but it should not cause the response to have no values. Are we OK with having different behaviors for different the hydra and kratos APIs?

We could make a second call if the Kratos response is empty so as to get the last page, but I don't think it is worth it. As long as the UI makes the calls correctly it should be fine.

syncronize-issues-to-jira[bot] commented 5 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/IAM-782.

This message was autogenerated

edlerd commented 5 months ago

This issue surfaced again on the ui implementation. It is currently not possible to read the page=0 of schemas. The endpoint skips the first page and responds with page=1 instead. It is exactly the same problem as described in #256 just for schemas here instead of identities there.

canonical / identity-platform-admin-ui

List Identity Schemas needs pagination memory #44