Email claim not present on ID token after issuing with refresh and webhook enabled

Preflight checklist

[X] I could not find a solution in the existing issues, docs, nor discussions.
[X] I agree to follow this project's Code of Conduct.
[X] I have read and am following this repository's Contribution Guidelines.
[X] I have joined the Ory Community Slack.
[X] I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

Issue Summary

The ID token lacks the email claim when issued using a refresh token, despite having a configured webhook as described in the Ory Hydra documentation.

This issue was initially reported in issue #3852, which was subsequently closed. However, further investigation has allowed us to isolate the problem with more precision.

Reproducing the bug

Generate a new Ory environment.

Create a OAuth2 client with scopes openid, offline_access and email.

Validate works without webhook

Using ex. Postman go through a OIDC flow and validate what ID token has email claim.

Also validate, that ID token after issuing with refresh token, has email claim.

curl --location 'https://PROJECT-SLUG.projects.oryapis.com/oauth2/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=refresh_token' \
--data-urlencode 'refresh_token=REFRESH_TOKEN' \
--data-urlencode 'client_id=CLIENT_ID' \
--data-urlencode 'scope=openid email offline_access'

Enable webhook and see email claim disappear

Generate a minimal client which can be used as webhook. Example

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net/http"
)

func claimsHandler(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "Only POST methid is allowed", http.StatusMethodNotAllowed)
        return
    }

    var data map[string]interface{}
    err := json.NewDecoder(r.Body).Decode(&data)
    if err != nil {
        http.Error(w, "Invalid JSON input", http.StatusBadRequest)
        return
    }

    // Convert the received JSON object to a pretty-printed JSON string
    prettyJSON, err := json.MarshalIndent(data, "", "  ")
    if err != nil {
        log.Printf("Error formatting JSON: %v", err)
    } else {
        fmt.Printf("Received JSON:\n%s\n", string(prettyJSON))
    }

    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]interface{}{}) // Respond with empty JSON
    fmt.Printf("Done...\n")
}

func main() {

    http.HandleFunc("/claims", claimsHandler)

    fmt.Println("Starting server on :8080...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("Server failed to start: %v", err)
    }
}

Create a local tunnel ex. by using ngrok.

Enable webhook following documentation https://www.ory.sh/docs/hydra/guides/claims-at-refresh#webhook-payload

ory patch oauth2-config --project PROJECT --workspace WORKSPACE \
--add '/oauth2/token_hook/url="https://TUNNEL/claims"' \
--add '/oauth2/token_hook/auth/type="api_key"' \
--add '/oauth2/token_hook/auth/config/in="header"' \
--add '/oauth2/token_hook/auth/config/name="X-API-Key"' \
--add '/oauth2/token_hook/auth/config/value="SOME_API_KEY"' \
--format yaml

Again do a OIDC flow (using ex. Postman). First time token endpoint is called after login, we receive

{
  ..
  "session": {
    ..
    "extra": {},
    "id_token": {
      ..
      "id_token_claims": {
        "acr": "",
        "amr": [
          ..
        ],
        "at_hash": "",
        "aud": [
          ..
        ],
        ..
        "ext": {
          "email": "some@email.com",
          "sid": "30142f7e-50c5-4ff0-8469-447ffd9555a8"
        },
        ..
      },
      ..
    },
    ..
  }
}

and ID token has email claim

{
  ..
  "email": "some@email.com",
  ..
}

first time we call token endpoint with refresh token we correctly in webhook get

{
  ..
  "session": {
    ..
    "extra": {},
    "id_token": {
      ..
      "id_token_claims": {
        "acr": "",
        "amr": [
          ..
        ],
        "at_hash": "",
        "aud": [
          ..
        ],
        ..
        "ext": {
          "email": "some@email.com",
          "sid": "30142f7e-50c5-4ff0-8469-447ffd9555a8"
        },
        ..
      },
      ..
    },
    ..
  }
}

but ID token is missing email claim.

Now second time we refresh token and print request in webhook, ext is missing

{
  ..
  "session": {
    ..
    "extra": {},
    "id_token": {
      ..
      "id_token_claims": {
        "acr": "",
        "amr": [
          ..
        ],
        "at_hash": "",
        "aud": [
          ..
        ],
        ..
      },
      ..
    },
    ..
  }
}

and still no email claim on ID token.

I tried to look into the code, and it may seems like the response body is overwriting ID token extra claim. We however send a empty response back. Could this be the issue? https://github.com/ory/hydra/blob/0ce9d7a0d479222951fdf1cde3367e7b91228a45/oauth2/token_hook.go#L149

Relevant log output

No response

Relevant configuration

No response

Version

Ory hosted

On which operating system are you observing this issue?

None

In which environment are you deploying?

None

Additional Context

No response

I can “keep the email claim alive” if I extract it from the request and add it as part of the response of the webhook.

Hence if I change the handler to

func claimsHandler(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "Only POST methid is allowed", http.StatusMethodNotAllowed)
        return
    }

    var data map[string]interface{}
    err := json.NewDecoder(r.Body).Decode(&data)
    if err != nil {
        http.Error(w, "Invalid JSON input", http.StatusBadRequest)
        return
    }

    // Convert the received JSON object to a pretty-printed JSON string
    prettyJSON, err := json.MarshalIndent(data, "", "  ")
    if err != nil {
        log.Printf("Error formatting JSON: %v", err)
    } else {
        fmt.Printf("Received JSON:\n%s\n", string(prettyJSON))
    }

    // Navigate the nested structure to extract "email"
    email := ""
    if session, ok := data["session"].(map[string]interface{}); ok {
        if idToken, ok := session["id_token"].(map[string]interface{}); ok {
            if idTokenClaims, ok := idToken["id_token_claims"].(map[string]interface{}); ok {
                if ext, ok := idTokenClaims["ext"].(map[string]interface{}); ok {
                    if emailVal, ok := ext["email"].(string); ok {
                        email = emailVal
                    }
                }
            }
        }
    }

    // If email is not found, return an error
    if email == "" {
        http.Error(w, "Email field is missing in the JSON input", http.StatusBadRequest)
        return
    }

    response := map[string]interface{}{
        "session": map[string]interface{}{
            "id_token": map[string]string{
                "email": email,
                "hello": "world",
            },
        },
    }

    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(response)
    fmt.Printf("Done...\n")
}

Then my ID token keep the email claim, but I can see the claim sid (which is also part of ext) disappears.

ory / hydra