Addition of a DH-side endpoint for querying the status of a consent establishment flow

[markskript]

Description

There is a general consensus in the Accredited Data Recipient (ADR) community that there are higher-than-acceptable levels of dropoffs during the consent flow after the ADR has handed off the customer to the data holder. Right now there is no way of determining the status of a consent flow after the handoff has occurred, unless the data holder completes either a \'successful\' or \'error\' hand back to the ADR platform, which is not always feasible.

Area Affected

• Consent flow

Change Proposed

We propose the following options to help address the issue, with Option 3 being our recommended approach.

Option 1: Manual case raising in the CDR Portal

This is the current solution, and we feel it is inadequate, especially as consent volumes grow in the eco-system. Ignoring the manual effort involved by the ADR, data holder and ACCC in managing the support cases, there are considerable gaps in the information that is collected prior to the handoff from ADR to data holder which makes it hard to provide the data holder with enough \'useful\' information to debug on a case-by-case basis.

Option 2: We rely on summarised data from GetMetrics

The GetMetrics V4/V5 enhancements add more granular reporting from the data holder back to the DSB on dropoff cases, and while this is great for viewing trends at the eco-system level, it does not assist in debugging specific consumer issues on a case-by-case basis.

Option 3: A new data holder side endpoint for real-time querying

We propose a new endpoint be added at the data-holder-side that allows ADR\'s to pass in the request_uri value from the PAR and obtain a \'status\'. The proposed design of that endpoint is as follows.

Data Holder's OpenID Provider Metadata proposed new property:

• pushed_authorization_request_introspection_endpoint: The URL of the PAR Introspection End Point

Example

GET /.well-known/openid-configuration HTTP/1.1
Host: www.dh.com.au
HTTP/1.1 200 OK
Content-Type: application/json

{
  ...
  "pushed_authorization_request_introspection_endpoint": "https://data.holder.com.au/par/introspect",
  ...
}

PAR Introspection End Point

Description	Value
Hosted By	Data Holder
Transport Security	MTLS
Client Authentication Required	Yes
Bearer Token Required	No

HTTP Method: POST

The location of the Data Holder PAR Introspection End Point is determined by the pushed_authorization_request_introspection_endpoint in the Data Holder\'s OpenID Provider metadata.

This end point will be implemented according to the following:

• Data Holders MUST publish their PAR Introspection End Point using their OpenID Provider Metadata Discovery End Point

• The request MUST include the following parameters using the application/x-www-form-urlencoded format in the HTTP request entity-body:

  • request_uri: The response value from the PAR request used to obtain the status.

• A PAR Introspection End Point Response should include the following fields:

  • status: Enum, mandatory, status of the consent flow
  • description: string, optional, human-readable state description with more information

• Data Holders MUST return the appropriate consent flow status during the period of when the PAR is provided to AT LEAST 24 hours after the latest action on the consent, regardless of the PAR expires_in value. Valid requests outside that time window should return UNKNOWN_REQUEST_URI status.

Status Enumerated Values

Where logical, the following status values have been aligned with the V5 GetMetrics standard.

Value	Description
UNKNOWN_REQUEST_URI	The Data Holder cannot recognise the request_uri value, or more than 15 minutes have passed since the latest action on the consent.
REQUEST_URI_PROVIDED	The request_uri value is recognised, but the authorisation endpoint has not been invoked yet.
TIMED_OUT	The URL expired before invoking the authorisation endpoint.
PRE_IDENTIFICATION	The client loaded the authorisation endpoint page and we are waiting for the client to enter the details.
PRE_AUTHENTICATION	The client provided the initial information, such as the client id. An One Time Password (OTP) has been sent, and we are waiting for	the client to proceed.
MANUAL_STEP_REQUIRED	The client is shown a message that a manual step is required before a consent can be completed.
ABANDONED	A timeout occurred while waiting for a client action in the UI.
PRE_ACCOUNT_SELECTION	The client is presented with the list of accounts.
NO_ACCOUNTS_AVAILABLE	There are no accounts available for the client.
PRE_AUTHORISATION	Waiting for the final confirmation from the client.
CODE_PENDING	The client was redirected back to the redirect_url with a JARM, but the /token endpoint with grant_type=authorization_code and the code was not invoked - a refresh token has not been provided yet.
SUCCESS	Consent was successfully granted, and a refresh token is provided.
UNKNOWN_ERROR	Occurs when an unknown error happens. This can only be used for states that are not explicitly enumerated.

Response Codes

The following responses are in addition to error responses covered by normative references. Error scenarios in the following table MUST use the error structure defined in the Payload Conventions.

Response Code	Situation	Description
200 OK	Success	The consent flow status is provided successfully.
400 Bad Request	Request is not valid	The request is not valid, for instance when request_uri is not provided.

Non-Normative Request Example

POST https://data.holder.com.au/par/introspect
Host: data.holder.com.au
Content-Type: application/x-www-form-urlencoded

client_id=afb73e61-a6a6-4cf4-9a49-13018809f9f5&
client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&
client_assertion=eyJhbGciOiJQUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjEyMyJ9.ey\...&
request_uri=urn%3Adata.holder.com.au%3Abwc4JK-ESC0w8acc191e-Y1LTC2

Successful Response Example

HTTP/1.1 200 OK
Content-Type: application/json

{
  "status": "AUTHENTICATION_PENDING",
  "description": "The client was presented the authorisation screen"
}

Error Response Example

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "errors": [
    {
      "code": "urn:au-cds:error:cds-all:Field/Missing",
      "title": "Missing Required Field",
      "detail": "request_uri"
    }
  ]
}

Non-functional requirements

It is expected that this introspection endpoint will only be called in non-successful and long-lasting flows, and as such, could be defined as Low Priority as per https://consumerdatastandardsaustralia.github.io/standards/#performance-requirements

Scope of change

Note that this option does not currently allow for the "recovery" of a consent flow, and while we see this as being potentially valuable we feel it increases the scope of the change too much to be warranted at this stage.

Feedback

We welcome further suggestions, discussions and feedback on the above from everyone, especially from the data holders on how best to approach this to minimise implementation effort as much as possible, acknowledging that many data holders have outsourced this part of their platform.

[perlboy]

Implementing this presupposes that all PAR requests will involve an interactive consent flow of a single individual. While that's broadly (but not entirely) true for now, future modes as a result of https://github.com/ConsumerDataStandardsAustralia/standards/issues/327 will invalidate this. I think there's probably more longevity in establishing a way of tracking the lifecycle of an arrangement (of any form) from the beginning (i.e. pre authorisation) rather than the status of a specific PAR.

[markskript]

Thanks for the feedback @perlboy !

When putting together this request we tried to balance the size of the change vs the benefit. Tracking arrangements end-to-end I agree would be optimal longer term (potentially as a mandatory output of DP327?). What I was hoping for here was a, comparatively, "quick win" for what we feel is one of the major issues blocking CDR uptake. Or, at least something that will stem discussion on ways to address this problem in the short-to-medium term.

[joshuanicholson]

@markskript Thanks for raising this; broadly speaking, we are in agreement with you that dropoffs are a significant concern, and some form of tracking & ability to determine status would assist all stakeholders regarding consumer support.

Your suggestion makes a lot of sense, and again, we'd support this, even though we acknowledge & agree with the issue raised by @perlboy . This issue has rattled around internally without any clear winning solution. Liquid stronger than coffee was required.

We will consider this some more and give a more detailed piece of feedback soon

[DougFromPayPal]

Are there any statistics on how many of these dropouts are happening per DH? I'd like to understand how widespread this issue is, given there may be a decent amount of development work on our side to support. If I missed a link or mention to said statistics - my apologies!

[markskript]

The ACCC presented some statistics on one of the implementation calls late last year. These were based on situations where the problem was raised as a ticket in the CDR support portal. I know that one of the other ADRs reported much higher percentages, that due to the nature of their product were never reported as the customer just "gave up and went away".

For us, the percentages are high. This is predominantly because we work with business/corporate customers and are running afoul of the pre-consent requirements of some DHs (i.e. paper forms that need to be completed before the consent is established).

[markverstege]

Hi @markskript, I definitely agree having a mechanism to query the status of an authorisation on the DH side is valuable. I don't agree with using the request_uri or extending the PAR endpoint to cater for this. The request_uri defined in section 2.2 of RFC9126 recommends a 90 second expiry time. Further to this, there is no guarantee that the request_uri remains valid after presentation at the Authorization Endpoint. The expiry time is concerned with the grace period for the relying party to receive the request_uri then present it to the Authorization Endpoint. The consequence is that how IAM implementations deal with the request_uri after presentation is non deterministic and there is no guarantee it will persist in a data store after presentation.

Leveraging or extending PAR would be off spec and create custom code on an oAuth endpoint. If you're open to it, can I suggest we use this thread to explore alternative options.

I also support @DougFromPayPal's comments that getting some insight into the frequency of this issue would be helpful.

What we need is a way to look up the status of an authorisation after the user has been redirected to the Data Holder (in other words, after a call is made to the Authorization Endpoint). This could (should) also provide a way to inspect the properties of the authorisation after its establishment. This would then provide a mechanism to check on the status of what the consumer has authorised and allow the RP to poll the endpoint periodically to identify any changes (either status or grant details such as the list of accounts associated with the grant)

Outcomes:

1. Obtain the status of an authorisation during establishment
2. Obtain the properties (including status) of the authorisation after establishment
3. A mechanism to discover the endpoint
4. A state machine for the authorisation
5. A way for a consumer to obtain a non-repudiable authorisation receipt like a physical paper receipt that provides proof of authorisation.

Please feel free to suggest changes or additional outcomes.

Solution Properties:

binding_identifier: We need some binding identifier that connects the lodgement of the authorisation request to the status of the authorisation and the final binding of the authorisation response. inspection_endpoint: An endpoint to inspect the status of the authorisation related to the binding_identifier.

Binding Identifier For the binding_identifier we could extend the utility of the cdr_arrangement_id to allow RPs to present an arrangement identifier they have generated with their own algorithm. Alternatively we could look at an RP generated consent_id and continue to have the cdr_arrangement_id be a DH issued identifier only.

Inspection Endpoint For the inspection_endpoint I am intentionally avoiding calling this an introspection endpoint because we don't have a signed/encrypted token we are inspecting.

This sort of capability has been envisaged for a while and I see a few options: (a) We could use the Grant Management API but the GM API needs some work to better define behaviour in these scenarios.

(b) We could define a "consent" API where the RP presents a binding_identifier and gets back the details of the authorisation.

(c) A lodged intent pattern is used to stage the authorisation and in return receives a binding_identifier that it can then call as a resource outside the lifecycle of the authorisation. This is effectively (b) but the DH creates the “binding_identifier” as a resource id.

Personally I prefer (a) as we can define our requirements to feed into an upstream spec that would have vendor supportability.

Discovery I envisage we may want to version both the state machine enumeration, the inspection request and the inspection response payload. How we deal with versioning is in part dictated by the inspection endpoint approach.

State Machine I think what @markskript has proposed is a great start. I do wonder if a URN format for the enumeration is better suited rather than strings.

Other questions to consider:

1. What versioning is required? Especially if we want to extend or change the state machine.
2. Are there any scenarios where the status of the authorisation should not be disclosed or limited disclosure be provided?
3. Should this solution also act as a “CDR Authorisation Receipt”?
4. What client and consumer binding properties are required? These may change (before authorisation a consumer is not known, but afterwards it is)
5. How long after authorisation expiry can it be inspected? If this is a CDR Authorisation Receipt we may want this to live on for a long time (e.g. years)
6. What other security concerns are relevant?
7. Are there any extensibility requirements? That is to say, should Data Holders be able to extend this with their own functionality according to the Extensibility standards?
8. What transition considerations exist, and can we retrospectively apply this to existing active authorisations and/or expired authorisations?
9. Do we also need a way to obtain change history to the authorisation (not just the current state)?

[markskript]

Thanks @markverstege - that is fantastic feedback. Will review and post if we have any feedback on your proposal. We were just trying to find a "quick win" knowing the challenges with resourcing at the DH's.

[AGL-CDR]

AGL requests that before we consider introducing a new end point, that we should confirm:

a) The higher-than-acceptable levels of drop offs during consent flow are occurring at DH end inappropriately, and not as a direct result of Rules and Standards, etc. b) That the introduction of this additional end point will have the direct result that drop offs will abate.

We have recently performed an identical review having received this feedback internally and the most common reasons for drop off were:

• Consumers were attempting to share data that was not their own (not the account holder)
• Consumers were attempting to access CDR using details that did not match their registered online account
• Consumers were ineligible, no active accounts present with DH
• Consumers did not enter the OTP provided and instead entered their online account password

With these observations in mind, I am skeptical that the introduction of a new end point would lead directly to the reduction of drop offs in the consent flow.

[dpostnikov]

There is a general consensus in the Accredited Data Recipient (ADR) community that there are higher-than-acceptable levels of drop offs during the consent flow after the ADR has handed off the customer to the data holder.

• What is an acceptable level of drop off? Where is it defined?
• How many customers are allowed to change their mind?
• How many customers are allowed to forget their password and drop out of their flow?

Apart from us having not set any benchmarks to refer to, we are probably dealing with a number of reasons from different domains that need to be dealt with separately.

Right now there is no way of determining the status of a consent flow after the handoff has occurred, unless the data holder completes either a 'successful' or 'error' hand back to the ADR platform, which is not always feasible.

Are we mixing technical request issues with UI user behaviour and consent management?

Our technical authorisation flow is based on the FAPI family of standards. We should not be hacking this technical flow that is implemented by many implementers and vendors in the ecosystem, formally proofed, conformance tested and deployed globally.

1. If we are trying to expose the status of the consent / grant / authorisation we should be using grant management APIs as @markverstege suggested. BUT, the only internally defined status available are: ACTIVE, REVOKED and EXPIRED. And also, UI / user behaviour tracking should not be added to consent management as this is typically managed by an Authorisation Server software.

2. We can’t use PAR for request tracking (pre-consent). Or we shouldn’t be if you follow the standards. PAR specification is designed to offer a one-way backchannel delivery mechanism for an authorisation request. There is no requirement for an Authorisation Server implementing PAR specification to manage a life cycle of the request (although implied internal status would be ACTIVE, USED or EXPIRED) or even to keep the request past its use or expiry. And also, UI / user behaviour tracking should not be added to PAR request implementation as this is typically managed by an Authorisation Server software.

3. Unfortunately, there is no standard mechanism to track drop off events related to CX journey, the ones mentioned in the original request. GetMetric tried to do some of this tracking but I can’t see how this can be consistently implemented to deliver tangible results.

There is definitely a value in exposing the status of the consent (#1) but it won’t give you the complete solution for drop offs.

I suggest trying to understand the issues your customers are experiencing one by one before trying to implement any solution even if it’s just reporting.

Some of the examples of different solutions to different problems:

1. I can guarantee that if the CDR lets Data Holders use their existing channels / apps (aka x2app and x2web) and existing authentication methods, you would see a significant improvement in drop off rates and we wouldn’t need to fix CDR’s custom OTP flow now.. See more here.

2. If we implement Grant Management discussed in 2020, we would have:

   • fine grained consent management which adds more privacy to CDR and enables a key building block for action initiation.
   • data recipient’s ability to interrogate consent status This is also supported by further discussion in 2021 and in the feedback paper from ABA members https://github.com/ConsumerDataStandardsAustralia/standards/issues/182#issuecomment-885412178.

3. If we replace custom arrangement revocation notification with a profile of Shared Signals and Events, we would be able to share other events via the same standard based pipe.

I appreciate why the ticket was raised by @markskript. We (the ecosystem) definitely need to try to improve customer's experience reduce drop offs where it makes sense.

We can review consumer issues, categorise them into technical, CX, rules or operational. Once we list the technical issues here, we can prioritise them, design and implement solutions for them.

[markverstege]

Hi @dpostnikov thanks for your thoughts - I agree - if we go down a path of getting authorisation status, the the GM API makes sense although to my knowledge it still has limited vendor adoption and is working towards a final implementer's draft.

I've also previously proposed that we leverage Shared Signals and Events in DP 182 for security events as well as adding a business event layer for pushing secure notifications to ADRs. So it sounds like we're aligned in our thinking. I think that makes a lot of sense not just in this scenario but more broadly for a variety of events that may occur with the consumer's consent or at the resource layer including data correction notifications.

In the Maintenance Iteration call today we also discussed more informative error responses delivered using the oAuth error structure when authorisations fail.

A few takeaways from the discussion:

• ADRs are seeking to provide consumers with in-flow support. This doesn't imply the DH is doing something wrong, but ADRs are seeking mechanisms to assist consumers through the consent /authorisation flow when issues occur, and to also have a mechanism to follow up on issues such as drop offs through the CDR incident management process / CDR support portal
• Consensus across the group that leveraging open standards rather than customising oAuth endpoints is the way to go
• We agreed to get further data on the types of issues that ADRs are facing during authorisation and frequency of these issues. The aim is to bring this data back to the MI call for discussion before progressing solutions
• An event driven model that notifies the ADR of errors had strong support by ADRs and this generally preferred by participants on the call rather than a polling method explored previously in this thread
• Leveraging the oAuth error messaging more effectively is another solution path to consider
• Either (or perhaps both) a binding identifier supplied by the ADR or a unique error code generated by the DH would help with offline triaging of issues, but this is not the repurposing of the PAR request_uri
• There is inherently a CX consideration to how we develop more informative error messaging and better consumer guidance throughout the authorisation process
• There is a need to balance this without disclosing fine grained entitlement decisions made by the DH when they are honestly rejecting an authorisation request
• Once we have more data to discuss the issues, this CR may be best dealt with as two change requests (event/error code messaging, and CX standards for better assisting consumers).

[jimbasiq]

As discussed in the meeting today, Basiq is supportive of Option 3 but believe the scope should be extended outside of the Consent Flow from a consumer perspective and should also include the data retrieval for a consumer. The reason is, from a consumer perspective, the interaction is not complete until the data is retrieved.

We also made clear, from our perspective, the outcome we are all trying to achieve is a lower friction interaction for a consumer and an improved conversion rate. In Basiq's opinion, the best way to do this is to reduce the interaction repeated for every DH arrangement e.g. data being shared.

[CDR-API-Stream]

Attached is a report detailing the findings from DSBs work with a number of ADRs to understand the factors contributing to Authentication and Authorisation drop-offs.

This report has been presented to DSAC and identifies a number of opportunities available to ecosystem participants to change some of the behaviours.

The DSB would like to thank the ADRs for their open and collaborative participation in this forum.

Auth drop off WG.pdf

[markskript]

@CDR-API-Stream - what is the expected ETA on a response from DSAC on this so we can decide how to move forward?