Optionality of critical fields is facilitating data quality issues across Data Holder implementations

[ShaneDoolanAdatree]

Description

The below is taken from the Payload Conventions section of the Standards: https://consumerdatastandardsaustralia.github.io/standards/#payload-conventions

Note that optional fields are not considered optionally implementable by a Data Holder. For instance, if a Data Holder holds data in digital form for a Customer that is represented in a payload then it is expected that this data will be shared when authorised by the Customer. For payloads unrelated to Customers, such as product reference data, there is more discretion for the Data Holder but other drivers, such as complementary regulation or the requirement to align to other channels, should be taken into consideration.

Unfortunately for ADRs and consumers, despite this explicit statement, some major Data Holders are taking optional fields as literally optional. The worst and most worrying example of this is within the BankingAccountDetail payload where for loan account types they are not including obviously required data such as lendingRate and the lendingRates array of BankingProductLendingRate type.

This behaviour is not only in breach of the above statement from the Standards, but we believe it is also a breach of Schedule 3 Rule 1.4 of the Rules. But, making the fields optional in the Standards, despite the above Standards statement on optionality, gives Data Holders and their implementing teams some room to debate the issue and/or make mistakes.

Area Affected

https://consumerdatastandardsaustralia.github.io/standards/#tocSbankingaccountdetailv2

Change Proposed

As an urgent change I would request the below fields that are part of BankingAccountDetailV2 be made conditional with a statement to the effect of "if a loan has a rate then include the rate/rates" (blatantly obvious as that may seem but... here we are).

• lendingRate
• lendingRates

As a more forward looking change I would request that we review the use of the word optional within the Standards entirely as the explicit statement about optionality by the Standards seems to be easy for Data Holders to ignore and hard for ADRs to enforce.

[ZacHortaraSherlok]

Sherlok supports this change.

We’re currently experiencing significant impact on our ability to deliver the value of CDS/CDR to our customers, since we’re frequently seeing no information provided under the lendingRates key.

In addition, we’re often experiencing data being omitted when it has been provided. loanPurpose is often null, and some providers have rate incorrectly formatted ("rate": "5.33",), which according to the docs for RateString would imply that the interest on the loan is 533%.

For us to deliver a usable product to our customers, we’ve had to work around these issues by implementing means for users to fill in the gaps that CDS/CDR was meant to.

[markskript]

Skript also supports this change.

Customers expect that if the data is available to them via traditional Internet Banking channels then that same data will be available through the CDR - despite the 'technical' classification of certain data being optional.

[perlboy]

While I can easily see the points of view of ADRs in this thread there is a few nuances that I think need explicit clarity:

1. Many internet banking experiences do not disclose current interest rates (lendingRate) within IB. If anything these are only contained in statements (ie. statically generated PDFs), it remains to be seen if this is considered "digital". In business accounts land this is even more so, in fact when looking at an ANZ business account it shows N/A and a note of "# For interest bearing accounts refer to account terms and conditions to determine the method of calculation."
2. Most internet banking experiences do not disclose complex rate (ie. lendingRates) structures, particularly lending structures and in some cases show deposit rate structures for lending accounts should they have a positive balance (which would be unusual). This means that them being missing is still parity with digital experiences.
3. This problem exists for both depositRate(s) and lendingRate(s) which means nearly all account types are in scope. This presents a scale issue.
4. Many core banking vendors (who are often also the IB vendors) are highly secretive about how to find these values in their data stores. This poses a significant challenge with implementing CDR without further buying into proprietary lock-in.
5. If there was a mechanism that directly tied product identifiers to published PRD information that seems like a good baseline. While negotiation definitely happens there remains a significant number of banking agreements which are "by the book" and ultimately if a rate is negotiated favourably with the consumer it therefore incentivises the Holder to explicitly disclose this in the available fields and then "fall through" to a product id correlated with PRD data.
6. The interest rate fields only show the current state. While this is "ok" for quite basic comparison use cases if organisations are going to reasonably "pitch" interest rate comparisons they really should be evaluating a sample set of historical data, for instance, 12 months of transactions analysed for interest charged against balances. It's worthwhile noting this because there are a variety of responsible disclosure obligations for finance professionals that would expect this behaviour and software solutions don't obviate this requirement.

From an implementation perspective (1) is usually relatively easy however for (2) complex lending rate structures in a number of popular core banking systems is at least an n based problem but often an n2 or even n log n based one. In the absence of a fronting data store, which isn't typically required for small scale delivery of IB, there is a consequential NFR problem.

In essence I suspect that ADRs are suffering in favour of organisations achieving NFRs because NFRs is what is reported to the Regulator and published to the public. Biza has previously provided feedback, in September 2021, outlining the challenges of retrieving data from source systems. This feedback also included the following statement "Biza.io also notes that the recent Frollo report on Open Banking Performance appears to include low latency responses from organisations we have reason to suspect have underlying data quality issues. Put another way, it is easy to serve a blank answer quickly but more important to serve the right answer in a reasonable time". This thread seems to be validating this statement.

[ShaneDoolanAdatree]

Appreciate your insight @perlboy. In terms of the data being digital or not and aligned with Internet Banking (IB) our interpretation of the Rules (which aligns with the Regulator view AFAIK) is that if the data exists in 'digital form' it must be disclosed and 'digital form' does not solely mean IB. It would be a harmful loophole, to ADRs, consumers and competition in general, if CDR CX guidance meant data holders could avoid data disclosure obligations by taking something off their website. We don't believe the Rules allow this.

This Zendesk article is being citing by data holders we've raised this with as grounds to not disclose data when our understanding is that the Standards and CDR 'How-to' articles do not supersede the Rules. The article linked gives guidance to implementers on aligning CDR CX to existing IB CX to make consumers comfortable. If enabling a keystone CDR banking industry use case comes with loosening of NFRs to allow data holders to comply with the Rules I would bet any sensible ADR would support that.

[perlboy]

@ShaneDoolanAdatree appreciate yours too. We're starting to split hairs here especially if we throw in the "that's not guidance given by other Regulatory representatives" bit but what I'd note is that the digital form that many of these systems store interest rates in aren't the presentation form required for the Data Standards. I'm certainly not suggesting this "makes it ok" but if technologists are forced down a Rules line they will start questioning what "digital form" something is stored in. No-one will win with this approach.

Certainly on our side we're committed to making as much of the data available as we possibly can but we are arbitrarily limited by the NFRs in multiple implementations. This is heavily driven by archaic core banking systems coupled with the sometimes prohibitive cost of implementing a high speed source.

[nils-work]

Hi @ShaneDoolanAdatree The DSB is proposing this issue be addressed as part of Decision Proposal 306. Any feedback would be welcome.