Permit CLDR rather than ISO 4217 for CurrencyDigits?

[littledan]

CurrencyDigits is an abstract operation which normatively references ISO 4217 to get the default number of digits to display for a currency. That data can be found on ISO 4217's website in this XML file.

CLDR has its own notion of how many digits should be used for currency display by default. Here is the data for CLDR 30.

These charts actually differ in a number of places. For example, afghani (AFN) is listed as having 2 decimal digits in ISO 4217, but 0 in CLDR 30. According to Wikipedia, the new afghani (AFN) has never had subdivisions issued, though it is nominally divided into 100 smaller units.

I'm not sure what accounts for this difference, whether ISO 4217 is going for more legalistic definitions and CLDR is going for what's linguistically relevant, or whether one has fresher data than the other, or something else. I'd be interested if anyone has more context here.

V8 has been using CLDR data for CurrencyDigits, whereas SpiderMonkey (and now JSC) has been using ISO 4217 data. I wrote patches to switch SpiderMonkey and JSC to switch to CLDR, not realizing, until it was caught in SpiderMonkey's test automation (thanks @syg!), that this was actually a semantic change, and introduced a spec violation caught by a test262 test.

Apparently, either source is web-compatible, but I don't know which one is better. Does anyone have more background here?

cc @jungshik

[NorbertLindenberg]

The ISO 4217 definition of the decimal digits for currencies is indeed legalistic and sometimes detached from reality. I contacted the maintenance agency in 2013 to ask about the Indonesian rupiah, which was listed as having two decimal digits. In reality, the smallest coin in Indonesia is 50 rupiah, which is worth about half a US cent, and a 1000:1 redenomination of the rupiah has been considered for years. The maintenance agency contacted the Central Bank of Indonesia, which replied that the minor unit was still used in accounting and therefore should remain. And so it remained.

[littledan]

@NorbertLindenberg Interesting details here. This sort of logic seems reasonable for ISO 4217 itself, to be a place where central banks may give details of official denominations. At the same time, I can see why CLDR would diverge.

Which do you think is more salient for web platform users who want to display numbers appropriately in a particular currency? Would you object to switching to an unspecified data source, with a note which makes a CLDR optional cross-reference suggestion, as many other parts of the ECMA 402 spec are?

[NorbertLindenberg]

In general, the standard should prescribe the behavior of ECMAScript API as closely as possible. The Internationalization API got some dispensation in that regard for the reasons outlined in section 4.3, but since there is a standard source for the decimal digits for currencies, even if it's not ideal, switching to an unspecified data source would be heading in the wrong direction. Switching to CLDR as a normative data source, on the other hand, might be acceptable, although somebody with more experience in dealing with ISO should confirm whether it indeed is. It might help if someone from the CLDR team could describe or point to their rules for determining currency information.

I think applications that use the API for business or accounting purposes should specify the decimal digits themselves anyway. Relying on the API implementation to provide this information is more acceptable for applications that provide prices in the user's currency as estimates only, e.g., when displaying the rates for hotels in foreign countries that charge in their local currency.

[littledan]

@allenwb @ecmageneva Do you have any understanding about what it would mean for ISO to normatively reference CLDR from a TC39 spec? Or who to ask?

@bterlson What would you think of normatively referencing to CLDR here?

@NorbertLindenberg I can see the high-level goal of making things as tied down as possible, but the ECMA 402 specifcation defers to unspecified information sources for all of its core functionality. It only includes a couple normative references at the edges which go beyond Unicode and BCP 47 (here, the timezone database). Given that practical implementations will be tying into an existing internationalization library with its stance on these issues (whether it's ICU or Windows Unicode libraries or something else), it seems reasonable to me to say, "you can use your platform's best understanding of what's linguistically appropriate here", as is done elsewhere; this would be a step up from normatively referencing something that we know to be less accurate compared to data that's already in the platform.

[caridy]

@bterlson What would you think of normatively referencing to CLDR here?

This is the point of contention here. We will have to wait for Microsoft's formal response about this, and hopefully @bterlson can help.

Somebody with more experience in dealing with ISO should confirm whether it indeed is. It might help if someone from the CLDR team could describe or point to their rules for determining currency information.

Absolutely, let's loop me more folks that might help to clarify this. /cc @srl295 @JCEmmons

[JCEmmons]

I think @NorbertLindenberg was absolutely correct when describing ISO 4217 definition as somewhat legalistic and often outdated. In CLDR land, we try to go for "every day usage" ( i.e. what would you want to see if you were buying something online ). Often I will use things like Wikipedia as a reference. I will look to see what the lowest value coin in circulation is. I have had numerous times over the years where I have tried to get ISO to change. Sometimes the only place where you would see fractional digits would be in banking transactions. Now in CLDR we actually have added "cashDigits" and "cashRounding" to help handle some of the edge case currencies like TWD, where street usage is definitely 0 decimal places, but 2 decimal places in the banks.

Kind of like some of the oddities you see here in the U.S. When you go to the gas station, you will almost always see the price as $x.xx9 per gallon. Why the extra 9/10 cent I will never know.....

[mattjohnsonpint]

Why the extra 9/10 cent I will never know.....

I've always thought that was about taxes, but the Internet says it's a marketing ploy.

[littledan]

OK, so if the CLDR data is considered better than the ISO 4217 data, would it be acceptable to change the specification to allow using it? I'm having trouble making sense of @NorbertLindenberg 's comments--I could make a very well-specified but very wrong database for just about anything in Intl, but most things are left to be provided by the implementation to permit greater accuracy.

[littledan]

FWIW someone actually filed a bug against Chrome for violating the spec and using CLDR data. However, it appears that the real issue here was its differing output from Safari and Firefox.

[annevk]

FWIW, the HTML Standard also uses CLDR for one thing. Seems much less problematic than ICU at least.

[littledan]

Would anyone be interested in compiling a case of differences between CLDR and ISO 4217, as @apaprocki suggested on Twitter ?

[srl295]

@annevk

FWIW, the HTML Standard also uses CLDR for one thing. Seems much less problematic than ICU at least.

What does this mean?

@littledan I responded on Twitter… Maybe someone would contribute a list that could be a part of CLDR noting deviation from 4217, or maybe a tool to perform the diff?

[apaprocki]

@NorbertLindenberg The Bloomberg Terminal is in agreement with CLDR over the Indonesian rupiah; it is quoted with 0 decimal places in the system

@littledan Interestingly, though, the Afghani is quoted in (and uses) 4 decimal places. In this case, it appears an Afghan bank rate provider does only quote out to 2 decimal places, but other FX venues go out to 4 decimal places, so the composite spot uses 4. I would need to ask around to dig deeper on this particular currency. At first glance, though, I'd venture the CLDR value of 0 has to be wrong.

[apaprocki]

@littledan For the AFN case, it was added to the supplementalData.xml file way back in 2008 with no real description of why all the values in the diff should be 0 or what methodology was used to fact-check / reality-check it, and it hasn't changed since.

[jungshik]

@apaprocki would you file a ticket against CLDR about AFN ?

[jungshik]

https://bugs.chromium.org/p/v8/issues/detail?id=7472 is a v8 bug for failing to pass test262/intl402/NumberFormat/currency-digits .

[jungshik]

See also https://unicode.org/cldr/trac/ticket/11586 .

[littledan]

We discussed this issue on a tangent in the November 2018 Intl meeting. At the time, there seemed to be interest in moving towards a specification which permits tailoring. Let's investigate details in a PR.

[littledan]

@spectranaut Would you be interested in taking this on as a follow-up to the case tailoring issue?

[felixfbecker]

I just stumbled across this going down the rabbit hole of why some currencies are formatted differently in Chrome compared to other browsers. One part that is especially confusing here is that the documentation on MDN explicitly states that the default number of fractional digits is taken from ISO 4217, when apparently that is actually not mandated by the spec and not the case in Chrome (where CLDR is used).

[srl295]

CLDR carefully takes in iso 4217 updates (I've done several of them recently) but as with anything can adjust for reality "on the ground". Something I'm not sure is captured in this issue is that CLDR has separate info for cash vs non cash transactions. Perhaps the non cash might still differ from 4217

[felixfbecker]

Interesting thought. The non cash values still differ from ISO 4217 – for example IQD (Iraqi Dinar) is listed as 3 minor units in ISO 4217 [1] but listed as 0 minor units in CLDR [2], with extra no value listed in the cash units column.

[1] https://www.six-group.com/dam/download/financial-information/data-center/iso-currrency/lists/list-one.xml search for "IQD" [2] https://github.com/unicode-org/cldr/blob/release-43-1/common/supplemental/supplementalData.xml#L36 or in the release charts supplemental/detailed_territory_currency_information.html#IQD

[sffc]

CC @ben-allen to put together a PR for changing to CLDR as the normative reference so that we can gather feedback.