search

unicode-org / icu4x

Solving i18n for client-side and resource-constrained environments.
https://icu4x.unicode.org
Other
1.37k stars 176 forks source link

Split date time data into smaller data keys? #257

Open zbraniecki opened 4 years ago

zbraniecki commented 4 years ago

As I'm implementing Dates in DataProvider and testing them using DateTimeFormat, I have some questions about how should we structure that.

Generally, the data in question looks like this: https://github.com/unicode-cldr/cldr-dates-modern/tree/master/main/en

It has (per locale):

For now, we need:

Display names come in different:

but they also can be for different calendar systems (I see at least "generic" and "gregorian").

As far as I understand DataProvider we have some flexibility in what do we request and what we get in response.

We could, for example, put "months/format/narrow" as a variant in DataEntry and gregory in DataKey and get just a list of month names in "format" and "narrow" for "gregory" calendar.

Or, we can just ask for "gregory" and set no variant in DataEntry and get all display names for all contexts and all widths.

@sffc - what are your thoughts on that? How should a request/response look like?

zbraniecki commented 4 years ago

Based on @sffc comments in that PR, it also seems like he'd suggest we split the dates data into multiple keys:

I assume that for things like time zone names, interval patterns and relative display names he's suggestion we also create separate keys.

sffc commented 4 years ago

First, please leave skeletons (including available_formats) aside. Those need their own discussion.

Responses to the several sub-questions:

DataProvider Flexibility

As far as I understand DataProvider we have some flexibility in what do we request and what we get in response.

Right. The intended goal of the data provider keys is not to mimic CLDR. It's to return data that is easy and efficient to consume at runtime. Mapping from CLDR format to ICU4X format happens in CldrJsonDataProvider. I want to make sure we are aligned on that goal.

Number of Data Keys

Regarding the breakdown of the data keys. I feel strongly that there should be a minimum of three distinct data keys for DateTimeFormat, which Zibi listed in https://github.com/unicode-org/icu4x/issues/257#issuecomment-695760621. Here are my reasons:

  1. A component can request the data it needs and nothing more.
  2. The same key can be shared by multiple code paths; for example, datesym can be used by both the datetimestyle path and the skeleton path.
  3. Smaller data hunks mean that more locales will be able to point to shared hunks and make data smaller on disk.

Format Widths for Display Names

I've been thinking about the format widths (long/short/narrow) for display names and whether they belong in the data requests (key/entry) or not. I'm thinking that no, they don't belong in the data request; long/short/narrow should be together in the same data key and entry, and they should be a leaf of the struct. My reasoning:

  1. The width trio (long/short/narrow) tends to be strongly correlated; if one changes, so do the others. Therefore, considering the trio as a single key does not decrease the ability to create shared data hunks.
  2. It is not uncommon to fall back between widths; for example, if narrow is not present, then we fall back to short.
  3. In order for a split data key to work as a method to reduce code size, we would need to track the requested width through much of the call stack in a way suitable for code slicing. I think this is too fine-grained to track.

Format Widths for Patterns

The situation is a bit different for pattern widths (dateStyle/timeStyle). None of the three aforementioned conditions apply here: the width patterns are not strongly correlated; they do not fall back; and we should be able to slice the data very early in the call stack.

Therefore, I think it makes sense to put format widths into either the data key or the data entry.

I would like to make this judgement after we have the code written and we can sit down and look at the concrete implications to the data bundles.

Calendar Systems

I'm arriving at the conclusion that calendar systems should be in the data key. Reasons:

  1. The data for each calendar system is unique. Not all systems are represented by 12 month names. When we need a new struct (data schema), we need a new data key.
  2. The code needed to run calendar systems needs to be written for each individual calendar, so the data keys will correspond to those modules of code.

We may at some point want to add an all-in-one calendar data key, but this is not relevant right now. We should cross that bridge later when we add calendar math to ICU4X.

zbraniecki commented 4 years ago

Thank you! This is so helpful!

sffc commented 4 years ago

Because of reasons I discussed in the new doc datetime-input.md, I now believe that we should not have different data keys for different calendar systems. We should pool the essential symbols for calendar systems into the same data key. We could consider filtering the data in the data entry or the offline build tool.

However, I do still feel that we should have different data keys for patterns, date symbols, and time symbols. We should probably go further and split the date symbols down into eras, months, day periods, and time zone names.

sffc commented 3 years ago

Concretely, I envision DateTimeFormat using the following separate, orthogonal keys, which covers all formatting except for time zone (which I want to leave to a separate discussion):

Display Names

  1. datetime/era@1 collects era display names covering all desired calendar systems
  2. datetime/cycyear@ covers cyclic year names
  3. datetime/quarter@1 covers quarter names
  4. datetime/month@1 covers month names
  5. datetime/weekday@1 covers weekday names
  6. datetime/dayperiod@1 covers the a and b day periods (am, noon, pm, and midnight)
  7. datetime/flexperiod@1 covers the B day periods (in the morning, in the afternoon, …)

Format Patterns

  1. datetime/patterns@1 covers long/medium/short date patterns, time patterns, and glue patterns
  2. datetime/skeletons@1 covers availableFormats: the mapping from skeletons to patterns

Why more keys instead of fewer keys? I listed reasons in https://github.com/unicode-org/icu4x/issues/257#issuecomment-695830848, but to reiterate:

  1. A component can request the data it needs and nothing more.
  2. The same key can be shared by multiple code paths; for example, month names can be used by both the datetimestyle path and the skeleton path.
  3. Smaller data hunks mean that more locales will be able to point to shared hunks and make data smaller on disk.
zbraniecki commented 3 years ago

I'm convinced. This looks like a great design. One additional benefit of it is that version changes will be less common and more isolated in a more chunked model.

sffc commented 3 years ago

There are still some open questions in my mind about how exactly to provision data across calendar systems, but that question is being tracked in #355.

sffc commented 3 years ago

Shane to implement this along with #355.

sffc commented 3 years ago

Blocked on #409 like #355

sffc commented 3 years ago

Also migrate the date data provider structs to have real lifetime parameters.

sffc commented 3 years ago

I'm going to punt this to Q2 because I want to wait for the work on availableFormats to stabilize. I don't see a need to introduce merge conflicts.

zbraniecki commented 3 years ago

stealing from Shane with his blessing.

sffc commented 3 years ago

Re-opening to address the following remaining issues:

  1. Skeletons should be separated from patterns
  2. Time symbols should be separated from date symbols
zbraniecki commented 3 years ago

In #791 I'm introducing conditional symbols loading.

I imagine the next step would be to:

zbraniecki commented 3 years ago

I think further splits should wait for no alloc provider because only then we'll be able to reason about actual wins.

zbraniecki commented 3 years ago

@sffc In your Oct 31 2020 comment you said:

Format Patterns

datetime/patterns@1 covers long/medium/short date patterns, time patterns, and glue patterns datetime/skeletons@1 covers availableFormats: the mapping from skeletons to patterns

Can you help me reason about holding patterns@1 vs time_patterns@1, date_patterns@1 and date_time_patterns@1?

This split would make it easy to destruct each payload into a single pattern, but maybe at some point having DateTimePattern have to ask for 4-5-6 traits and 3-4 payloads becomes costly? I'm not sure how to strike the best balance here.

zbraniecki commented 3 years ago

I'll do some of that for #519.

sffc commented 3 years ago

I'm hoping that @gregtatum's design work today puts us on a path to eventually resolve these questions about what should be in the data keys. We should have a deeper discussion on this. In the mean time, the expedient thing is probably to avoid disrupting the data provider resource key layout in major ways while we are adopting ZeroVec.

sffc commented 2 years ago

This should be one of the last things to do in 1.0 after DTF has stabilized.

We should do this before 1.0 because it impacts data file stability.

gregtatum commented 2 years ago

I think this has two parts, one part is the data keys for the ECMA-402 compatible components bag, and the other is for the ideal components bag. Blocking for 1.0 will be ensuring we have the best split for the ECMA-402 compatible components bag.

sffc commented 2 years ago

Make sure to look at the data representation of the glue pattern and make changes if necessary for future-proofing. See #1131

sffc commented 2 years ago

Action: @sffc to split off remaining work into a new 2.0 issue and close this one.

sffc commented 2 years ago

We have split symbols from patterns and date from time; this is sufficient for the first release. I would still like to explore even more-granular splitting, but there's no time in 1.0 and we should coordinate this with the Ideal Components Bag work.

Manishearth commented 7 months ago

The neo date time format stuff does this

sffc commented 1 month ago

Posting this here because it seems like as good of a place as any:

I'l looking into the minimal set of patterns required for year formatting. A year can take three forms, which we could make dynamically selectable at runtime based on the value of the year:

  1. Partial precision, such as: '00
  2. Full precision, such as: 2000
  3. With era, such as: 2000 AD

I looked into whether the patterns used for case 1 and 2 differ other than the length of the year field. I found that, at least according to the CLDR algorithm and data, the patterns for one are mostly identical to the patterns for the other with the year width swapped out. There are a few exceptions:

Legend: pattern on the left is the possibly-reduced-precision year with a full-precision year substituted. Pattern on the right is the pattern resulting from a full-precision year used during skeleton selection. DataLocale{ug}/"chinese": "r-M-d" != "r-MM-dd" DataLocale{qu-BO}/"gregorian": "d-M-y" != "dd-MM-y" DataLocale{ky}/"gregorian": "y-d-M" != "y-dd-MM" DataLocale{sat}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{bs-Cyrl}/"gregorian": "d.M.y." != "dd.MM.y." DataLocale{en-SG}/"gregorian": "d/M/y" != "dd/MM/y" DataLocale{ne}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{hr-BA}/"gregorian": "d. M. y." != "dd. MM. y." DataLocale{zu}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{en-MV}/"gregorian": "d/M/y" != "dd/MM/y" DataLocale{qu}/"gregorian": "d-M-y" != "dd-MM-y" DataLocale{ha-GH}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{en-AU}/"gregorian": "d/M/y" != "dd/MM/y" DataLocale{qu-EC}/"gregorian": "d-M-y" != "dd-MM-y" DataLocale{sah}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{ha}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{lkt}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{ne-IN}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{sat}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{ha-NE}/"gregorian": "y-M-d" != "y-MM-dd" DataLocale{ug}/"dangi": "r-M-d" != "r-MM-dd"

Spot-checking, in most of these cases, the pattern is inherited from the root locale, which makes me question the quality of the data.

If I were to store the patterns separately, I could use another flag in the data struct. We currently have a single byte reserved for flags in the packed data structure, and 4 bits are used, so this would mean using one more bit. However, we could keep the bit unset if the locale results in equivalent data as observed above.

https://github.com/unicode-org/icu4x/blob/main/components/datetime/src/provider/neo.rs#L560

This could be done as a follow-up so long as the bit remains available.