`:accept_language` not respecting configured languages

[rubas]

Cldr.AcceptLanguage.best_match(accept_language, options.backend) is not respecting the configured locales.

Problem

The header Accept-Language: en-GB,en;q=0.9 is matched to en-GB instead of en, which is not accepted by Router.LocalizedHelpers from Cldr.Route.

cldr.ex

defmodule MarkoCldr do
  @moduledoc false

  use Cldr,
    default_locale: "en",
    locales: ~w(de en fr),
    gettext: MarkoGettext,
    otp_app: :marko,
    providers: [
      Cldr.Calendar,
      Cldr.DateTime,
      Cldr.Language,
      Cldr.List,
      Cldr.Message,
      Cldr.Number,
      Cldr.Route,
      Cldr.Territory,
      Cldr.Unit
    ],
    generate_docs: true,
    force_locale_download: Mix.env() == :prod
end

router.ex

  pipeline :browser do
    [..]
    plug(Cldr.Plug.SetLocale,
      apps: [:cldr, :gettext],
      from: [:assigns, :accept_language],
      gettext: MarkoGettext,
      cldr: MarkoCldr
    )
    [..]
  end

iex

iex(1)> Cldr.known_locale_names(MarkoCldr)
[:de, :en, :fr]

Argument Error no function clause for MarkoWeb.Router.LocalizedHelpers.home_live_path/3 for locale #Cldr.LanguageTag<en-GB [validated]>

[kipcole9]

Very good catch. The issues is that the code currently matches on the complete Cldr locale when it should only be matching on the :gettext_locale_name. I'm on it.

[rubas]

Not sure. I would expect Cldr.AcceptLanguage.best_match to respect the configured locales and give me back en as I have not defined en-GB as a valid locale.

That is the behaviour I'm used to and inline with lookup algo described in https://datatracker.ietf.org/doc/html/rfc4647#section-3.4

Normally when I do a matching over the AcceptLanguage header, I'm interested in a normalized output. For example to work with the vary http header and having a http cache per normalized language ("en", "de" .. ).

[kipcole9]

As of commit 9ee082d the helpers now match only on the :gettext_locale_name field of a %Cldr.LanguageTag.

[kipcole9]

Not sure. I would expect Cldr.AcceptLanguage.best_match to respect the configured locales and give me back en as I have not defined en-GB as a valid locale.

Actually it is respecting the configured locales. Here's an example (its not related to AcceptLanguage, that's just an symptom):

When en-GB is not configured as a locale

The user requests en-GB but we haven't configured that. However we do have en configured so we'll use that. But still preserve the users requested territory, even though it won't be used for most localisations.

iex> MyApp.Cldr.known_locale_names
[:de, :en, :es, :fr]
iex> {:ok, locale} = MyApp.Cldr.validate_locale("en-GB")
{:ok, #Cldr.LanguageTag<en-GB [validated]>}
iex> locale.cldr_locale_name
:en
iex> locale.territory
:GB

When en-GB is configured as a locale

The user requests en-GB and we have that configured so that becomes the value of the :cldr_locale_name field.

iex> MyApp.Cldr.known_locale_names                      
[:de, :en, :"en-GB", :es, :fr]
iex> {:ok, locale} = MyApp.Cldr.validate_locale("en-GB")
{:ok, #Cldr.LanguageTag<en-GB [validated]>}
iex> locale.cldr_locale_name                            
:"en-GB"
iex> locale.territory                                   
:GB

Summary of behaviour

The principle is that Cldr should do its very best to honour the users requested locale and preserve as much knowledge of that request as possible. But also that a locale is also only valid if it is configured in your backend configuration so there is a process of deducing what the "best match" is between what the user requested and what Cldr and the backend configuration can provide.

[rubas]

Got you! I never checked locale.cldr_locale_name. Just looked at the "headline" ...

[kipcole9]

Now you some idea of how my one-weekend-to-learn-elixir project turned into a 6 year effort :-)

Normally when I do a matching over the AcceptLanguage header, I'm interested in a normalized output. For example to work with the vary http header and having a http cache per normalized language ("en", "de" .. ).

If you are caching just by the language part, you can use locale.language as the key. Its guaranteed to be a valid ISO 3166 Alpha-2 territory code. I suspect locale.cldr_locale_name might be the better VARY key since its guaranteed to be a valid CLDR known locale.

[kipcole9]

Fixed in 9ee082d