[Help] with Cldr.DateTime.Formatter is not available

[coladarci]

When doing this locally: Cldr.DateTime.to_string(~N[2020-05-28 20:48:08], format: "EE, MMM d, y") All works as I'd expect; I get back {:ok, "Thu, May 28, 2020"}

But when my app is compiled into our staging environment, the same code results in:

function Utils.Cldr.DateTime.Formatter.format/4 is undefined (module Utils.Cldr.DateTime.Formatter is not available)

I'm having a hard time understanding what could lead to this working locally and in my tests but not when compiled.

This is how I am useing Cldr

defmodule Utils.Cldr do
  @moduledoc false

  use Cldr,
    default_locale: "en",
    locales: ["en"],
    providers: [Cldr.Calendar, Cldr.DateTime, Cldr.Number],
    generate_docs: false

Any theories? Thanks a bunch as always...

[kipcole9]

Yes, that’s a pain. May I ask a couple of questions?

1. Version of ex_cldr you’re using? In the last couple of versions I made a small change to backend module compilation that might help. I’m not optimistic, but it’s possible. The latest is ex_cldr 2.15.0

2. What Mix environment are you using in staging. Is it still :dev Or something else?

3. How are you starting the app in staging? A release, or with mix?

One thing you can try as we try to diagnose this is to add the following at the top of the module where you get the exception.

# Force loading of the backend
require Utils.Cldr.DateTime.Formatter

[coladarci]

Thanks -

1) Mix lock: "ex_cldr": {:hex, :ex_cldr, "2.15.0", 2) stage is prod env 3) We use elixir releases

I'll push up that change but it'll take a few for me to get it into that env to test it out. I haven't seen something like this happen before; normally this bombs in travis tests when it's an issue that only affects compile-time.

[kipcole9]

Thanks much, that helps - I can work on this in a few hours (its late Saturday night here). I haven’t done any testing with releases so its about time I did that. Although I don’t have any other issues reported across the various cldr libs there is clearly something going on.

One last thought, can you confirm that Elixir.Utils.Cldr.DateTime.Formatter.beam is in the _build directory? Be super surprising if it isn’t but better to check.

[coladarci]

Thanks! The require didn't help, and to be fair, I am using CLDR throughout w/out problem; it's just something happening w/ my new code. It feels like I'm doing something stupid; but for it to work everywhere but env prod feels like a bigger issue..

So I went into the docker file produced and you are correct; it's not there!!

/app # ls lib/utils-0.1.0/ebin/
Elixir.Utils.Cldr.Number.Formatter.Decimal.beam 
Elixir.Utils.Cldr.Rbnf.NumberSystem.beam        
Elixir.Utils.Cldr.Number.Ordinal.beam            
Elixir.Utils.Cldr.Rbnf.Ordinal.beam             
Elixir.Utils.Cldr.Currency.beam                  
Elixir.Utils.Cldr.Number.PluralRule.Range.beam   
Elixir.Utils.Cldr.Rbnf.Spellout.beam            
Elixir.Utils.Cldr.Gettext.Plural.beam          
Elixir.Utils.Cldr.Number.Symbol.beam            
Elixir.Utils.Cldr.beam                          
Elixir.Utils.Cldr.Locale.beam                 
Elixir.Utils.Cldr.Number.System.beam            
Elixir.Utils.Cldr.Number.Cardinal.beam          
Elixir.Utils.Cldr.Number.Transliterate.beam     
Elixir.Utils.Cldr.Number.Format.beam           
Elixir.Utils.Cldr.Number.beam                    

[kipcole9]

Ok, well that’s a useable fact. Perhaps try MIX_ENV=prod mix compile —force on your development machine and see if the beam file is there? That’s all that comes to mind for the moment. I’ll grab a few hours sleep and then see if I can replicate and fix.

[kipcole9]

I have created an empty app and compiled it with MIX_ENV=prod mix compile. The resulting BEAM files are as expected.

Backend

defmodule MyApp.Cldr do
  use Cldr,
    locales: ["en", "fr", "af", "ja", "de"],
    providers: [Cldr.Number, Cldr.Calendar, Cldr.DateTime],
    precompile_number_formats: ["#,##0"]
end

BEAM files

Elixir.MyApp.Cldr.Calendar.beam     
Elixir.MyApp.Cldr.Currency.beam     
Elixir.MyApp.Cldr.Date.beam     
Elixir.MyApp.Cldr.DateTime.Format.beam  
Elixir.MyApp.Cldr.DateTime.Formatter.beam
Elixir.MyApp.Cldr.DateTime.Relative.beam
Elixir.MyApp.Cldr.DateTime.beam     
Elixir.MyApp.Cldr.Locale.beam       
Elixir.MyApp.Cldr.Number.Cardinal.beam  
Elixir.MyApp.Cldr.Number.Format.beam    
Elixir.MyApp.Cldr.Number.Formatter.Decimal.beam
Elixir.MyApp.Cldr.Number.Ordinal.beam
Elixir.MyApp.Cldr.Number.PluralRule.Range.beam
Elixir.MyApp.Cldr.Number.Symbol.beam
Elixir.MyApp.Cldr.Number.System.beam
Elixir.MyApp.Cldr.Number.Transliterate.beam
Elixir.MyApp.Cldr.Number.beam
Elixir.MyApp.Cldr.Rbnf.NumberSystem.beam
Elixir.MyApp.Cldr.Rbnf.Ordinal.beam
Elixir.MyApp.Cldr.Rbnf.Spellout.beam
Elixir.MyApp.Cldr.Time.beam
Elixir.MyApp.Cldr.beam

In comparing with what you are seeing I notice you have none of the following:

Elixir.MyApp.Cldr.Date.beam     
Elixir.MyApp.Cldr.DateTime.Format.beam  
Elixir.MyApp.Cldr.DateTime.Formatter.beam
Elixir.MyApp.Cldr.DateTime.Relative.beam
Elixir.MyApp.Cldr.DateTime.beam
Elixir.MyApp.Cldr.Time.beam

Which is a concern because it suggests that the DateTime backend provider isn't being compiled. Which it most definitely is on my system.

Let me know if you get a chance to MIX_ENV=prod mix compile —force on your dev machine and confirm you have the expected BEAM files per the list above at the start of this message?

[coladarci]

Yes, they work when I compile w/ mix_env prod.. but they don't end up in my docker images. This wreaks of a bum cache somewhere, but I can't pinpoint it.

A few notes -

1) I'm in an umbrella APP 2) This is my Dockerfile:

####################
# Stage 1: builder #
####################

FROM hexpm/elixir:1.10.2-erlang-22.2.8-alpine-3.11.3 as builder

RUN apk --no-cache --update upgrade && \
  apk --no-cache add openssh alpine-sdk coreutils curl bash

RUN mix local.hex --force && \
  mix local.rebar --force

WORKDIR /app

COPY apps /app/apps
COPY config /app/config
COPY mix.exs /app/mix.exs
COPY mix.lock /app/mix.lock

ENV MIX_ENV prod

RUN mix do deps.get, deps.compile, release

####################
# Stage 2: runtime #
####################

FROM alpine:3.11.6

RUN apk --no-cache --update upgrade && \
  apk --no-cache add openssl ncurses alpine-sdk coreutils curl bash

WORKDIR /app

COPY --from=builder /app/_build/prod/rel/app_name/ /app/

CMD ["bin/app_name", "start"]

Anything else I can test?

[kipcole9]

I don't think its related to an umbrella app; I created a new umbrella and compiled and the files are there.

And I'm a docker neophyte so it'll take a little while for me to replicate the environment. I'm on it ....

[kipcole9]

Confirmed that a basic release builds correctly (well at least all the BEAM files are there). On to docker-land.

[kipcole9]

I am perplexed and not sure how to further diagnose this.

I created a minimal dockerfile based on your Dockerfile except I copied the whole app source tree:

####################
# Stage 1: builder #
####################

FROM hexpm/elixir:1.10.2-erlang-22.2.8-alpine-3.11.3 as builder

RUN apk --no-cache --update upgrade && \
  apk --no-cache add openssh alpine-sdk coreutils curl bash

RUN mix local.hex --force && \
  mix local.rebar --force

WORKDIR /app

# Just copy the whole app source tree
COPY . /app/

ENV MIX_ENV prod

RUN mix do deps.get, deps.compile, release

####################
# Stage 2: runtime #
####################

FROM alpine:3.11.6

RUN apk --no-cache --update upgrade && \
  apk --no-cache add openssl ncurses alpine-sdk coreutils curl bash

WORKDIR /app

COPY --from=builder /app/_build/prod/rel/what/ /app/

# CMD ["/bin/bash"]

Then build and run the container interactively and checked the contents and everything is there. Admittedly not an umbrella.

kip@Kips-iMac-Pro what % docker run --interactive --tty what
/app # ls lib/what-0.1.0/ebin
Elixir.What.Calendar.beam        Elixir.What.Number.Cardinal.beam       Elixir.What.Number.beam
Elixir.What.Currency.beam        Elixir.What.Number.Format.beam     Elixir.What.Rbnf.NumberSystem.beam
Elixir.What.Date.beam            Elixir.What.Number.Formatter.Decimal.beam  Elixir.What.Rbnf.Ordinal.beam
Elixir.What.DateTime.Format.beam     Elixir.What.Number.Ordinal.beam        Elixir.What.Rbnf.Spellout.beam
Elixir.What.DateTime.Formatter.beam  Elixir.What.Number.PluralRule.Range.beam   Elixir.What.Time.beam
Elixir.What.DateTime.Relative.beam   Elixir.What.Number.Symbol.beam     Elixir.What.beam
Elixir.What.DateTime.beam        Elixir.What.Number.System.beam     what.app
Elixir.What.Locale.beam          Elixir.What.Number.Transliterate.beam

[coladarci]

OK, I got to the bottom of this, and as we both (probably?) suspected; this was me doing something incorrect.

I don't understand why this wouldn't have been caught by my tests, which is likely the only remaining piece to care about. So here are the steps to reproduce (and they do involve an umbrella):

1) Create an umbrella w/ two apps. App1, App2 2) In App1 and App2, add :ex_cldr as a dependency 3) In App1 add ex_cldr_dates_times but don't add it to App2 4) Define your CLDR in App2 (i.e create an App2.Cldr) and include [Cldr.Number, Cldr.Calendar, Cldr.DateTime] as providers. **This is the moment things went wrong; this shouldn't work because app2 doesn't include ex_cldr_dates_times as a dep. App1 shouldn't need any cldr deps. 5) Write a test inside App2 to show that App2.Cldr.DateTime.to_string(~N[2020-05-28 20:48:08], format: "EE, MMM d, y") works; it shouldn't because when you run the tests from directly inside App2 (i.e you cd into apps/app2 and run mix test) it should fail because you've listed a provider that the app doesn't technically know about.

There is a very good chance this falls outside the hands of your packages; umbrella dependencies are a disaster and prone to all sort of problems, this being an example. But after you've invested so much time into helping me, I wanted to at least get you some steps to reproduce.

Please feel encouraged to close the issue when you have felt you've done all that is reasonable on your end.. thanks again for everything!

[kipcole9]

Thanks very much for the explanation and for getting to the bottom of this. It still sounds like I should be able to provide some kind of "early warning" though. At compile time when building a backend module there is a log message if a provider can't be found. So for whatever reason, the dependency seems to "leak" from one umbrella app to another. Is that the intention in an umbrella?

Any thoughts on what I could do to detect this situation?

[kipcole9]

BTW, some cool date/time interval formatting coming up in the next version. So given two dates/times you'll be able to format them as

• Jan 10–12, 2020 if the difference in dates is only days
• Jan 10 – Feb 12, 2020 if the difference is months and
• Jan 10, 2020 – Feb 12, 2021 if the difference is years

Similar for times and date/times with some short, medium and long formats. </ end of gratuitous ad>

[coladarci]

That's great! We do event software so your money lib + cldr is our entire world. date ranges are a big part of what we do so I'll be sure to try this out. Does it work w/ times? that'd be huge for us. Jan 10th 9am - 2pm versus Jan 10th 9am - Jan 11th 5pm etc.

As far as deps leaking between apps, that's exactly the problem. They shouldn't but since there is a single mix.lock file not to mention a single config file, the idea that each app has it's own dependencies is a bit of a stretch. Where it normally bites us if you screw up your dependencies like this:

App1: {:greg, "1.0"} App2: [no deps]

App2.Greg.go!()

This should fail because App2 doesn't include the dep. The problem is that if you run your tests from the root of the umbrella they typically WON'T fail. But if you cd apps/app2 and run mix test they typically WILL fail.

[kipcole9]

Yes, definitely for times and date times too. Of course localised. It will auto detect the precision difference and apply the right format. I finished the heavy lifting of ingesting the data from CLDR and generating the format functions. Now just the public api so should be done by end of the week.

If you’re ok I think I’ll close this issue since I don’t think I have a way to remediate the issue. I’ll put a warning in the docs since I think that’s about all I can do.

BTW, events (as in calendar events) with recurrence and mapping to schema.org and activity streams is probably my next project. You’re welcome to throw requirements at me about that.

Sent from my iPhone

On 31 May 2020, at 22:44, Greg Coladarci notifications@github.com wrote:

 That's great! We do event software so your money lib + cldr is our entire world. date ranges are a big part of what we do so I'll be sure to try this out. Does it work w/ times? that'd be huge for us. Jan 10th 9am - 2pm versus Jan 10th 9am - Jan 11th 5pm etc.

As far as deps leaking between apps, that's exactly the problem. They shouldn't but since there is a single mix.lock file not to mention a single config file, the idea that each app has it's own dependencies is a bit of a stretch. Where it normally bites us if you screw up your dependencies like this:

App1: {:greg, "1.0"} App2: [no deps]

App2.Greg.go!()

This should fail because App2 doesn't include the dep. The problem is that if you run your tests from the root of the umbrella they typically WON'T fail. But if you cd apps/app2 and run mix test they typically WILL fail.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[coladarci]

Take a look at what we've done here - curious your thoughts on it! https://github.com/peek-travel/cocktail