mix deps.compile can't find reference to a struct from an already compiled dep

[Sinc63]

Elixir and Erlang/OTP versions

Erlang/OTP 25 [erts-13.2.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]

Elixir 1.15.5 (compiled with Erlang/OTP 25)

Operating system

Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Wed May 11 19:58:59 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Current behavior

I'm working on updating our environment from Elixir 1.13 to Elixir 1.15. As such I'm in a process of trying to get all our dependencies to compile, which is a stepwise process.

I first encountered an error compiling phoenix_markdown:

==> phoenix_markdown
Compiling 2 files (.ex)
error: Earmark.Options.__struct__/0 is undefined, cannot expand struct Earmark.Options. Make sure the struct name is correct. If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code
  lib/phoenix_markdown/engine.ex:25: PhoenixMarkdown.Engine.compile/2

== Compilation error in file lib/phoenix_markdown/engine.ex ==
** (CompileError) lib/phoenix_markdown/engine.ex: cannot compile module PhoenixMarkdown.Engine (errors have been logged)
    (stdlib 4.3.1.2) lists.erl:1462: :lists.mapfoldl_1/3
could not compile dependency :phoenix_markdown, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile phoenix_markdown --force", update it with "mix deps.update phoenix_markdown" or clean it with "mix deps.clean phoenix_markdown"

earmark had been compiled in my first pass, this was my second.

Now I have this pair of results, which shows to me that for some reason while compiling a dependency that uses a struct from another dependency, the compiler is somehow unable to find and refer to the struct from the previously compiled dependency.

[root]# mix deps.compile mogrify link_preview
==> link_preview
Compiling 11 files (.ex)
error: Mogrify.Image.__struct__/0 is undefined, cannot expand struct Mogrify.Image. Make sure the struct name is correct. If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code
  lib/link_preview/parsers/html.ex:139: LinkPreview.Parsers.Html.filter_small_image/2

== Compilation error in file lib/link_preview/parsers/html.ex ==
** (CompileError) lib/link_preview/parsers/html.ex: cannot compile module LinkPreview.Parsers.Html (errors have been logged)
    (stdlib 4.3.1.2) lists.erl:1462: :lists.mapfoldl_1/3
    (stdlib 4.3.1.2) lists.erl:1463: :lists.mapfoldl_1/3
could not compile dependency :link_preview, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile link_preview --force", update it with "mix deps.update link_preview" or clean it with "mix deps.clean link_preview"
[root]# mix deps.compile mogrify link_preview --force
==> mogrify
Compiling 9 files (.ex)
Generated mogrify app
==> link_preview
Compiling 11 files (.ex)
warning: Floki.parse/1 is deprecated. Use `parse_document/1` or `parse_fragment/1` instead.
Invalid call found at 3 locations:
  lib/link_preview/parsers/opengraph.ex:20: LinkPreview.Parsers.Opengraph.title/2
  lib/link_preview/parsers/opengraph.ex:40: LinkPreview.Parsers.Opengraph.description/2
  lib/link_preview/parsers/opengraph.ex:67: LinkPreview.Parsers.Opengraph.images/2

warning: Tempfile.random/1 is undefined (module Tempfile is not available or is yet to be defined)
  lib/link_preview/parsers/html.ex:137: LinkPreview.Parsers.Html.filter_small_image/2

warning: Floki.parse/1 is deprecated. Use `parse_document/1` or `parse_fragment/1` instead.
Invalid call found at 3 locations:
  lib/link_preview/parsers/html.ex:20: LinkPreview.Parsers.Html.title/2
  lib/link_preview/parsers/html.ex:69: LinkPreview.Parsers.Html.images/2
  lib/link_preview/parsers/html.ex:91: LinkPreview.Parsers.Html.search_h/2

Generated link_preview app

The struct from mogrify should be available to link_preview even though mogrify was compiled in a previous pass of deps.compile.

Expected behavior

Two related dependencies should not need to be compiled in the same pass of deps.compile for the second app to refer to a struct from the first app. As long as the app containing the struct is already compiled it's structs should be available to other apps when compiled separately.

[Sinc63]

Update. It's not just a struct:

==> phoenix_markdown
Compiling 2 files (.ex)
error: Earmark.Options.__struct__/0 is undefined, cannot expand struct Earmark.Options. Make sure the struct name is correct. If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code
  lib/phoenix_markdown/engine.ex:25: PhoenixMarkdown.Engine.compile/2

== Compilation error in file lib/phoenix_markdown/engine.ex ==
** (CompileError) lib/phoenix_markdown/engine.ex: cannot compile module PhoenixMarkdown.Engine (errors have been logged)
    (stdlib 4.3.1.2) lists.erl:1462: :lists.mapfoldl_1/3
could not compile dependency :phoenix_markdown, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile phoenix_markdown --force", update it with "mix deps.update phoenix_markdown" or clean it with "mix deps.clean phoenix_markdown"
[root]# mix deps.compile earmark phoenix_markdown --force
==> earmark
Compiling 9 files (.ex)
Generated earmark app
==> phoenix_markdown
Compiling 2 files (.ex)
warning: HtmlEntities.decode/1 is undefined (module HtmlEntities is not available or is yet to be defined)
  lib/phoenix_markdown/engine.ex:58: PhoenixMarkdown.Engine.do_restore_smart_tags/2

warning: Regex.regex?/1 is deprecated. Use Kernel.is_struct(term, Regex) or pattern match on %Regex{} instead
Invalid call found at 2 locations:
  lib/phoenix_markdown/engine.ex:82: PhoenixMarkdown.Engine.only?/3
  lib/phoenix_markdown/engine.ex:106: PhoenixMarkdown.Engine.except?/3

Generated phoenix_markdown app
[root]# mix deps.compile earmark html_entities phoenix_markdown --force
==> html_entities
Compiling 2 files (.ex)
Generated html_entities app
==> earmark
Compiling 9 files (.ex)
Generated earmark app
==> phoenix_markdown
Compiling 2 files (.ex)
warning: Regex.regex?/1 is deprecated. Use Kernel.is_struct(term, Regex) or pattern match on %Regex{} instead
Invalid call found at 2 locations:
  (phoenix_markdown 1.0.3) lib/phoenix_markdown/engine.ex:82: PhoenixMarkdown.Engine.only?/3
  (phoenix_markdown 1.0.3) lib/phoenix_markdown/engine.ex:106: PhoenixMarkdown.Engine.except?/3

Generated phoenix_markdown app

Note that in the attempt to compile earmark and markdown I got an error that HtmlEntities.decode is undefined, but when I compiled that dependency at the same time the problem with that also went away. So it seems that compiling dependencies doesn't accurately load the symbols from dependencies compiled in a separate pass.

That sounds pretty scary.

[josevalim]

mix deps.compile does not resolve dependencies. So if you ask it to compile “foo”, you are responsible for precompiling its dependencies. I will improve the error message here but I wanted to make it clear that it is a low level operation on purpose.

[Sinc63]

mix deps.compile does not resolve dependencies. So if you ask it to compile “foo”, you are responsible for precompiling its dependencies. I will improve the error message here but I wanted to make it clear that it is a low level operation on purpose.

Are you saying that it is expected that if bar depends on foo, I can't do mix deps.compile foo; mix deps.compile bar, I have to domix deps.compile foo barall in one command? What if I have foo already compiled and get a new version of bar? Can I not simply domix deps.compile` and let the compilation of the new bar make use of the existing compilation of foo? If that's the case why is a partial deps.compile ever done, and ever successful?

I just did another test. I modified the link_preview code and then invoked a deps.compile for just that module. It worked fine, now that I got through getting everything to compile once. So it seems that my problem only occurs while doing the first round of compilation. So that suggests to me that there is something in the process that is only done once all the dependencies have been compiled once. I don't have any knowledge of compilation so I don't know what that would be, but it seems wrong to me.

[josevalim]

You can do it in parts but you need to compile foo before bar if bar depends on foo.

[Sinc63]

You can do it in parts but you need to compile foo before bar if bar depends on foo.

Look back in the first problem description, but the second part showing mogrify and link_preview. I compiled it specifying both, and the compiler skipped mogrify because it was already compiled, then hit the error about not recognizing the struct from Mogrify. When I told it to force compile both I didn't get that error. So it wasn't sufficient to have mogrify compiled first. But again, this seems to be confirmed to only happen if I haven't yet completed a full deps.compile because of stopping errors in the middle (i.e. between foo and bar).

[josevalim]

Thank you @Sinc63. The issue may be related somehow to optional dependencies. I am trying to reproduce it.

[josevalim]

@Sinc63 I have tried to reproduce this in several different ways, using the same OTP and Elixir versions, and failed. I tried with optional dependencies, non-optional dependencies, etc. Unfortunately I will need a mechanism to reproduce this issue to move forward. Thanks!

[josevalim]

@Sinc63 ping :)

[thbar]

I have somehow stumbled upon what appears to be exactly the same situation, with the same library phoenix_markdown, something that is curious enough to mention it here. I will investigate on our side (e.g. how reproducible it is etc).

• https://github.com/etalab/transport-site/issues/3499

[Sinc63]

@josevalim pong. Mix.exs:

defmodule CompileDeps.MixProject do
  use Mix.Project

  def project do
    [
      app: :compile_deps,
      version: "0.1.0",
      elixir: "~> 1.13",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger]
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      {:mogrify, "~> 0.4"},
      {:link_preview, github: "E-MetroTel/link_preview", branch: "upgrade_floki"},
      # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"}
    ]
  end
end

cat .tool-versions

erlang 25.3.2.5
elixir 1.15.5-otp-25

console:

mix deps.compile --force floki mogrify mime tesla
==> floki
Compiling 2 files (.erl)
Compiling 29 files (.ex)
Generated floki app
==> mime
Compiling 1 file (.ex)
Generated mime app
==> tesla
Compiling 34 files (.ex)
Generated tesla app
==> mogrify
Compiling 9 files (.ex)
Generated mogrify app

mix deps.compile link_preview --force
==> link_preview
Compiling 11 files (.ex)
error: Mogrify.Image.__struct__/0 is undefined, cannot expand struct Mogrify.Image. Make sure the struct name is correct. If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code
  (link_preview 1.0.3) lib/link_preview/parsers/html.ex:147: LinkPreview.Parsers.Html.filter_small_image/2

== Compilation error in file lib/link_preview/parsers/html.ex ==
** (CompileError) lib/link_preview/parsers/html.ex: cannot compile module LinkPreview.Parsers.Html (errors have been logged)
    (stdlib 4.3.1.2) lists.erl:1462: :lists.mapfoldl_1/3
    (stdlib 4.3.1.2) lists.erl:1463: :lists.mapfoldl_1/3
could not compile dependency :link_preview, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile link_preview --force", update it with "mix deps.update link_preview" or clean it with "mix deps.clean link_preview"

1.14.5-otp-25 does not appear to exhibit the same problem.

[josevalim]

Here is the bug:

https://github.com/E-MetroTel/link_preview/blob/upgrade_floki/mix.exs#L54

link_preview is not listing mogrify as an application it depends on, and because Elixir v1.15 is more strict about application dependencies, it fails. My suggestion is to replace the linked line by [extra_applications: [:logger]] and let Elixir compute the applications for you. :)

[josevalim]

@thbar it is the same issue on Phoenix Markdown, I will submit a PR there.

[josevalim]

In both cases, you should most likely be getting warnings for this in previous Elixir versions, which were on purpose supporting broken behaviour, but we wanted to easy migration. :) The issue is that, in past cases, if you kept the warning on, you would find out only when there was a release.

[AntoineAugusti]

Getting this on scrivener_html as well

error: module Phoenix.HTML is not loaded and could not be found. This may be happening because the module you are trying to load directly or indirectly depends on the current module
  lib/scrivener/html.ex:2: Scrivener.HTML (module)

error: module Phoenix.HTML is not loaded and could not be found. This may be happening because the module you are trying to load directly or indirectly depends on the current module
  lib/scrivener/html/seo.ex:15: Scrivener.HTML.SEO (module)

== Compilation error in file lib/scrivener/html/seo.ex ==
** (CompileError) lib/scrivener/html/seo.ex: cannot compile module Scrivener.HTML.SEO (errors have been logged)
    (elixir 1.15.5) expanding macro: Kernel.use/1
    lib/scrivener/html/seo.ex:15: Scrivener.HTML.SEO (module)
could not compile dependency :scrivener_html, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile scrivener_html --force", update it with "mix deps.update scrivener_html" or clean it with "mix deps.clean scrivener_html"

[josevalim]

Same issue: https://github.com/mgwidmann/scrivener_html/blob/master/mix.exs#L32

It should be extra_applications and it seems there were warnings back to Elixir v1.11 :)

[AntoineAugusti]

Thanks @josevalim, this does the trick! Submitted a PR.