Describe usage with umbrella apps

[hauleth]

What is needed to get simultaneous output in umbrella project is that you need to add:

     test_coverage: [output: "../../cover"],

In your Mix config. However this still doesn't provide CLI output.

[alfert]

Hmm, what do you mean: There is no CLI output at all or no summarized output for the entire umbrella project? The latter is what I would expect, since umbrellas simply arrange a common build across a set of projects. Running mix test or mix compile for example does not give you a summarized report either (nevertheless that would be nice, but should be handled in mix first).

[hauleth]

Yes. Output of mix test --cover is:

==> kokon
Cover compiling modules ...
.........

Finished in 0.2 seconds
9 tests, 0 failures

Randomized with seed 376705

Generating cover results ...
==> kokon_web
Cover compiling modules ...
...

Finished in 0.05 seconds
3 tests, 0 failures

Randomized with seed 762656

Generating cover results ...

One app mix.exs

defmodule Kokon.Mixfile do
  use Mix.Project

  def project do
    [app: :kokon,
     version: "0.0.1",
     build_path: "../../_build",
     config_path: "../../config/config.exs",
     deps_path: "../../deps",
     lockfile: "../../mix.lock",
     elixir: "~> 1.4",
     elixirc_paths: elixirc_paths(Mix.env),
     start_permanent: Mix.env == :prod,
     test_coverage: [output: "../../cover"],
     aliases: aliases(),
     deps: deps()]
  end

  # Configuration for the OTP application.
  #
  # Type `mix help compile.app` for more information.
  def application do
    [mod: {Kokon.Application, []},
     extra_applications: [:logger, :runtime_tools]]
  end

  # Specifies which paths to compile per environment.
  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_),     do: ["lib"]

  # Specifies your project dependencies.
  #
  # Type `mix help deps` for examples and options.
  defp deps do
    [{:postgrex, ">= 0.0.0"},
     {:ecto, "~> 2.1"},
     {:comeonin, "~> 3.0"},
     {:faker, "~> 0.7", only: :test},
     {:ex_machina, "~> 2.0", only: :test, runtime: false}]
  end

  # Aliases are shortcuts or tasks specific to the current project.
  # For example, to create, migrate and run the seeds file at once:
  #
  #     $ mix ecto.setup
  #
  # See the documentation for `Mix` for more info on aliases.
  defp aliases do
    ["ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
     "ecto.reset": ["ecto.drop", "ecto.setup"],
     "test": ["ecto.create --quiet", "ecto.migrate", "test"]]
  end
end

Second app mix.exs

defmodule Kokon.Web.Mixfile do
  use Mix.Project

  def project do
    [app: :kokon_web,
     version: "0.0.1",
     build_path: "../../_build",
     config_path: "../../config/config.exs",
     deps_path: "../../deps",
     lockfile: "../../mix.lock",
     elixir: "~> 1.4",
     elixirc_paths: elixirc_paths(Mix.env),
     compilers: [:phoenix, :gettext] ++ Mix.compilers,
     start_permanent: Mix.env == :prod,
     test_coverage: [output: "../../cover"],
     aliases: aliases(),
     deps: deps()]
  end

  # Configuration for the OTP application.
  #
  # Type `mix help compile.app` for more information.
  def application do
    [mod: {Kokon.Web.Application, []},
     extra_applications: [:logger, :runtime_tools]]
  end

  # Specifies which paths to compile per environment.
  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_),     do: ["lib"]

  # Specifies your project dependencies.
  #
  # Type `mix help deps` for examples and options.
  defp deps do
    [{:phoenix, "~> 1.3.0-rc"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_ecto, "~> 3.2"},
     {:gettext, "~> 0.11"},
     {:kokon, in_umbrella: true},
     {:cowboy, "~> 1.0"}]
  end

  # Aliases are shortcuts or tasks specific to the current project.
  # For example, we extend the test task to create and migrate the database.
  #
  # See the documentation for `Mix` for more info on aliases.
  defp aliases do
    ["test": ["ecto.create --quiet", "ecto.migrate", "test"]]
  end
end

Global mix.exs:

defmodule Kokon.Umbrella.Mixfile do
  use Mix.Project

  def project do
    [apps_path: "apps",
     start_permanent: Mix.env == :prod,
     dialyzer: [plt_add_deps: :transitive],
     test_coverage: [tool: Coverex.Task],
     deps: deps()]
  end

  # Dependencies can be Hex packages:
  #
  #   {:mydep, "~> 0.3.0"}
  #
  # Or git/path repositories:
  #
  #   {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}
  #
  # Type "mix help deps" for more examples and options.
  #
  # Dependencies listed here are available only for this project
  # and cannot be accessed from applications inside the apps folder
  defp deps do
    [{:ex_doc, ">= 0.0.0", only: :dev, runtime: false},
     {:credo, ">= 0.0.0", only: :dev, runtime: false},
     {:distillery, "~> 1.0", only: :dev, runtime: false},
     {:dialyxir, ">= 0.0.0", only: :dev, runtime: false},
     {:coverex, "~> 1.4", only: :test, runtime: false},
     {:mix_test_watch, "~> 0.3", only: :dev, runtime: false}]
  end
end

[raarts]

Won't that overwrite some filenames, like modules.html/functions.html?

[schnittchen]

@raarts is right:

✓ wc -l apps/*/cover/functions.html | grep total
 12184 total

vs.

✓ wc -l cover/functions.html 
9530 cover/functions.htm

[schnittchen]

When umbrella applications depend on another, dependent modules appear in the dependency's report.

Coverage in umbrellas seems to be only sensible when results are merged.

[alfert]

As explained above, coverage in umbrellas works per project, i.e. every project in an umbrella has its own coverage test run and report. Therefore, dependent apps from the umbrella are measures several times, but again consistently as they are required in the project's test run.

What @hauleth proposes does not work, since this will override the results from each test run, i.e. the last run project will win. Providing a global reporting for umbrellas would be nice, but does not exist for compiling and test results either. As @schnittchen points out, (global) coverage reports make only sense if results are merged. But this requires a completely different handling of jobs inside a mix build when running on an umbrella project.

[alfert]

I close this thread, since I will not change the behaviour unless mix itself changes the way of building umbrella projects.

[raarts]

Two notes: maybe it would be prudent to add a note to the README about the bit that coverage in umbrellas works per app, second, I propose the idea of storing output for each app in umbrella-root/cover/<appname>. Making it easier to publish the entire coverage report as one directory (or save it during across CI stages).

[alfert]

Good points, I will consider that.

Von unterwegs gesendet

Am 29.06.2018 um 11:18 schrieb Ron Arts notifications@github.com:

Two notes: maybe it would be prudent to add a note to the README about the bit that coverage in umbrellas works per app, second, I propose the idea of storing output for each app in umbrella-root/cover/. Making it easier to publish the entire coverage report as one directory (or save it during across CI stages).

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.