elixir-lang / elixir

Elixir is a dynamic, functional language for building scalable and maintainable applications
https://elixir-lang.org/
Apache License 2.0
24.3k stars 3.35k forks source link

Elixir v1.16 unit tests rely on Erlang being compiled with documentation #13516

Closed deepankar-j closed 4 months ago

deepankar-j commented 4 months ago

Elixir and Erlang/OTP versions

Erlang/OTP 26 [erts-14.2.4] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Elixir 1.16.2 (compiled with Erlang/OTP 26)

Operating system

Ubuntu 20.04

Current behavior

Compiling Elixir v1.16 from source and running unit tests results in the following five failures within our CI/CD piplines.

We compile both Erlang and Elixir from source within our pipelines (this gives us the most flexibility to upgrade without relying on package managers). For simplicity, we compile Erlang without documentation. In prior Elixir releases, we were able to compile Elixir and run tests successfully without requiring Erlang documentation to be in place. It appears that with Elixir v1.16, we have to compile Erlang with documentation in order to be able to successfully run the unit tests.

Would it be possible to have these unit tests not rely on Erlang being compiled with documentation?

#12 104.9 ==> iex (ex_unit)
#12 106.4 ..................................................
#12 106.7 
#12 106.7   1) test h prints Erlang module documentation (IEx.HelpersTest)
#12 106.7      test/iex/helpers_test.exs:335
#12 106.7      Assertion with =~ failed
#12 106.7      code:  assert captured =~ "This module provides useful functions related to time."
#12 106.7      left:  ":timer was not compiled with docs\n"
#12 106.7      right: "This module provides useful functions related to time."
#12 106.7      stacktrace:
#12 106.7        test/iex/helpers_test.exs:337: (test)
#12 106.7 
#12 106.8 ..............
#12 106.9 
#12 106.9   2) test h handles non-existing Erlang module function (IEx.HelpersTest)
#12 106.9      test/iex/helpers_test.exs:346
#12 106.9      Assertion with =~ failed
#12 106.9      code:  assert captured =~ "No documentation for :timer.baz was found"
#12 106.9      left:  ":timer was not compiled with docs\n"
#12 106.9      right: "No documentation for :timer.baz was found"
#12 106.9      stacktrace:
#12 106.9        test/iex/helpers_test.exs:348: (test)
#12 106.9 
#12 107.0 .........................................
#12 108.1 
#12 108.1   3) test t prints all types in erlang module (IEx.HelpersTest)
#12 108.1      test/iex/helpers_test.exs:1011
#12 108.1      Assertion with =~ failed
#12 108.1      code:  assert captured =~ "-opaque queue(Item)"
#12 108.1      left:  "\tqueue\n\nThese types are documented in this module:\n\n  -type queue() :: queue(_).\n\n"
#12 108.1      right: "-opaque queue(Item)"
#12 108.1      stacktrace:
#12 108.1        test/iex/helpers_test.exs:1014: (test)
#12 108.1 
#12 108.1 ..
#12 108.1 
#12 108.1   4) test h prints Erlang module function specs (IEx.HelpersTest)
#12 108.1      test/iex/helpers_test.exs:340
#12 108.1      Assertion with =~ failed
#12 108.1      code:  assert captured =~ "-spec sleep(Time) -> ok when Time :: timeout()."
#12 108.1      left:  "\n                                 :timer.sleep/1\n\n  @spec sleep(time) :: :ok when time: timeout()\n\nModule was compiled without docs. Showing only specs.\n\n"
#12 108.1      right: "-spec sleep(Time) -> ok when Time :: timeout()."
#12 108.1      stacktrace:
#12 108.1        test/iex/helpers_test.exs:343: (test)
#12 108.1 
#12 108.1 .................................
#12 108.9 
#12 108.9   5) test t prints single type from erlang module (IEx.HelpersTest)
#12 108.9      test/iex/helpers_test.exs:1017
#12 108.9      Assertion with =~ failed
#12 108.9      code:  assert captured =~ "-type iovec() :: [binary()]"
#12 108.9      left:  "@type iovec() :: [binary()]\n\n"
#12 108.9      right: "-type iovec() :: [binary()]"
#12 108.9      stacktrace:
#12 108.9        test/iex/helpers_test.exs:1019: (test)
#12 108.9 
#12 108.9 .............................................................................................................
#12 113.2 Finished in 7.8 seconds (0.9s on load, 0.3s async, 6.5s sync)
#12 113.2 254 tests, 5 failures
#12 113.2 
#12 113.2 Randomized with seed 758894

Expected behavior

It would be nice to not have to compile Erlang with documentation only to satisfy unit tests. A not so great alternative is that we can skip running Elixir unit tests within our CI/CD pipeline.

josevalim commented 4 months ago

Those tests should be tagged and we should skip them if documentation is not available. But apparently something is not working in your scenario, PRs are welcome.

maltekrupa commented 4 months ago

I see the same behaviour on FreeBSD 14.0-RELEASE-p6.

Running Code.fetch(:array) in an 1.16.2 iex shell returns the following which looks like some form of documentation. But is it the right one? It matches what the code in test_helper.exs expects and does not skip the tests.

$ ./bin/iex
Erlang/OTP 26 [erts-14.2.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns] [dtrace] [sharing-preserving]

Interactive Elixir (1.16.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Code.fetch_docs(:array)
{:docs_v1, [file: ~c"array.erl", location: 0], :erlang,
 "application/erlang+html", :hidden, %{otp_doc_vsn: {1, 0, 0}},
 [
   {{:function, :default, 1}, [file: ~c"array.erl", location: 0], ["default/1"],
    :hidden, %{}},
   {{:function, :fix, 1}, [file: ~c"array.erl", location: 0], ["fix/1"],
    :hidden, %{}},
   {{:function, :foldl, 3}, [file: ~c"array.erl", location: 0], ["foldl/3"],
    :hidden, %{}},
   {{:function, :foldr, 3}, [file: ~c"array.erl", location: 0], ["foldr/3"],
    :hidden, %{}},
   {{:function, :from_list, 1}, [file: ~c"array.erl", location: 0],
    ["from_list/1"], :hidden, %{}},
   {{:function, :from_list, 2}, [file: ~c"array.erl", location: 0],
    ["from_list/2"], :hidden, %{}},
   {{:function, :from_orddict, 1}, [file: ~c"array.erl", location: 0],
    ["from_orddict/1"], :hidden, %{}},
   {{:function, :from_orddict, 2}, [file: ~c"array.erl", location: 0],
    ["from_orddict/2"], :hidden, %{}},
   {{:function, :get, 2}, [file: ~c"array.erl", location: 0], ["get/2"],
    :hidden, %{}},
   {{:function, :is_array, 1}, [file: ~c"array.erl", location: 0],
    ["is_array/1"], :hidden, %{}},
   {{:function, :is_fix, 1}, [file: ~c"array.erl", location: 0], ["is_fix/1"],
    :hidden, %{}},
   {{:function, :map, 2}, [file: ~c"array.erl", location: 0], ["map/2"],
    :hidden, %{}},
   {{:function, :new, 0}, [file: ~c"array.erl", location: 0], ["new/0"],
    :hidden, %{}},
   {{:function, :new, 1}, [file: ~c"array.erl", location: 0], ["new/1"],
    :hidden, %{}},
   {{:function, :new, 2}, [file: ~c"array.erl", location: 0], ["new/2"],
    :hidden, %{}},
   {{:function, :relax, 1}, [file: ~c"array.erl", location: 0], ["relax/1"],
    :hidden, %{}},
   {{:function, :reset, 2}, [file: ~c"array.erl", location: 0], ["reset/2"],
    :hidden, %{}},
   {{:function, :resize, 1}, [file: ~c"array.erl", location: 0], ["resize/1"],
    :hidden, %{}},
   {{:function, :resize, 2}, [file: ~c"array.erl", location: 0], ["resize/2"],
    :hidden, %{}},
   {{:function, :set, 3}, [file: ~c"array.erl", location: 0], ["set/3"],
    :hidden, %{}},
   {{:function, :size, 1}, [file: ~c"array.erl", location: 0], ["size/1"],
    :hidden, %{}},
   {{:function, :sparse_foldl, 3}, [file: ~c"array.erl", location: 0],
    ["sparse_foldl/3"], :hidden, %{}},
   {{:function, :sparse_foldr, 3}, [file: ~c"array.erl", location: 0],
    ["sparse_foldr/3"], :hidden, %{}},
   {{:function, :sparse_map, 2}, [file: ~c"array.erl", location: 0],
    ["sparse_map/2"], :hidden, %{}},
   {{:function, :sparse_size, 1}, [file: ~c"array.erl", location: 0],
    ["sparse_size/1"], :hidden, %{}},
   {{:function, :sparse_to_list, 1}, [file: ~c"array.erl", location: 0],
    ["sparse_to_list/1"], :hidden, %{}},
   {{:function, :sparse_to_orddict, 1}, [file: ~c"array.erl", location: 0],
    ["sparse_to_orddict/1"], :hidden, %{}},
   {{:function, :to_list, 1}, [file: ~c"array.erl", location: 0], ["to_list/1"],
    :hidden, %{}},
   {{:function, :to_orddict, 1}, [file: ~c"array.erl", location: 0],
    ["to_orddict/1"], :hidden, %{}}
 ]}
sabiwara commented 4 months ago

Maybe we can check that the param after "application/erlang+html" is a map / not :hidden?

When compiled with docs:

{:docs_v1, {15, 2}, :erlang, "text/markdown",
 %{
   "en" => "Functional, extendible arrays.\n\nArrays can have fixed size, or can grow automatically as needed. A default value\nis used for entries that have not been explicitly set.\n\nArrays uses _zero_-based indexing. This is a deliberate design choice and\ndiffers from other Erlang data structures, for example, tuples.\n\nUnless specified by the user when the array is created, the default value is the\natom `undefined`. There is no difference between an unset entry and an entry\nthat has been explicitly set to the same value as the default one (compare\n`reset/2`). If you need to differentiate between unset and set entries, ensure\nthat the default value cannot be confused with the values of set entries.\n\nThe array never shrinks automatically. If an index `I` has been used to set an\nentry successfully, all indices in the range `[0,I]` stay accessible unless the\narray size is explicitly changed by calling `resize/2`.\n\n_Examples:_\n\nCreate a fixed-size array with entries 0-9 set to `undefined`:\n\n```\nA0 = array:new(10).\n10 = array:size(A0).\n```\n\nCreate an extendible array and set entry 17 to `true`, causing the array to grow\nautomatically:\n\n```\nA1 = array:set(17, true, array:new()).\n18 = array:size(A1).\n```\n\nRead back a stored value:\n\n```\ntrue = array:get(17, A1).\n```\n\nAccessing an unset entry returns default value:\n\n```\nundefined = array:get(3, A1)\n```\n\nAccessing an entry beyond the last set entry also returns the default value, if\nthe array does not have fixed size:\n\n```\nundefined = array:get(18, A1).\n```\n\n\"Sparse\" functions ignore default-valued entries:\n\n```\nA2 = array:set(4, false, A1).\n[{4, false}, {17, true}] = array:sparse_to_orddict(A2).\n```\n\nAn extendible array can be made fixed-size later:\n\n```\nA3 = array:fix(A2).\n```\n\nA fixed-size array does not grow automatically and does not allow accesses\nbeyond the last set entry:\n\n```\n{'EXIT',{badarg,_}} = (catch array:set(18, true, A3)).\n{'EXIT',{badarg,_}} = (catch array:get(18, A3)).\n```"
 },
 %{
   authors: ["Richard Carlsson <carlsson.richard@gmail.com>",
    "Dan Gudmundsson <dgud@erix.ericsson.se>"],
   otp_doc_vsn: {1, 0, 0}
 },
 [
   {{:type, :indx_pairs, 1}, {204, 2}, ["indx_pairs(Type)"], :none,
    %{exported: false}},
...
maltekrupa commented 4 months ago

Maybe we can check that the param after "application/erlang+html" is a map / not :hidden?

Sounds like it could work.

Hm, I can find the function in test_helper.exs in the main branch, but not in 1.16.2.

sabiwara commented 4 months ago

@maltekrupa you should target the v1.16 branch. Or rather the main branch and we'll backport to v1.16.