Qqwy
commented
2 years ago :thinking: Compilation time being affected this badly is not something that is expected nor intended. My best guess as to what causes it, is that the current approach of fully inlining the type-check at each of the places where it is used combines badly with having nested structs containing many fields.
A potential solution would then be to add an option to TypeCheck to not inline certain kinds of checks. But before looking into this, we first need to find out whether this indeed is the problem.
One thing you could try, is to compare the difference in compile times between enabling/disabling runtime checks in the various modules, to see whether the problem indeed originates from the runtime checks being generated (but I'm quite certain that it does).
One more thing you could try, just to test the difference in compile-time, is to see how fast/slow compilation is with version 0.9.x of the codebase (vs the current, 0.10.4 version): Because we want to support type-checking passed-in functions, a number of the type-checks that are generated were rewritten to more complex forms in 0.10.x (and I have not gotten around to switch between the two based on whether there actually is a function value inside). This would be an interesting datapoint, but I don't recommend using version 0.9.x in production because there are a number of bugfixes (only) in newer versions.
And besides this, I'll try to come up with a minimal example as a separate public repo, so it's easier to profile and reason about.
As for the general strategy w.r.t. type-checking 'large' structs: I think that 'check them fully everywhere' is a good default. Being able to override it is however important for performance (moreso runtime performance, but as we see now, likely compile-time as well). For many structs in our codebases we can (up to a point) guarantee that they are only constructed by e.g. a MyModule.new as you also mention. It might make sense to allow people to configure 'check depth' between:
- Check all structs everywhere
- Only perform the full check on
%MyModule{}when passing them to functions ofMyModule. Otherwise only check whether their:__struct__key isMyModule. - Perform shallow checks everywhere (and allow this to be overridden locally for e.g.
MyModule.new.
Even nicer might be to perform checks based on the opaqueness of a type's typespec. Unfortunately, in practice this leads into a mire where Dialyzer and Elixir's defaults don't work well together as checking whether a value even is a particular struct is strictly speaking not allowed on opaque types. (c.f. https://github.com/elixir-lang/elixir/issues/8463).
Some discussion about this also happened in the past in the ElixirForum topic, but it was not the only thing we talked about, so it might be a difficult to look through the conversation.
aj-foster
Hello there!
I enjoyed your talk at the Amsterdam Elixir meetup, and decided to give TypeCheck a try on our codebase. We have a few structs with a relatively large number of fields, and their old
@type tdefinitions fully specified each field. Upon changing these to@type!and implementing a few@spec!s on functions that used them, we quickly found that our compilation time rose to 3–4 minutes.For reference, the rough setup is:
With six function specifications on module
Cthat reference all three types in various combinations.So, the question: is this expected? I can imagine that the sheer number of fields on each struct, especially when nested, creates an explosion of code generation. In the documentation you mention:
Could this be true for compile time as well?
More generally, do you have any recommendations on strategies for type-checking structs? Perhaps the full specification of each field would make sense on a constructor function, but the canonical
ttype should stop at%__MODULE__{}with no fields.Thanks for your work! Excited for more people to see it at ElixirConf EU.