Make it easier to spot compile-time graphs

[josevalim]

As discussed in https://github.com/phoenixframework/phoenix/issues/5893, if you have a compile-time dependency in a cycle, it makes it so any runtime dependency in a file becomes a compile-time dependency. Therefore, it is extremely important that we make these cases easier to spot.

My suggestion is to introduce mix xref cycles, bringing it at the top level, and allow cycles to be classified by compile / export / runtime.

[zachdaniel]

I think in addition to this that there may be something wrong with the current implementation of mix xref graph that causes it not to revisit a file (likely to prevent infinite loops)

What I would have expected to see in the linked example would be something like this:

router.ex
  - test.ex (compile)
    -  router.ex (runtime)
      - some_live_view.ex (runtime)
        - gettext.ex(compile)

but it actually just stops here

router.ex
  - test.ex (compile)

[josevalim]

Yes, we have to stop there because, as you said, it would be an infinite loop otherwise. We are trying to print a graph as lines of texts, limitations are expected. What we could though is this:

router.ex
  - test.ex (compile)
    -  router.ex (cycle!)

So at least it becomes clearer. I will look into this as well.

[josevalim]

Even better if we can show it as (compile cycle!) but no promises.

[zachdaniel]

👍 perfect. And I'd absolutely add something like mix xref cycles --fail-above 0 to CI :)

[josevalim]

It is hard to avoid all cycles and runtime ones are harmless. I am also seeing there are benefits in keeping --format cycles because it can use the same helpers.

Perhaps what we really need to do is to add a function called mix xref doctor which looks for the number of compile-connected and cycles with exports/compile entries in them.

[zachdaniel]

Yeah, I was assuming mix xref cycles was only showing compile cycles, but if its including runtime cycles then yeah I wouldn't want that to fail CI 😆

[josevalim]

cc @marcandre who may also have ideas here.

[zachdaniel]

FWIW I think that there could possibly be a way to make this visible in the existing tasks that could be minimally invasive. Specifically, if we're stopping because we've already "seen" a given node, we could continue one time if this time it's a compile time dependency whereas last time it was a runtime dependency (or was the module we started at).

Then cycles would be displayed in mix xref graph --label compile-connected.

I haven't looked at the implementation, so perhaps what I'm saying is incorrect, but I would suggest that, if possible, making mix xref graph behave that way might ought to be done regardless of other tasks like mix xref cycles or mix xref doctor.

router.ex
  - test.ex (compile)
    -  router.ex (cycle!)
      - some_live_view.ex (runtime)
        - gettext.ex(compile)

In that example, it doesn't show test.ex again, it only shows new compile-connected dependencies. So I don't think it would be an infinite loop, and we may only have to go through the cycle one time?

[marcandre]

I'm not sure I understand what is so particular with cycles. In my codebases, I have a single criteria to avoid all transitive dependencies which is no "compiled-connected".

It gets slightly more complicated than this as there are usually a few really basic files refer to themselves compile-time, but to nothing else so the actual 100% fool-proof way is no compiled-connected dependency when removing that core group, and no (runtime) dependencies from that core group, i.e.:

mix xref graph --label=compile-connected --group a,b,c,d,e --exclude a --fail-above 0`
# and
mix xref graph --group a,b,c,d,e --source a --fail-above 0

I believe that you can make the above work if and only if a codebase will not recompile needlessly files (unless one touches a, b, ..., or e, which is rare)

Personally, I would rather improve xref graph to facilitate the above than anything else.

Are there cases where the above is not the correct approach but looking at cycles is?

[josevalim]

I'm not sure I understand what is so particular with cycles. In my codebases, I have a single criteria to avoid all transitive dependencies which is no "compiled-connected".

You are correct, the root cause is still a compile-connected dependency somewhere. However, imagine you have 30 compile-connected dependencies in a project. The ones that will give you the biggest benefit in addressing are the ones that also form a cycle, because if you have a compile dependency in a cycle, it means that all runtime dependencies of any element in the graph now becomes a compile-time dependency.

[marcandre]

Thinking more about it, here's why I think it's a mistake to focus on cycles instead of compile-connected dependencies.

Imagine a code base in good order. It has a file c.ex which is a compile-dependency of some other files, and c.ex does not refer to anything. All good so far.

Now Anton modifies c.ex and introduces a runtime dependency to a new file y.ex which doesn't refer to anything.

This is a compile-connected dependency, and there is no (new) cycle. You might say "But Marc-André, I don't really care if only y.ex recompiles for nothing, it's so quick on my M9". Ok.

A month later, Bob adds a runtime dependency in y.ex to a new file z.ex (which itself refers to nothing). Again "Marc-André, it's ok, it's only two files that recompile for nothing from time to time".

Another month passes, and Christian wants to modify z.ex to have a runtime dependency on a.ex which itself refers to just about everything else in the app, so now you have cycles and CI fails.

The issue is that Christian did nothing wrong and has no context on what needs fixing. It might make a lot of sense for z.ex to refer to A. Christian now has to analyze what is going on, realize that Bob also probably did nothing wrong, and that really the issue is what Anton did two months ago, 142 commits before, it is that dependency (c.ex -> y.ex) that needs to be severed, which has nothing to do with what he's working on.

I believe that transitive dependencies, as "innocent" as they might be at first, can become issues later on and are better fixed as soon as introduced, and although cycles make them much worse, they are not necessary.

[marcandre]

I was writing at the same time...

However, imagine you have 30 compile-connected dependencies in a project. The ones that will give you the biggest benefit in addressing

I'm not convinced there's any other good way forward than to sever the 30 compile-connected dependencies, but that's correct.

Maybe an option that would list the compile-connected dependencies in descending order of the magnitude of the connected part would be helpful. Cycles would be automatically first.

[josevalim]

Maybe an option that would list the compile-connected dependencies in descending order of the magnitude of the connected part would be helpful. Cycles would be automatically first.

Yes, that's what I am thinking. Compile-connected in cycles get priority 99, the other ones get priority 95.

[marcandre]

Maybe an option that would list the compile-connected dependencies in descending order of the magnitude of the connected part would be helpful. Cycles would be automatically first.

Yes, that's what I am thinking. Compile-connected in cycles get priority 99, the other ones get priority 95.

One could imagine a compile-time dependency refering to a small cycle of 3 files (which wouldn't be too bad), and a huge tree of 300 files (i.e. cycle-free), and the priority should be on severing the reference to the big tree, no? That's why I think simply counting how many files are in the connected component(s) of the compile-time dependency is a better metric.

[zachdaniel]

Not to be a nuisance, but I do want to make it clear that my original issue is that there was no good way to spot a compile-connected dependency that comes about from a cycle.

I think that is even more important to fix. There are hidden compile-connected dependencies in the example app I linked from the other issue.

❯ mix xref graph --label compile-connected
lib/test_web/router.ex
└── lib/test.ex (compile)

That output isn't helpful, as it's showing a normal compile time dependency that I was aware of. It doesn't indicate to me at all that lib/test.ex has a runtime dependency on lib/test_web/router.ex, and the downstream effects of that.

I'm all for general purpose tooling to help people with these kinds of problems, but to me this is the more pressing issue.

[marcandre]

@zachdaniel that is correct, compile-connected only shows the compile dependency, it might be more useful to show the connected dependency too (e.g. those of lib/test.ex). I usually just use --source lib/test.ex, but that step wouldn't be necessary.

I believe this is true if there's a cycle or not.

[marcandre]

@josevalim Should I make a PR to show both the compile-time dependency and connections (just one level down)?

[zachdaniel]

🤔 How am I supposed to read that in that case? Is it just saying "this module you have a compile time dependency on depends on something else at runtime"? How would I go about using that information to figure out what the cause is?

What I would expect --label compile-connected to do is to show me the full path to the modules that a module depends on at compile time because of a transitive compile time dependency.

Basically the equivalent of first running --label compile, and then from the leaf nodes running --label runtime.

[josevalim]

@zachdaniel what you are asking is not possible given the visual representation of a terminal. You are asking for us to show this:

router.ex
  - test.ex (compile)
    -  router.ex (cycle!)
      - some_live_view.ex (runtime)
        - gettext.ex(compile)

while we show this:

router.ex
  - test.ex (compile)
    -  router.ex (cycle!)
  - some_live_view.ex (runtime)
    - gettext.ex(compile)

The first thing is to realize that both representations are the same. Because there is a cycle, you can append the -some_live_view subtree on any of the router nodes. However, you are asking for us to continue within a cycle, and that can be quite problematic. Besides the possibility of loops and a lot of nesting, we would be very order dependent. For example, if we happened to show this:

router.ex
  - some_live_view.ex (runtime)
    - gettext.ex(compile)
  - test.ex (compile)
    -  router.ex (cycle!)

Then the dependencies have already been placed and we wouldn't nest either way. Unfortunately I don't think this is feasible, so I propose we drop this subthread. :)

[josevalim]

@zachdaniel if you plot any of the three representations above in a piece of paper, you would see they are the same representation and the dependency into the graph would become obvious. It is just hard to do in a terminal.

It is also worth pointing you two are asking different questions. @zachdaniel is asking "why is gettext causing router.ex to recompile". @marcandre is asking "which files I need to improve to reduce compilation woes". @marcandre is pointing out that, by asking his question, you will eventually solve the gettext issue, even if you can't dissect it. I think there is wisdom in @marcandre's argument. For any considerably large project, trying to understand why it happens is very hard, because there are so many nodes.

[zachdaniel]

🤔 I'm not necessarily trying to ask for any specific representation. I proposed some ideas, yes.

All I'm really trying to say is that this:

❯ mix xref graph --label compile-connected
lib/test_web/router.ex
└── lib/test.ex (compile)

is the full output of that command from the test application I linked, and It doesn't display any runtime dependencies that are causing the issue. I would be totally fine if the output wasn't displayed nested but instead on top of each other as you displayed, but I don't currently get either. The current output is very confusing, as it is not display even a single source/destination that are connected via a transitive dependency. It is only displaying one compile time dependency.

I understand if we can't necessarily display the nested information, or if some other tact must be taken, but I'm looking for a specific solution to that user experience issue.

It is also worth pointing you two are asking different questions. @zachdaniel is asking "why is gettext causing router.ex to recompile". @marcandre is asking "which files I need to improve to reduce compilation woes". @marcandre is pointing out that, by asking his question, you will eventually solve the gettext issue, even if you can't dissect it. I think there is wisdom in @marcandre's argument. For any considerably large project, trying to understand why it happens is very hard, because there are so many nodes.

I agree that folks should in general be asking "what can I do to make my compile times better", but as a framework author I have to understand the specifics of how the macros we are shipping to users work. I wasn't investigating this for a single application, I was trying to solve a problem that happens for all users of a particular macro being used in Phoenix routers was causing compile time issues.

[josevalim]

is the full output of that command from the test application I linked, and It doesn't display any runtime dependencies that are causing the issue

That's because it is not the runtime dependency that is causing the issue. It is the test.ex printed in the output. The gettext in your example is a red herring.

[josevalim]

If your question is the user experience, look at compile-connected. If you want to understand why the compile connected is there, use mix xref trace.

[zachdaniel]

If I didn't have any runtime dependencies, nothing would have shown up, right? So it is the combination of the two that makes a transitive compile time dependency. I understand now that --label compile-connected only shows compile time connections, if the destination of that connection has a runtime dependency, and I can use that information to figure this out in the future.

But when I say --label compile-connected I'd expect to see, on this graph (I understand not in a direct line due to the way the output works) a module and a path to some module that it has a transitive compile time dependency on.

As it stands right now, --label compile-connected behaves fundamentally differently from any other --label because of that. It is more like --label compile --where-destination-has-a-runtime-dependency 😆.

I'll leave it with that, perhaps what I'm saying doesn't make sense, and in that case I'm sorry for wasting your time 🙇 But from the standpoint of a user of xref, this feels like an inconsistency that ultimately led to lots of confusion on my part.

EDIT:

Like, if we were just talking about it, we'd say "router.ex has a transitive compile time dependency on gettext.ex". not "router.ex has a transitive compile time dependency on test.ex". That is what makes it confusing to use --label compile-connected.

[josevalim]

Keep in mind that the whole point of the --label option is to emit only entries that match the given label. The fact --label compile-connected does not show any runtime dependency is precisely its documented behaviour.

[josevalim]

And --label compile-connected does not really behave any different than --label compile, they are all filters of the relationships.

[josevalim]

One could imagine a compile-time dependency refering to a small cycle of 3 files (which wouldn't be too bad), and a huge tree of 300 files (i.e. cycle-free), and the priority should be on severing the reference to the big tree, no? That's why I think simply counting how many files are in the connected component(s) of the compile-time dependency is a better metric.

You are 100% correct. I have pushed a commit where the --label flag returns more meaningful values under --format stats and --format cycles. Now you can use --format stats --label compile-connected and get the top 10 files with the most incoming compile connected dependencies. Here is the result for Livebook:

Tracked files: 274 (nodes)
Compile dependencies: 163 (edges)
Exports dependencies: 389 (edges)
Runtime dependencies: 973 (edges)
Cycles: 22

Top 10 files with most outgoing dependencies:
  * proto/lib/livebook_proto/event.pb.ex (16)
  * proto/lib/livebook_proto/user_connected.pb.ex (5)
  * proto/lib/livebook_proto/agent_connected.pb.ex (5)
  * lib/livebook_web/endpoint.ex (4)
  * lib/livebook/config.ex (4)
  * proto/lib/livebook_proto/deployment_group_updated.pb.ex (2)
  * proto/lib/livebook_proto/deployment_group.pb.ex (2)
  * lib/livebook_web/iframe_endpoint.ex (2)
  * lib/livebook_web/components/app_components.ex (2)
  * lib/livebook/notebook/learn.ex (2)

Top 10 files with most incoming dependencies:
  * lib/livebook_web.ex (97)
  * lib/livebook/config.ex (3)
  * proto/lib/livebook_proto/deployment_group.pb.ex (2)
  * lib/livebook_web/plugs/memory_provider.ex (2)
  * proto/lib/livebook_proto/user_connected.pb.ex (1)
  * proto/lib/livebook_proto/event.pb.ex (1)
  * proto/lib/livebook_proto/deployment_group_updated.pb.ex (1)
  * proto/lib/livebook_proto/deployment_group_created.pb.ex (1)
  * proto/lib/livebook_proto/app_deployment_status.pb.ex (1)
  * proto/lib/livebook_proto/app_deployment_started.pb.ex (1)

We of course want to fix the top one. :) --format cycles --label compile-connected also pointed to the same direction, but the stats one above is the most obvious.

Thanks everyone for the convo. Elixir got better tonight because of it.

[josevalim]

And here is a commit in Livebook which fixes the one above:

https://github.com/livebook-dev/livebook/commit/edefa6649ab783c1c2f6a2c067b44fa6dc9de642

It mostly extracted all runtime dependencies into a single module which does not depend on anything else.

[marcandre]

@ zachdaniel Your understanding is correct, and your remarks make me think that it might be best to show both the compile dependency and the "connection" dependencies. Sometimes you want to remove the compile-time dependency, but often it's the "connection" dependencies that should be removed. @josevalim would you be against showing them?

Nice improvement with stats @josevalim 🩷

The count is only the number of direct dependency, though, right? Wouldn't a better metric be the number of direct and indirect dependencies?

[josevalim]

@marcandre i am not against showing them but it may slightly change the meaning of the label option. We have to try.

About stats, for incoming, it is only the direct ones, right? If I depend on A which depends on B at compile time, I am not recompiled if A changes.

for outgoing, the whole graph matters more. But I am not sure how to put those in stats and if computing them won’t be too expensive in large projects.

[Gladear]

I'm working on quite a big project (around 2000 files in our lib/ directory, for around 200k lines of code). Lately, I've been working on reducing of number of transitive dependencies on the project (hence PRs on some libraries, https://github.com/elixir-ecto/ecto/pull/4468, https://github.com/phoenixframework/phoenix_live_view/pull/3381). Here's some feedback based on my experience.

(I'll name the parts so it's easier for you to answer to a specific part)

1. Cycles may not be the problem To begin with, in our project, cycles weren't even the biggest issue.

if you have a compile-time dependency in a cycle, it makes it so any runtime dependency in a file becomes a compile-time dependency

Having to recompile a file when any of your project file changes arises in other cases. The doc talks a lot about the following structure : a.ex -> (compile) b.ex -> (runtime) c.ex, and how modifying c.ex will recompile a.ex, but it doesn't talk about the fact that if c.ex depends on other files, if those files are modified, a.ex will be recompiled too. So it's easy to get to the point where c.ex depends on most of your project (e.g. c.ex depends on your Router, which depends on all your controllers, which depend on all your contexts, etc. you get the idea).

This is fixed by following the rule explained in mix xref - "the modules you depend on at compile-time must avoid runtime dependencies within the same project" - but let's be honest, we only go to mix xref when there is an issue with compilation, so I think it would be great to put this forward, in more visible places, maybe in "Meta-programming anti-patterns" in Elixir for example.

2. Hard to understand I must say, I think the changes made in the doc in https://github.com/elixir-lang/elixir/commit/4aace69e01e2f3b2cb686173aeb8f2287c4f446b are a truly great improvement to grasping dependencies between file, and especially what's wrong with these dependencies.

This is the second time I focused on reducing the number of files recompiled when basically any of our files changed. The first time, I didn't even understand how transitive compile dependencies were the problem. I may have overlooked some part of the documentation, but anyway, I remember trying to remove all compile dependencies, not the compile-connected ones. That worked, but wasn't really time-effective.

Anyway, like I said above, I think the changes to xref doc will help greatly. I shared the new documentation to my team, and I'm waiting to see if it helps them understand how dependencies between files work, and how transitive compile dependencies are a problem.

3. Hard to debug Once I understood more clearly how the dependency system worked and what to look for, things began to be easier. Except that they're not exactly always straightforward, and I can't expect everyone to do a deep dive into file dependencies before starting to code in Elixir.

I created a script that tries to evaluate and guide the developer to the source of an issue, basically looking for dependencies in modules that are used as macros. The output looks something like the following:

Issue introduced in the following file(s):

    lib/a.ex

This should not happen 🚨

Diagnosing the issue, this can take a few seconds...

Affected file         : lib/a.ex
Transitive dependency : lib/b.ex
Dependencies:
    lib/c.ex (runtime)

Looking for suspicious lines in the transitive dependency...
    lib/b.ex:2: call C.world/0 (runtime)

Under "Dependencies:" is the output of mix xref graph --source "$transitive_file". Under "suspicious lines" is the output of mix xref trace "$transitive_file".

It's not perfect, sometimes the "suspicious lines" won't be the cause of the problem (see "5. Dependencies"), and I do not have feedback from my team on this tool yet, as it has only been added this week, but I feel like a tool like this could be a great addition to mix. It could probably be much more precise by adding it directly in mix 🙂

4. Not straightforward to prevent Fixing issues is great, preventing them from happening would be greater even. The script I created (see "3. Hard to debug") is called from the CI to prevent us from mistakenly adding new transitive dependencies to our project, but well, it required specific development, a built-in command doing this would be nice.

For the record, the script also as a concept of "safe list" (aka, files we want to ignore through the --exclude flag) which is required for some modules that will cause ok-ish transitive dependencies (e.g. we need to sanitize some output in gettext, requiring the MyApp.Gettext module to rely on other modules... and there isn't much we can do about it, but that's ok because these modules don't depend on other modules from the project). It also has an "ignore list" of known issues, so that we don't have to fix everything all at once.

5. Dependencies Sometimes the problem comes from our code, but sometimes libraries come in and cause trouble too (see the PRs I linked above).

I doubt there's much to be done about it, but well, that's part of what I had to fix 😄 I think that's a reason more to add more information in "Meta-programming anti-patterns", as library writers should be aware of such caveats.

And, that's it ! Sorry for the long speech, still I hope this can help improve mix xref and the compilation of Elixir projects 🙂 On a side note, I saw all the changes made to Elixir to improve compilation (especially in 1.15, that release was a game changer) and I'm really glad for what was done. So, thanks for making our life easier 😄

[josevalim]

so I think it would be great to put this forward, in more visible places, maybe in "Meta-programming anti-patterns" in Elixir for example.

I think this is a great idea. Would you like to take a stab at it?

[Gladear]

Sure ! I'll try to write this in the afternoon (CET) 👌

[mindreader]

I hate to reap this, but the context of this thread is entirely relevant. What would really, truly help me, would be to know which files are actually compiling when I hit save.

Is it possible to type a command with xref that will tell me exactly which files will recompile when I edit a specific file? As it is, all I see is Compiling 16 files (.ex) and I have no idea which files are recompiling or why. That leaves me in the situation of looking at a graph of literally thousands of files, or just a handful of files, wondering if I should've used --sink or --source in my xref command, or --label compile or compile-connected? All of these give me different counts of files sometimes the same file duplicated at different locations, some files have compile labels, others don't.

All I'm left with are doubts and guesses.

[josevalim]

You can save the file and pass the —verbose flag to mix compile to see all compiled files.

[josevalim]

You can then pass the file you changed as the sink (others are compiled because it changed) and the one you are interested as the source. The CLI prints duplicated entries because it is still printing a graph. So if there are many entries pointing to the same place, they appear again.

[mindreader]

This is incredibly enlightening. I can see now where a lot of this comes from. I really wish that I'd known about this years ago.

A major one is plugs. Every plug depends on the router at compile time because plug uses a macro. And though none have any strange code in them, any runtime reference to business logic from within any plug, which feels pretty unavoidable, (get the api key rate limit, pattern match a schema) chains all the way down into the business logic causing any of those business logic files to cause the router to recompile.

Another thing that is hitting us, is some devs write code like

alias My.Schema.User
alias My.Schema.Account
@module_map %{
  account: Account,
  user: User,
  ...
}

This is merely used to convert :user to User in code. While this could have been written as a function, my devs keep trying to move things into attributes as an optimization and if any of the things in the attribute happen to be actual schema atoms, this creates compile time dependencies everywhere.

Okay well thank you for the tip. I think that will help a lot.

edit: What's really weird about this to me, and what is tripping me up is that if I have any reference to an atom that happens to be a module within the same program, it creates a runtime reference. Like if I go.

def foo do
  mod = My.Schema.User # creates runtime dep on the module because it happens to exist
  mod2 = My.Schema.DoesntExist # fine because this module doesn't exist
  IO.puts("#{mod} #{mod2}")
end

$ mix xref trace lib/blah.ex
Compiling 1 file (.ex)
lib/blah.ex:4: alias My.Schema.User (runtime)

What is the rationale for creating these dependencies for merely referencing an atom that happens to be a module somewhere, even if you don't use a function or pattern match or anything else?

[josevalim]

alias My.Schema.User
alias My.Schema.Account
@module_map %{
  account: Account,
  user: User,
  ...
}

IIRC correctly these do not generate compile time dependencies if the attributes are used inside a function only. Run mix trace to double check though.

What is the rationale for creating these dependencies for merely referencing an atom that happens to be a module somewhere, even if you don't use a function or pattern match or anything else?

That's because you could pass them to a function and that function could call something in it.

[marcandre]

alias My.Schema.User
alias My.Schema.Account
@module_map %{
  account: Account,
  user: User,
  ...
}

IIRC correctly these do not generate compile time dependencies if the attributes are used inside a function only. Run mix trace to double check though.

Wow, I double-checked because I also thought these generated compile-time dependencies, and of course you are correct, very nice. The output that mix trace gives is a bit surprising, it lists at the end of a file with "nofile:" as the source, e.g., when line 28 uses the @example to call a function of a module

Compiling 1 file (.ex)
lib/my_file.ex:5: require MyApp.Something... (export)
lib/my_file.ex:5: call MyApp.Something.../1 (compile)
lib/my_file.ex:11: alias MyApp.Something... (runtime)
lib/my_file.ex:27: alias MyApp.Something... (runtime)
lib/my_file.ex:48: alias MyApp.Something... (runtime)
...
lib/my_file.ex:282: call MyApp.Something.../2 (runtime)
lib/my_file.ex:283: call MyApp.Something.../2 (runtime)
nofile:27: alias MyApp.Something... (compile)

What is the rationale for creating these dependencies for merely referencing an atom that happens to be a module somewhere, even if you don't use a function or pattern match or anything else?

That's because you could pass them to a function and that function could call something in it.

On that subject, there are ways to cheat, e.g. :"Elixir.MyApp.MyModule, or using Module.concat

[mindreader]

Edit: I'm not able to replicate these conclusions on the latest Elixir compiler and so I think I need to verify that these issues haven't been fixed already or that there isn't some extra complication in my code leading to this extra recomp.

IIRC correctly these do not generate compile time dependencies

That's true, they are just runtime. But if there is a chain of runtime deps that back up into a compile time dep (as my plugs all do), then any of those runtime deps cause a compile time dep on my router. So even modules that looked completely clean to me, no calls to external functions, actually had dependencies in them to many other modules, purely because they used sensibly named atoms as guards to its own local functions. This lead to a spider web of secret dependencies that caused extra compilation.

That's because you could pass them to a function and that function could call something in it. I guess I don't really understand this. If I have an atom stored in a variable and or passed to a function which is then used to run a function on the module, none of that matters anyways, right? The code accepting that atom may not even use it or reference anything in the original file. Even if it did, the compiler isn't smart enough to know whether or when it will be used, it won't cause different code to be generated in the originating module, so why even track it in the first place?

Like if I use apply(My.Schema.User, :get_user, []) why does that create a runtime dependency? It is a runtime error if User doesn't exist, or if the function doesn't exist, and so tracking the dependency gets us nothing, but greatly increases the coupling of code. Any macro that calls a chain of functions that includes this code is not going to generate different code once User or anything with a runtime dependency down from User it is recompiled, so why recompile it?