Open hauleth opened 3 years ago
Yeah that's an unfortunate limitation of how we have to set things up for CT which wants a specific weird file layout. We end up copying files into an extra directory from which we run CT, and when files are renamed, we don't detect in that directory that the old one doesn't exist and instead the file is left hanging there. Next invocations of CT find it and try to run it regardless.
Yea, that and we don't just want to delete and re-copy all files since we use timestamp to do compilation -- and symlinks aren't an option because they'd put beam files in the test dir.
Is the best way to just do a pre or post copy check that looks to see if each module has a corresponding module in the original test dir and delete if not?
@tsloughter TBH I would expect tests to be recompiled on each rebar3 ct
call as environment could change in the meantime and it could also invalidate generated modules.
They are recompiled if they changed. The environment and compile options are tracked in the DAG and if you modify them, things get recompiled. But when we can safely infer that nothing changed in the files (and we drive the analysis starting from the source files, not the build artifacts), we don't recompile.
The thing is that CT asks that the source be present next to the .beam files in a directory that executes tests. So the thing we need to do to please CT is to copy the source file along with the .beam files as if they were a build artifacts, but just for common tests and nothing else ever. If you go look into _build/test/lib/<yourapp>/test/
you will see that the directory is special and is a copy of all your test files with the .beam files right in there rather than in ebin/
. If you have many test directories, they all get merged down to this temporary one. This is the weird thing that CT requires but is never really documented. The only great way to work around it is to pair and track each source file in that magic directory with any place where they'd be from, and do some pairwise comparison to know if a file disappeared from the source location to take it out. This tends to cause funky issues if you moved the file because you could risk deleting a source file that was moved but not renamed and then I'm not 100% sure what would happen.
The fix was generally considered so shitty and expensive (it would require duplicating and tweaking compiler-related code for one-off magic rules that other test frameworks do not require) that we more or less decided to eat the loss on it and just have it misbehave when files are renamed since we assume people won't just reorganize their repo every week or so. People generally had the habit of blowing away the _build directory when things went bad and it's one of the cases where they had to fall back to that pattern.
As an aside this pairing of location and source file is also why the rebar3 ct --retry
hook cheat thing tends to break a lot. The reinvocation of CT in that directory has it try to understand things based on source files that aren't necessarily pre-loaded properly (I'm not 100% sure) and it tries to complete paths from the incomplete picture and keeps choking.
My approach would be to remove _build/test/lib/<app>/tests
after each rebar3 ct
, but I assume that it is unwanted solution.
In my mix_erlang
library I am compiling test modules in memory only, but I assume that it is also not something that is expected.
yeah, removing them all would essentially massively slow down builds for people who have hundreds of applications and files in their repositories. They'd need to be recompiled every time and that can represent a significant amount of time as the compiler slowed down a lot in the last versions in attempts to do more optimization passes.
Compiling to memory would be interesting, but that would require compiler changes as well, and just for CT, since the common test handler currently just calls to the same compiler modules we use for everything else (.yrl, .mib, .erl, etc.). If we are to drive custom compiler changes, it feels to me like it would be simpler to track the copied .erl files as another build artifact in the DAG, which would let us auto-remove them when we detect the original source is gone. It's just that tracking these files is not needed anywhere else and it really doesn't make sense for any other compiling job to do that.
Yea, much simpler to just compare files across the two directories.
I realized it should be done prior to the compilation because the rebar3 compiler already has the code for removing files from the DAG, so if CT can remove files after its copy but before the call to the compiler then it would handle the rest itself.
@ferd I feel like we didn't understood each other. I was talking about always recompiling only Common Test files (if that is feasible in Rebar3), rest of the compilation artifacts should be left as is.
So was I. People with hundred of OTP applications in their projects also have hundreds if not thousands of test files that wouldn't always need recompiling unless they modified them.
Current behaviour
When I rename/remove the common test suite module then running
rebar3 ct
will still try to run that, which will fail as this module no longer exist. Example from running first tests on current master of OpenTelemetry and then checking out version before rename:rebar3 as test clean
do not fixes the problemExpected behaviour
Old test shouldn't be ran.