Bisect regenerates the ground-truth.csv result file (by running all baseline tests) many times throughout a bisect run. This may be seen by calling bisect with VERBOSE defined in the Make environment and looking for calls to ./gtrun --output ground-truth.csv, or by analyzing log events in #331.
It should be noted that these unexpected event duplications break the logical program dependencies used in tracing the FLiT process for #331.
Bisect currently has no means to pass Makefile output to an arbitrary stream; all output from makefiles is currently conveyed through stdout during the bisect process, although Python output may be written elsewhere via command line arguments.
Possible Fixes
Extraneous test runs:
Ground truth tests should only be run once, so eliminate any extraneous test calls to gtrun.
The cause is undetermined, but dependencies of the gtrun executable need to be thoroughly reviewed between both the base Makefile.in and the bisect Makefile_bisect_binary.in files.
Tests should be added to check for extraneous events (particularly compilation, links, test runs, and test comparisons.)
Make output:
This is minor, but makes for inconsistent integration testing at least. A couple immediate thoughts:
Leave as is, and adapt integration testing to access stdout for parsing as necessary.
Propagate stream argument to all bisect functions for consistency
Bug Report
Describe the problem
Bisect regenerates the
ground-truth.csv
result file (by running all baseline tests) many times throughout a bisect run. This may be seen by calling bisect withVERBOSE
defined in theMake
environment and looking for calls to./gtrun --output ground-truth.csv
, or by analyzing log events in #331.It should be noted that these unexpected event duplications break the logical program dependencies used in tracing the FLiT process for #331.
Bisect currently has no means to pass
Makefile
output to an arbitrary stream; all output from makefiles is currently conveyed throughstdout
during the bisect process, although Python output may be written elsewhere via command line arguments.Possible Fixes
gtrun
.gtrun
executable need to be thoroughly reviewed between both the baseMakefile.in
and the bisectMakefile_bisect_binary.in
files.stdout
for parsing as necessary.