openmopac / mopac

Molecular Orbital PACkage
http://openmopac.net
GNU Lesser General Public License v3.0
111 stars 31 forks source link

Fix false negatives in testing #167

Closed godotalgorithm closed 1 year ago

godotalgorithm commented 1 year ago

I noticed a few weeks ago that the testing system was generating false negatives for cases where the test either crashed, or there were differences in the number of filtered lines of output. These have been upgraded to testing failures, and silent failures associated with recent changes to INDO outputs have been fixed by updating the test reference data. There is also a crash occurring in one of the INDO tests (the C.A.S. keyword) that is being fixed (WIP).

Also, I've snuck in a bug fix that fixes #166.

Status

codecov[bot] commented 1 year ago

Codecov Report

Merging #167 (b4be27f) into main (7afda4a) will increase coverage by 0.00%. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #167   +/-   ##
=======================================
  Coverage   66.63%   66.64%           
=======================================
  Files         331      331           
  Lines       73848    73849    +1     
=======================================
+ Hits        49209    49215    +6     
+ Misses      24639    24634    -5     
Impacted Files Coverage Δ
src/INDO/ci.F90 76.94% <100.00%> (ø)
src/input/geo_ref.F90 63.41% <100.00%> (ø)
src/run_mopac.F90 76.34% <100.00%> (+0.03%) :arrow_up:

... and 2 files with indirect coverage changes

godotalgorithm commented 1 year ago

While this resolves the originally observed false negatives and the crashes that they were hiding, there are still some residual false negatives because the compare_output.py script is filtering way too much right now in some cases. This script is literally the first Python program that I ever wrote that wasn't a matplotlib script, and it is rather poorly written in hindsight. I need to rewrite this when I resume the keyword-based testing to expand testing coverage and consider expanding regression testing to other output files beyond the main one.