search

hildogjr / KiCost

Build cost spreadsheet for a KiCad project.
MIT License
503 stars 98 forks source link

CI: Comparing test results with golden files #404

Closed mdeweerd closed 4 years ago

mdeweerd commented 4 years ago

I've added golden files to the test setup.

The generated xlsx files are converted to CSV files which are then compared to the golden files.

This will allow the detection of changes in the future.

An error is generated if this fails.

(Hopefully) works with travis.

hildogjr commented 4 years ago

Are you planing to create some automatic CI to compare the XLXS results at any pull request file changes?

mdeweerd commented 4 years ago

Yes, this is the purpose - it seems to work - Py3.7 passed now after removing some files that should not have been there. I compare CSV files created from the XLSX files.

mdeweerd commented 4 years ago

https://travis-ci.com/github/xesscorp/KiCost/jobs/339512671 fails because there is a different result than with the other python versions. So this probably reveils some kind of bug.

mdeweerd commented 4 years ago

I consider that the tests are now effective and capable of identifying bugs and changes (the golden files may need to be updated of course after making a functional change - just copy the "result_test" directory to the "expected_test" directory if needed.

test_small.sh woks with "result_test_small" and "expected_test_small" .

hildogjr commented 4 years ago

Can I merge it now?

mdeweerd commented 4 years ago

Sometimes it is a matter of order (in the references field) - to be fixed by ordering the values in this field.

But there are also more profound differences:

"C1,C2",000004V1RA,CAP CER 27PF 50V NP0 0402,000004V1RA,Kemet,C0402C270J5GACTU,0,0,0

"C1,C2",000004V1RA,CAP CER 27PF 50V NP0 0402,CAP CER 27PF 50V NP0 0402,Kemet,C0402C270J5GACTU,0,0,0

with diff -r result_test/TestParts.csv expected_test/TestParts.csv on Python 3.5.6

mdeweerd commented 4 years ago

You can merge it ;-)