Regression testing

[Ruudjhuu]

I implemented a start/skeleton/quick idea/POC regarding regression testing. This python implementation is able to test modules individual. This is an answer on my initial post #51. I make too many mistakes when expanding this code base where I do not yet fully understand all calculations. Regression test will help identify if ideas will work and not break other models. The process is as follows:

• A scad file is generated with the bare minimum to create a object from a module
• A ascii stl is generated by invoking the openscad command
• The output stl is compared with an expected stl

I developed this on a Linux system with python3.10.9 and would like to know if it also works for Windows. There might be an issue with the use of ".." for a parent directory on Windows. I tried my best to not use external libraries, but I am not yet sure if I succeeded.

Before I create more testcases I would like your opinion on this approach.

To run the tests clone this repo, go inside the folder and execute:

python -m unittest tests/utility_test.py -v

The output should look something like this:

test_cutter_weight (tests.utility_test.BaseplateTest) ... Run with command: openscad --enable fast-csg -o ./openscad_test_22022023_180744/out.stl ./openscad_test_22022023_180744/test.scad
ok
test_block_base_hole_style_0 (tests.utility_test.UtilityTest) ... Run with command: openscad --enable fast-csg -o ./openscad_test_22022023_180744/out.stl ./openscad_test_22022023_180744/test.scad
ok
test_block_base_hole_style_1 (tests.utility_test.UtilityTest) ... Run with command: openscad --enable fast-csg -o ./openscad_test_22022023_180744/out.stl ./openscad_test_22022023_180744/test.scad
ok
test_block_base_hole_style_2 (tests.utility_test.UtilityTest) ... Run with command: openscad --enable fast-csg -o ./openscad_test_22022023_180744/out.stl ./openscad_test_22022023_180744/test.scad
ok
test_block_base_hole_style_3 (tests.utility_test.UtilityTest) ... Run with command: openscad --enable fast-csg -o ./openscad_test_22022023_180744/out.stl ./openscad_test_22022023_180744/test.scad
ok

----------------------------------------------------------------------
Ran 5 tests in 0.374s

OK

When the testcases are going to be expanded, I would also like to introduce running all tests automatically on every pull request.

Let me know what you think.

[Ruudjhuu]

@kennetek Could you please look at this feature (you set minimal reviewers needed to 1, and I can not approve it myself). I really want this feature before developing more features. There is too much hard too follow code in this repo and 1 line change could potentially break 90% of the features without it being notified.

[kennetek]

Well this is overwhelming and very very cool. Do you have anything you want me specifically to look at now? It seems very developed at this point and I probably can't contribute much. Although, I cloned this branch and ran the unit and integration tests, and got 2 and 22 failures respectively. That does not seem like the expected output, but I have no clue.

[Ruudjhuu]

I understand this is overwhelming. The things I would like you to look at are:

• Reproducibility Are you able to run the tests with success on your machine/machines? You already tried, that is much appreciated. The downside of these tests is that I needed to create a dependency on openscad stable version as the results of the nigltly can (and did) change overnight. Could you try it with OpenScad 2021.01?
• Usability Do you understand how a test should be written? If you look at tests/test_unit_utility.py line 116 where sweep_rounded module is tested (I think it is the most complicated test in this branch), do you understand the flow and are you able to create such a test yourself? Is it more or less logical how a test should be written and more important: if not, what is holding you back?

A little bit python unittest background: When creating tests, a class should be created containing the tests. All functions in that class beginning with: test_' will be executed as standalone test.setUp()` will be executed before every test defined in that class.

I did not changed the openscad code and made the assumption that all implemented openscad functionality is as expected and should stay that way.

I did already use these tests to validate the regression of bug fixes in other branches I created and to validate pull requests of others. I already prevented breaking some random stuff with the help of these tests. Currently the integration tests are most valuable for bug fixing and sanity checking. Unit/module tests will be valuable for new features and refactoring as they are faster then I'd ever expected.

[kennetek]

After cloning your branch, I tried to run the python scripts again with the 2021 version of OpenSCAD (you were right, I had the unstable build as my main install). However, it does not run as it fails to import open3d. I was not able to use pip to install it, it said module not found despite PyPi saying it should just be pip install open3d. What version of python are you using? Also, shouldn't there be a requirements file that includes open3d and any other libraries you are using?

[Ruudjhuu]

Ah, yes I just added the open3d dependency. Open3d is not available yet for python 3.11. but it should be for 3.7, 3.8, 3.9, and 3.10. The automated tasks on GitHub uses 3.10.

With the open3d dependency it might be possible that the unstable versions of openscad also works. I'm going to test it as we can speed things up with some expirimental features.

[Ruudjhuu]

Not interested anymore in getting this to work.