scad unit tests

[Ruudjhuu]

we have regression testing with https://github.com/kennetek/gridfinity-rebuilt-openscad/pull/76 but, we dont have unit testing possibilities. regression testing is good, when you already have a correct stl file and want to make sure, that the new build will also produce this stl file. Therefore regression testing can not be applied to new modeling. Thats where unit testing comes into play. I dont know how the whole model will look, but I know, that the z dimension will be 42. We need a way to make sure, that this assumption is true. Aka, the model really has a z dimension of 42.

[Ruudjhuu]

Test idea

Test scenario

Lets assume we want to test the folowing scad code (copied from code base):

module cut(x=0, y=0, w=1, h=1, t=1, s=1) {
    translate([0,0,-$dh-h_base])
    cut_move(x,y,w,h)
    block_cutter(clp(x,0,$gxx), clp(y,0,$gyy), clp(w,0,$gxx-x), clp(h,0,$gyy-y), t, s);
}

Test features

To unit-test the scad code block we need a few features in the test framework:

• A test name.
• Control the argument given to the module, also children if needed.
  if tests are writen in openscad, this would automaticaly be supported.
• Predefine the global variables
  • Can be defined in scad file (hard to test different values)
  • Add it as comment so the testframework can parse it, you can set different values for different testcases.
  • Don't use global variables. It's bad.
• Mock modules
  Can be possible by defining the mock in openscad. Add in a comment which function the definition should mock. Test framework should generate the mock.
  • mock cut_move.
  • mock block_cutter.
• assert outcome of the module.
  Test framework should support assert comments with easy to use assert posibilities.
• suport modules generating 3d and 2d output I think Openscad tells if the output is 2D or 3D but after the command has run, after you already need to know if it is a 2D or 3D output.

Proposal

Create seperate test .scad file for tests. Use comments to indicate how the test will look. The following snippet defines two testcases:

// Testcase: test_12345:3D
// Global: $gdh=2
// Global: $gxx=5
// Global: $gyy=10
// Mock: cut_move = cut_move_mock
// Mock: block_cutter = block_cutter_mock
// Assert: total_z = 22
cut(1,2,3,4,5);

// Testcase: test_54321:3D
// Global: $gdh=2
// Global: $gxx=5
// Global: $gyy=10
// Mock: cut_move = cut_move_mock
// Mock: block_cutter = block_cutter_mock
// Assert: total_z = 22
cut(5,4,3,2,1);

module cut_move_mock(a,b,c,f){translate[10,10,10]children();}
module block_cutter_mock(a,b,c,d,e,f){cube([10,10,10]);}

If the tested module has children it could look like:

// Testcase: test_children:3D
// Global: $gdh=2
// Global: $gxx=5
// Global: $gyy=10
// Mock: cut_move = cut_move_mock
// Mock: block_cutter = block_cutter_mock
// Assert: total_z = 22
cut(1,2,3,4,5){
  cube([1,1,1]);
  cube([2,2,2]);
}

module cut_move_mock(a,b,c,f){translate[10,10,10]children();}
module block_cutter_mock(a,b,c,d,e,f){cube([10,10,10]);}

Current implementation

This python snipit test exactly the same as the proposal.

class cut(TestCase):
    def setUp(self) -> None:
        self.module_test = ModuleTest(Module.from_file("cut", "gridfinity-rebuilt-utility.scad"), OutputType.STL)
        self.module_test.add_global_variable("gdh", 2)
        self.module_test.add_global_variable("gxx", 5)
        self.module_test.add_global_variable("gyy", 10)
        self.module_test.add_dependency(Module("cut_move", ["translate[10,10,10]children();";]))
        self.module_test.add_dependency(Module("block_cutter", ["cube([10,10,10]);"]))

    def tearDown(self) ->None:
        self.module_test.clean_up()

    def test_12345(self) -> None:
        self.module_test.add_arguments(1,2,3,4,5)
        self.module_test.run(self.id())
        self.assertEqual(self.module_test.svg_result.total_z, 22)

    def test_54321(self) -> None:
        self.module_test.add_arguments(5,4,3,2,1)
        self.module_test.run(self.id())
        self.assertEqual(self.module_test.svg_result.total_z, 22)

    # Testcase with children
    def test_children(self) -> None:
        self.module_test.add_arguments(5,4,3,2,1)
        self.module_test.add_children([ Cube([1,1,1]),
                                        Cube([2,2,2])])
        self.module_test.run(self.id())
        self.assertEqual(self.module_test.svg_result.total_z, 22)

[DJDDDM]

I think we arent fully on the same page, but we are getting definetly closer. Let me add some thoughts about your thoughts. My thoughts are in italic.

To unit-test the scad code block we need a few features in the test framework:

A test name. Not necessarily. All we really need to have is a possibility to know, which assertion failed. For this we could use the scad file name/path and line number. Either instead of the name, or as the default name. My Idea is to allow to write as few lines/characters as possible. Control the argument given to the module, also children if needed. Also not necessarily. Your approach is to import the whole module into python and then have it executed from there. My approach would be to have it run in scad/OpenSCAD and then just check the result. This could be done by using the show only modifier "!" written automatically into the scad code. if tests are writen in openscad, this would automaticaly be supported. are we able to create a shortcut in openscad to run all the tests? Predefine the global variables the global variables are defined by the openscad customizer profile. so tests can either just run using the current or a specifically named customizer profile Can be defined in scad file (hard to test different values) Add it as comment so the testframework can parse it, you can set different values for different testcases. Don't use global variables. It's bad. This is a different problem. This problem arises from the lag of nameable elements inside a bigger structure. We only have list/arrays and no classes, structs or dictionaries. Mock modules I highly doubt, that we will need Mocks. I also verry verry rarely use mocks in all the code I have written in my job. Can be possible by defining the mock in openscad. Add in a comment which function the definition should mock. Test framework should generate the mock. mock cut_move. mock block_cutter. assert outcome of the module. Test framework should support assert comments with easy to use assert posibilities. suport modules generating 3d and 2d output I think Openscad tells if the output is 2D or 3D but after the command has run, after you already need to know if it is a 2D or 3D output. IIRC all 2d objects are actually usable as 3d objects cause they have a certain height. At least in the preview_

// Testcase: test_12345:3D
// Global: $gdh=2
// Global: $gxx=5
// Global: $gyy=10
// Mock: cut_move = cut_move_mock
// Mock: block_cutter = block_cutter_mock
// Assert: total_z = 22
cut(1,2,3,4,5);

lets remove everything I think is not necessary. see above for the reasoning:

// Assert: total_z = 22
cut(1,2,3,4,5);

alternative Form:

cut(1,2,3,4,5); // Assert: total_z == 22

In my picture the first thing that happens is, that the test framework first applies the "!" operator to the code.

!cut(1,2,3,4,5); // Assert: total_z == 22

then it gets the updated preview from OpenSCAD.

the updated preview, together with the source line and its number is enough information to have the test run.

In my picture, all the end user developer has to do is to write a comment behind the call to the module and the rest is done by OpenSCAD and the test framework. The easier the tests the better it is. We can add more precision, if we have proven, that we need it.

[Ruudjhuu]

Lets first agree on the definition of a unit test:

• A unittest tests a "unit" of code. Can be one method or multiple methods which are part of the same logical concept.
• Should be automated
• Full controll over all pieces
• Consistent
• Runs Fast
• Maintainable
• Idealy all unittest together have 100% code and branch coverage.
• A unittest should test 1 thing. So one unitest per branch.

My coments on your comments

• Test name
  I fully agree, file name and line number is enough.

• Control arguments / children I think we are on the same page here. Currently openscad is executing all scad code. I isolate the module in a seperate file due to the mock capabilities I implemented. I'll touch the mocking stuff a little later in this story.
  In the end, we want to write multiple testscases for one module with different arguments.

• Are we able to create a shortcut in openscad to run all the tests? I don't think so. I use vscode for writing openscad. (see the folowing extentions: Openscad, Openscad language support, OpenSCAD formatter). In vscode it is also posible to run all tests with one button. But this is not in the scope of this discussion.

• Global variables
  With global variables I do not mean global constants as they are defined in one constance file (or multiple in the future) and that is fine as they should not change ever.
  I mean variables starting with $ and then not the special openscad variables. In the current code base we have them and they are not constant. They function as a hidden argument for many modules and are defined in 1 module depending on its arguments. In unittest you need to be able to controll every aspect of the unit you are testing. So also the wrongly used global variables.

• Mock modules.
  I do think we need them. I don not know what kind of code you usualy write, but mocks are a big part of unittesting. If you use file io, databases, network communication, anything that will take more than 500 ms. You want to mock it. Openscad is a functional language and extreamly slow. Usual with functional languages you get a huge callstack. For example if you want to test cutEqual it has a dependency tree like:

  • cut
  • cut_move
    • cut_move_unsafe
  • block_cutter
    • fillet_cutter
    • transform_main
    • profile_cutter
    • profile_cutter_tab

  Which all have their own dependencies, create complex results, take more time then needed and could break the unittest while the test does not intent to cover the lines that break. This will make the test unmaintainable. The only thing cutEqual does is a double for loop and some location calculations. You don't need a fancy negative cut object to test it.
  I did a litle testing myself for cutEqual:

  • With mocking:

    Test output

    test_normal (test_unit_mock_vs_nonmock.cutEqual_mock) ... ok
    
    ----------------------------------------------------------------------
    Ran 1 test in 0.425s
    
    OK

    OpenScad output

    Geometries in cache: 1
    Geometry cache size in bytes: 728
    CGAL Polyhedrons in cache: 2
    CGAL cache size in bytes: 22688
    Total rendering time: 0:00:00.464
    Top level object is a 3D object:
    Simple:        yes
    Vertices:        8
    Halfedges:      24
    Edges:          12
    Halffacets:     12
    Facets:          6
    Volumes:         2
    Rendering finished.

  • Without mocking:

    Test output

    cutEqual_nonmock) ... ok
    
    ----------------------------------------------------------------------
    Ran 1 test in 10.441s
    
    OK

    OpenScad output

    Geometries in cache: 46
    Geometry cache size in bytes: 114968
    CGAL Polyhedrons in cache: 85
    CGAL cache size in bytes: 43601904
    Total rendering time: 0:00:10.400
    Top level object is a 3D object:
    Simple:        yes
    Vertices:     4850
    Halfedges:   19250
    Edges:        9625
    Halffacets:   9650
    Facets:       4825
    Volumes:        26
    Rendering finished.

    The test without mocking takes 20x as long. 10 whole seconds, this can't be a unittest anymore. We need mocking capabilities for duration alone!

• 2D vs 3D
  You are correct. The limitation with openscad is that it only can export 2D results to 2D formats and 3D results to 3D formats. Afaik there is no hook to use the preview result for these kind of tests.

Personal Conclusion

I do not see the added value to not write the tests in python. If so, we would add more complexity, another parser just to write the exact same tests in comments in a different txt file.
When using python we can also use modern language features when writing tests. Like linting, spell check, autocomplete, enz. If everything is defined in comments, you rely on the quality of the self written parser to check if a test is correctly written and only find out runtime.

[kennetek]

Small slightly-related tangent: The variables inside some of the files that use the $ symbol were from refactoring the code so that everything would be contained in modules (if I remember correctly). I did it in that slightly-bad way because I wanted it done quickly. I don't think that is the proper way to do things, those variables should probably be passed as arguments. Although, that may cause the argument count for many modules to increase dramatically. (Edit: I see that you already plan to remove them)