Property-based testing driven library implementation

Summary

This post argues that applying property-based testing (PBT) to current codebase (and making it the driver for future development) is a good & realistic future research direction. I got my engineering degree from Telkom University, Indonesia, and I must disclaim here that at the moment I am unaffiliated officially with any academic institutions.

Proposal

While replicating the success story of PBT in other software projects (as introduced in 1) here is an enough merit of itself, the main goal that this project should strive for is readability.

Instructive codebase for learning

Closed-sourced OptiSystem provides a way in C++ Components to programmatically extend its features. The user has full control of the custom component they write themselves, but others black-boxes.

As a thought experiment, imagine an alternative simulator. This alternative software's source is open; the user can download it, fork it, et cetera. The user can assemble provided components, write a custom one, and rely on the simulator for the rest of the system. This OptiSystem alternative isn't necessarily better in terms of source transparency, because what if the downloadable source is untraceable to the physics equations level. What if an unaccountable amount of trust must be relegated to every lines that the library author wrote; the user might as well trust the entire thing as a black box.

One way to increase readability is automating the checking of what we understand about a function. Take the following snippet as an example:

def mzm(Ai, u, param=None):
    if param is None:
        param = []

    # check input parameters
    Vpi = getattr(param, "Vpi", 2)
    Vb = getattr(param, "Vb", -1)

    try:
        u.shape
    except AttributeError:
        u = np.array([u])

    try:
        if Ai.shape == () and u.shape != ():
            Ai = Ai * np.ones(u.shape)
        else:
            assert Ai.shape == u.shape, "Ai and u need to have the same dimensions"
    except AttributeError:
        Ai = Ai * np.ones(u.shape)

    π = np.pi¬                                                                                                                 
    return Ai * np.cos(0.5 / Vpi * (u + Vb) * π)¬

Uncaptured from above snippet is the insight that the typical values for driving voltages range from 3 V to about 6 V. We can communicate the idea as a property that says "mzm invocations with non-typical values are interesting events but nevertheless must never crash". We stand on the shoulder of PBT theory and gain confidence that the property holds if we can translate that statement into Hypothesis syntax. The idea that I'm proposing here is that such properties, if checked automatically based on sound theory, make for instructive & educative opticomm system simulation implementation.

PBT success doesn't always manifest in failing tests that capture bugs; warning system designers for possible anomalies, via sanity checks, is also productive when designing communication systems. Further exploration is needed to come up with good property examples to demonstrate effective PBT application in this domain, I concede.

Rationale

Is prototyping speed harmed?

There's execution speed, and there's also (new) features development speed. Execution speed (i.e., invocations of library-provided components) shouldn't be impacted because PBT code is only run during testing. Development speed is harder to predict; it varies due to programmers' styles. https://hypothesis.readthedocs.io/en/latest/ghostwriter.html looks like a useful link related to this matter.

Isn't a test suite with high coverage what matters, still?

Yes, but I don't think being a more robust opticomm simulator than existing proprietary alternatives is the most urgent goal that we have in this domain.

Request for comments

Does it align with your general vision that the project evolves in this direction?
What are the non-goals of this project? I assume 1-to-1 feature mapping with any existing proprietary software is never the goal for OptiCommPy, but it's good to have a list of non-goals documented.

Hello @tbmreza,

Request for comments

Does it align with your general vision that the project evolves in this direction?

I believe your points align well with the overall vision for this project. Regarding the example you mentioned, we have made efforts to ensure that most of the code is traceable to relevant physical and mathematical models by including references in the documentation. However, it is not our intention for the code to directly educate users about optical communications. It is intended to be a useful tool for someone who is learning and at least has some basic knowledge about optical communications, since its usability requires a minimum degree of understanding of the topic. While we welcome contributions to enhance the documentation, it is not a priority for us due to time constraints and the limited impact on our primary activities using OptiCommPy.

What are the non-goals of this project? I assume 1-to-1 feature mapping with any existing proprietary software is never the goal for OptiCommPy, but it's good to have a list of non-goals documented.

There is no goal of being a 1-to-1 feature mapping to any proprietary software, and there is no coverage goal. The coverage may grow depending on our needs, or on external contributions.

edsonportosilva / OptiCommPy