Functional/Unit/Integration Tests?

[westurner]

Are there tests for this software?

Is there a Software QA / Testing plan / recommended toolset for sugar or specifically for pippy?

I see: https://wiki.sugarlabs.org/go/Category:Testing

From https://bugzilla.redhat.com/show_bug.cgi?id=1855939#c1 ::

A test suite for this package (to be run post-packaging) would reduce the likelihood of these sorts of errors.

[quozl]

Not that I'm aware of. Testing is mostly by hand and without automation.

[pmccabe5]

Does the testing automation still need to be added for this project?

[quozl]

I don't think it is needed, but if you want to do it and are willing to keep it running forever, go right ahead. It's a big investment in personal time, and at the rate of change for Pippy (see commits) it hardly seems justified. A step by step guide for human testing might be a better investment. We have a guide now.

[westurner]

It's extremely unlikely that an investment of time into automated testing would be a waste relative to other development activities.

Testing [GUI] apps manually is a huge time suck; and humans skip steps and find bugs not yet present in the test.

Maybe start with coverage on the main() function? pytest-cov is one way to check test coverage: what percentage of source code lines and or expressions are executed by at least one test function:

pytest --cov=mypkg --cov-report=term-missing tests/

https://pypi.org/project/pytest-cov/

Is there a specific pytest plugin (e.g. pytest-qt) that the sugarlabs projects prefer to use for testing GUI apps in sugar; because quality assurance?

• https://pypi.org/project/pytest-qt/
  • https://pytest-qt.readthedocs.io/en/latest/intro.html

Manual testing is very slow, expensive, and error prone. For one, automated testing makes it possible to change without breaking other people's code: if it's going to be days or weeks to get a PR merged due to waiting for manual testing to occur, people don't consider contributing.

[pmccabe5]

My idea wasn't to develop the automated testing for just Pippy but rather a general framework from a project that I have worked on for the calculate activity and modify it so it can test different activities for the entire project (since this was one of the first projects that called for a need for automated testing).

[westurner]

Minimally, what does a GUI testing framework have?

• run whatever event loop, wait for messages/events, forward messages/events to the event loop, and shutdown said process(es) upon completion
• selectors; an interface for selecting gui nodes that as testers we don't otherwise have the references to
• assertions; custom object comparisons; recursive diff
  • https://github.com/pytest-dev/pytest/pull/6835

[sparshg]

I don't think a testing framework would be required. As @quozl mentioned, last commit was made 3 years ago, the frequency of commits is way too little to justify the efforts needed to build a whole testing framework for Pippy. I agree this could be helpful if built to integrate with any sugar activity, even though I don't think many activities have that active commit history, but then this issue shouldn't belong here. What do you think @chimosky

[westurner]

This works with and without pytest (and with python -O, which strips assert keyword statements and docstrings):

import unittest
test = unittest.TestCase()

def test_one():
    assert 1 == 1, "one does not equal one!"
    test.assertEqual(1, 1, "one doesn't = one")

def main(argv=None):
    if '-t' in argv or '--test' in argv:
        run_all_tests()  # pytest.main

Pytest preprocesses assert statements in order to upgrade assertion functions to better error messages.

Typically, we just run $ pytest in a shell with the correct path; though there's this trick to make a script self-testing:

From https://github.com/westurner/dotfiles/blob/a4fb3cb5fe2e49f5fbd8fb7903b13e60ffd334e1/scripts/git-statusbymtime.py#L165-L168 :

    if opts.run_tests:
        sys.argv = [sys.argv[0]] + args
        # return unittest.main()
        return subprocess.call(["pytest", "-v"] + args + [__file__])

Further notes on testing: https://github.com/sagemathinc/cocalc-doc/issues/40#issuecomment-1597678379

On Mon, Jul 3, 2023, 10:04 PM Sparsh Goenka @.***> wrote:

I don't think a testing framework would be required. As @quozl https://github.com/quozl mentioned, last commit was made 3 years ago, the frequency of commits is way too little to justify the efforts needed to build a whole testing framework for Pippy. I agree this could be helpful if built to integrate with any sugar activity, even though I don't think many activities have that active commit history, but then this issue won't belong here. What do you think @chimosky https://github.com/chimosky

— Reply to this email directly, view it on GitHub https://github.com/sugarlabs/Pippy/issues/84#issuecomment-1619355402, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMNSY5OOHKFLRYHYYGTSTXON23ZANCNFSM4OXFKZ3Q . You are receiving this because you authored the thread.Message ID: @.***>

[westurner]

A PR to demo testing without pytest or expect: https://github.com/4dsolutions/python_camp/pull/4/files

On Tue, Jul 4, 2023, 8:07 AM Wes Turner @.***> wrote:

This works with and without pytest (and with python -O, which strips assert keyword statements and docstrings):

import unittest
test = unittest.TestCase()

def test_one():
    assert 1 == 1, "one does not equal one!"
    test.assertEqual(1, 1, "one doesn't = one")

def main(argv=None):
    if '-t' in argv or '--test' in argv:
        run_all_tests()  # pytest.main

Pytest preprocesses assert statements in order to upgrade assertion functions to better error messages.

Typically, we just run $ pytest in a shell with the correct path; though there's this trick to make a script self-testing:

From https://github.com/westurner/dotfiles/blob/a4fb3cb5fe2e49f5fbd8fb7903b13e60ffd334e1/scripts/git-statusbymtime.py#L165-L168 :

    if opts.run_tests:
        sys.argv = [sys.argv[0]] + args
        # return unittest.main()
        return subprocess.call(["pytest", "-v"] + args + [__file__])

Further notes on testing: https://github.com/sagemathinc/cocalc-doc/issues/40#issuecomment-1597678379

On Mon, Jul 3, 2023, 10:04 PM Sparsh Goenka @.***> wrote:

I don't think a testing framework would be required. As @quozl https://github.com/quozl mentioned, last commit was made 3 years ago, the frequency of commits is way too little to justify the efforts needed to build a whole testing framework for Pippy. I agree this could be helpful if built to integrate with any sugar activity, even though I don't think many activities have that active commit history, but then this issue won't belong here. What do you think @chimosky https://github.com/chimosky

— Reply to this email directly, view it on GitHub https://github.com/sugarlabs/Pippy/issues/84#issuecomment-1619355402, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMNSY5OOHKFLRYHYYGTSTXON23ZANCNFSM4OXFKZ3Q . You are receiving this because you authored the thread.Message ID: @.***>

[chimosky]

I think testing would be of great help but right now except someone wants to take it on the themselves to write tests for this activity first before looking at the broader sugar activities then I'll gladly review it.

@westurner it'll be of great help if you have something you'd like to share.