Wokwi Featured Functions (RDT-568)

hfudev commented 11 months ago

Wokwi support more functionalities other than serial I/O. like pushing buttons, set/get the state of sensors (described here)

Once the API is settled, implement this in pytest-embedded-wokwi

hayschan commented 9 months ago

Hello Hanxi,

Thank you for your response in the previous discussion regarding the CI/CD docker.

Over the past few days, I've been delving into the Unity unit testing framework and pytest-embedded as part of our ESP-IDF software development. I'm currently working on a project that involves testing a matrix keyboard, and I've successfully completed the development of the test_app. To streamline the testing process and eliminate the need for manual intervention (i.e., physically pressing the keyboard), I am considering the implementation of Wokwi.

However, I've encountered a couple of areas where additional resources or documentation would be immensely helpful:

API Documentation: The API section for pytest-embedded on Wokwi appears to be incomplete or unpopulated at this time.
Practical Examples: The current repository lacks examples that demonstrate the integration of Wokwi with pytest-embedded.

Furthermore, the existing examples, such as confirming a "Hello World" output, don't fully represent complex, real-world scenarios. It would be highly advantageous to have examples that cover more intricate use cases, like I2C interaction or WiFi connectivity testing, timers.

I believe addressing these areas would significantly enhance the utility and applicability of pytest-embedded for more complex development environments.

Looking forward to further updates or guidance on these matters.

Best regards.

hfudev commented 9 months ago

Hi @hayschan! Thank you for your input.

API Documentation

pytest-embedded-wokwi is a wrapper of the wokwi-cli. For now both wokwi-cli and pytest-embedded-wokwi are in alpha. We're working together with @urish to expose more APIs for testing.

Practical Examples

Yes as you said, we're lack of real world examples here in this repo. We will improve it in a forseeable future. For now if you're working together with ESP-IDF projects, you may take a look at the example projects in ESP-IDF repo: wifi getting started example

please note that the pytest.mark.esp32 marker is not applicable with vanilla pytest-embedded. We wrote some customized conftest.py in the esp-idf repo to support the syntactic sugar. You may use

import pytest

pytest.mark.parametrize('target', [
    'esp32',
], indirect=True)

instead.

urish commented 9 months ago

Hello! I just created a demo repo, based on espressif's gh-esp-test-template, which shows a complete setup using pytest embedded together with Wokwi in GitHub CI:

https://github.com/wokwi/wokwi-esp-test-template

It supports (almost) all espressif's ESP32 chips (including the not-yet-released ESP32-P4), and could serve for building more elaborate examples in the future.

Example workflow run (with 7 targets): https://github.com/wokwi/wokwi-esp-test-template/actions/runs/7528389475

hayschan commented 9 months ago

based on espressif's gh-esp-test-template, which shows a complete setup using pytest embedded together with Wokwi in GitHub CI:

To make sure everything is working at least locally, I have tried to run the Wokwi CLI on my computer.

I have two files that is important:

test.cpp for writing the TEST_CASE
pytest_test.py for using the pytest-embedded API

I encountered some problem.

Original problem

In my test.cpp, I have serveral test cases and app_main.

The app_main() uses unity_run_menu().

extern "C" void app_main(void)
{
    unity_run_menu();
}

And the pytest-embedded python file uses run_all_single_board_cases().

import pytest
from pytest_embedded import Dut

def test_aip1629(dut: Dut)-> None:
    dut.run_all_single_board_cases()

Then, this will happen:

I built the app by idf.py build, and then run pytest and wokwi using pytest -s --embedded-services idf,wokwi --tb short.
The app is uploaded to Wokwi CLI with success

Received the following error very quickly

2024-01-16 21:21:05 Wokwi CLI v0.8.0
2024-01-16 21:21:06 Connected to Wokwi Simulation API 1.0.0-20240114
2024-01-16 21:12:18 Press ENTER to see the list of tests.
FAILED

=================================================== FAILURES =================================================== 
_________________________________________________ test_aip1629 _________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:120: in expect
    return self.pexpect_proc.expect(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:354: in expect
    return self.expect_list(compiled_pattern_list,
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:383: in expect_list
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x00000181F61CF350>
E   searcher: searcher_re:
E       0: re.compile(b"Here's the test menu, pick your combo:(.+)Enter test for running.")
E   <pytest_embedded.log.PexpectProcess object at 0x00000181F61CF350>
E   searcher: searcher_re:
E       0: re.compile(b"Here's the test menu, pick your combo:(.+)Enter test for running.")

The above exception was the direct cause of the following exception:
pytest_aip1629.py:38: in test_aip1629
    dut.run_all_single_board_cases()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:428: in run_all_single_board_cases
    for case in self.test_menu:
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:243: in test_menu
    self._test_menu = self._parse_test_menu()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:125: in _parse_test_menu
    res = self.confirm_write(trigger, expect_pattern=pattern)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:102: in confirm_write
    raise err
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:94: in confirm_write
    res = self.expect(expect_pattern, timeout=timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "Here's the test menu, pick your combo:(.+)Enter test for running."    
E   Bytes in current buffer (color code eliminated): .
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_aip1629 - pexpect.exceptions.TIMEOUT: Not found "Here's the test menu, pick your combo:(.+)Enter test for running."

My way to solve this

By simply changing the unity functions used in app_main from unity_run_menu() to unity_run_all_tests(), all test cases could be run successfully.

extern "C" void app_main(void)
{
-    unity_run_menu();
+    unity_run_all_tests();
}

New error occured

Strangely, even after running all tests, it will still fail with TIMEOUT with the following error code:

2024-01-16 21:21:32 ./main/aip1629_test.cpp:331:Test Display During Heating:PASS
2024-01-16 21:21:32 Running Test Invalid Inputs...
2024-01-16 21:21:32 ./main/aip1629_test.cpp:340:Test Invalid Inputs:PASS
2024-01-16 21:21:32 I (10950) main_task: Returned from app_main()
FAILED

=================================================== FAILURES =================================================== 
_________________________________________________ test_aip1629 _________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:141: in expect_exact
    return self.pexpect_proc.expect_exact(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:432: in expect_exact
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x0000019A33A84B50>
E   searcher: searcher_string:
E       0: b'Press ENTER to see the list of tests'
E   <pytest_embedded.log.PexpectProcess object at 0x0000019A33A84B50>
E   searcher: searcher_string:
E       0: b'Press ENTER to see the list of tests'

The above exception was the direct cause of the following exception:
pytest_aip1629.py:38: in test_aip1629
    dut.run_all_single_board_cases()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:428: in run_all_single_board_cases
    for case in self.test_menu:
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:243: in test_menu
    self._test_menu = self._parse_test_menu()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:124: in _parse_test_menu
    self.expect_exact(ready_line)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "Press ENTER to see the list of tests"
E   Bytes in current buffer (color code eliminated): _task: Returned from app_main()
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_aip1629 - pexpect.exceptions.TIMEOUT: Not found "Press ENTER to see the list of tests"
============================================== 1 failed in 30.24s ==============================================

hayschan commented 9 months ago

Another combination but error persists

I have also tried changing the python test file. So my combination of Python and C++ looks is as followed.

from pytest_embedded_idf.dut import IdfDut

def test_app(dut: IdfDut):
    dut.expect_unity_test_output()

extern "C" void app_main(void)
{
    unity_run_all_tests();
}

An error about regex failing to find test cases results will show.

2024-01-16 21:28:15 ./main/aip1629_test.cpp:340:Test Invalid Inputs:PASS
2024-01-16 21:28:15 I (10950) main_task: Returned from app_main()
2024-01-16 21:28:36 Timeout: simulation did not finish in 30000ms
FAILED

=================================================== FAILURES =================================================== 
___________________________________________________ test_app ___________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:120: in expect
    return self.pexpect_proc.expect(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:354: in expect
    return self.expect_list(compiled_pattern_list,
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:383: in expect_list
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x000002AC105CF350>
E   searcher: searcher_re:
E       0: re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)')
E   <pytest_embedded.log.PexpectProcess object at 0x000002AC105CF350>
E   searcher: searcher_re:
E       0: re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)')

The above exception was the direct cause of the following exception:
pytest_aip1629.py:44: in test_app
    dut.expect_unity_test_output()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:168: in expect_unity_test_output
    self.expect(UNITY_SUMMARY_LINE_REGEX, timeout=timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)', re.MULTILINE)"
E   Bytes in current buffer (color code eliminated): Wokwi CLI v0.8.0 (2a4bae54982c) Connected to Wokwi Simulation API 1.0.0-20240114-g15d4d362 Starting simulation... ESP-ROM:esp32s3-20210327 Build:Mar 27 2021 rst:0x1... (total 8457 bytes)
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_app - pexpect.exceptions.TIMEOUT: Not found "re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*...
========================================= 1 failed in 60.26s (0:01:00) =========================================

hfudev commented 9 months ago

@hayschan For your c code, it looks good to me. Usually a simple unity_run_menu works

ref to our test app

seems like wokwi-cli does not support "write" yet. I tried calling the wokwi-cli locally without pytest-embedded, and the results are about the same

Running console component tests

Press ENTER to see the list of tests.

1

2

(The 1, 2, and empty lines are what I typed.)

Please note that wokwi-cli is still under active development, and we appreciate your patience as we work to improve it.

hayschan commented 9 months ago

Thanks for the confirmation, @hfudev

I have subscribed to Wokwi's club to support @urish 's work. Really excited about wokwi-cli. Once it is completed, it will make my life way easier. 😆

For now, I will be using the GitHub action self hosted runner until the completion of wokwi-cli.

urish commented 9 months ago

Thanks for confirming! What's the quickest way for me to reproduce this locally?

hayschan commented 9 months ago

Thanks for confirming! What's the quickest way for me to reproduce this locally?

I have created an example repo for you to clone. @urish

Please go to the test_apps path for the component test project.

I have already tried on my computer. The project can be built and has the symptoms I mentioned.

urish commented 9 months ago

Thank you so much @hayschan for creating a reproducible repo. There's now a fix - if you want to test it without waiting for a new pytest release, just clone this repo, switch to the issue-233 branch, go into the pytest-embedded-wokwi directory and run pip install .

Note that you should also update your wokwi-cli to the latest version, 0.9.0. To update, just follow the installation instructions which will install the newest version on top of the existing one.

hayschan commented 9 months ago

Thank you so much @hayschan for creating a reproducible repo. There's now a fix

I've just test it. It works flawlessly. 👍🏼

pytest -s --embedded-services idf,wokwi --tb short
============================================== test session starts ==============================================
platform win32 -- Python 3.11.2, pytest-7.4.4, pluggy-1.3.0 -- d:\.espressif\python_env\idf5.1_py3.11_env\Scripts\python.exe
cachedir: .pytest_cache
configfile: pytest.ini
plugins: embedded-1.6.2, rerunfailures-13.0, timeout-2.2.0
collected 1 item

pytest_max6675.py::test_max6675 <- test_apps\pytest_max6675.py
------------------------------------------------ live log setup ------------------------------------------------- 
2024-01-18 12:32:36 INFO Executing wokwi-cli --interactive
2024-01-18 12:32:37 Wokwi CLI v0.9.0
2024-01-18 12:32:38 Connected to Wokwi Simulation API 1.0.0
2024-01-18 12:32:40 Starting simulation...
2024-01-18 12:32:42 ESP-ROM:esp32s3
------------------------------------------------
2024-01-18 12:32:43 ./main/test_max6675.c:36:MAX6675 Sensor Test:PASS
2024-01-18 12:32:43 Test ran in 24ms
2024-01-18 12:32:43
2024-01-18 12:32:43 -----------------------
2024-01-18 12:32:43 1 Tests 0 Failures 0 Ignored
2024-01-18 12:32:43 OK
2024-01-18 12:32:43 Enter next test, or 'enter' to see menu
PASSED
----------------------------------------------- live log teardown ----------------------------------------------- 
2024-01-18 12:32:43 INFO Created unity output junit report

if you want to test it without waiting for a new pytest release

@hfudev will you integrate the wokwi fix into a new pytest-embedded version soon? The change of this fix is minimal, with an addition of the --interactive flag.

super().__init__(
-            cmd=[wokwi_cli, app.app_path],
+            cmd=[wokwi_cli, '--interactive', app.app_path],

hfudev commented 9 months ago

@hayschan Hi, after merging PR #261, version 1.6.3 has been released. Please have a try.

espressif / pytest-embedded