Some test issues

[poesel]

'make' listet die tests nicht auf

FYI:

test/test_gprs_http.py::test_telemetry_gprs_http FAILED                  [ 21%]
test/test_lorawan.py::test_uplink_environmental_sensors FAILED           [ 31%]
test/test_sensors.py::test_sensors FAILED                                [ 57%]

Der Rest geht durch.

[amotl]

Dear @poesel,

thanks for your feedback!

List available tests

'make' listet die tests nicht auf

You might want to try this to enumerate all tests without executing them.

pytest --collect-only

Use this command to display all available markers.

pytest --markers

Failing tests

Some tests fail.

Can you share some more details about why these tests would fail for you? Did you invoke make setup and make setup-tests beforehand? I believe there might have been some changes.

Both test_sensors and test_uplink_environmental_sensors will include the 1-Wire/DS18B20 MicroPython machinery which needed to be updated to run on CPython through https://github.com/micropython/micropython/pull/5786.

While I don't know why test_telemetry_gprs_http would fail on your machine, cbc1ff92 might also have bumped the SIM800 driver to reflect the most recent changes from our pen [1]. There are chances these changes might be required to run the tests successfully.

Please let me know if any of these recommendations actually help to make those tests pass successfully on your end.

WIth kind regards, Andreas.

[1] https://github.com/hiveeyes/pythings-sim800/tree/pppos

[poesel]

Simply calling 'make' usually gives you a list with all options. Here it does not. Of course there are other ways.

Attached the log from the WSL. test.zip

[amotl]

Dear @poesel,

thanks for sharing the log.

test_telemetry_gprs_http

>       if sys.platform in ['WiPy', 'LoPy', 'LoPy4', 'SiPy', 'GPy', 'FiPy']:
E       NameError: name 'sys' is not defined

test_sensors and test_uplink_environmental_sensors

>       self.writebyte(MATCH_ROM)
E       NameError: name 'MATCH_ROM' is not defined

Both of these issues have been fixed by updating the 3rd-party modules as outlined in my post above. Please make sure you have a fresh copy of the dist-packages folder, populated by invoking make setup - you might try deleting the folder completely beforehand.

With kind regards, Andreas.

[amotl]

@thiasB just reported this error when invoking the test suite on Python 3.6.9.

FAILED test/test_basic.py::test_basic_esp32 - AttributeError: module 'time' has no attribute 'time_ns'

Probably because time_ns() just started existing from Python 3.7 onwards?

[amotl]

@poesel See cda1a18f for

$ make
[...]
setup-tests                   Setup requirements for running the testsuite
test                          Run testsuite
test-verbose                  Run testsuite, with verbose output
test-coverage                 Run testsuite, with coverage report
[...]

@thiasB see 23b8065a for compatibility with Python3.6.

Thanks!

[amotl]

@poesel and @thiasB: Please note that the requirements have also been updated. So, purging dist-packages and populating it again using make setup will be a good idea. Also, make setup-tests will add additional requirements to the testsuite.

[poesel]

Looks better:

root@Calvin:/home/markus/GitHub/terkintest/terkin-datalogger# make test
============================================================== test session starts ==============================================================
platform linux -- Python 3.7.3, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /mnt/c/Users/Markus/Documents/GitHub/terkintest/terkin-datalogger/.venv3/bin/python3
cachedir: .pytest_cache
rootdir: /mnt/c/Users/Markus/Documents/GitHub/terkintest/terkin-datalogger, inifile: pytest.ini
plugins: pyfakefs-4.0.2, httpserver-0.3.4, mock-2.0.0, pythonpath-0.7.3
collected 26 items

test/test_basic.py::test_basic_esp32 PASSED                                                                                               [  3%]
test/test_basic.py::test_basic_wipy PASSED                                                                                                [  7%]
test/test_basic.py::test_basic_cpython PASSED                                                                                             [ 11%]
test/test_gprs_http.py::test_telemetry_gprs_http PASSED                                                                                   [ 15%]
test/test_http_mocking.py::test_mocket_cpython_requests PASSED           [ 19%]
test/test_http_mocking.py::test_mocket_socket XFAIL                      [ 23%]
test/test_http_mocking.py::test_httpretty_cpython_requests PASSED        [ 26%]
test/test_http_mocking.py::test_httpretty_socket XFAIL                   [ 30%]
test/test_http_mocking.py::test_httpserver_cpython_requests PASSED       [ 34%]
test/test_http_mocking.py::test_httpserver_socket PASSED                 [ 38%]
test/test_lorawan.py::test_uplink_system_temperature PASSED                                                                               [ 42%]
test/test_lorawan.py::test_uplink_environmental_sensors PASSED                                                                            [ 46%]
test/test_lorawan.py::test_downlink_interval_set PASSED                                                                                   [ 50%]
test/test_lorawan.py::test_downlink_interval_reset PASSED                                                                                 [ 53%]
test/test_lorawan.py::test_downlink_pause PASSED                                                                                          [ 57%]
test/test_lorawan.py::test_downlink_unpause PASSED                                                                                        [ 61%]
test/test_sensors.py::test_sensors PASSED                                                                                                 [ 65%]
test/test_sleep.py::test_timesleep PASSED                                                                                                 [ 69%]
test/test_sleep.py::test_lightsleep PASSED                                                                                                [ 73%]
test/test_sleep.py::test_deepsleep PASSED                                                                                                 [ 76%]
test/test_sleep.py::test_maintenance PASSED                                                                                               [ 80%]
test/test_wifi.py::test_wifi_esp32 PASSED                                [ 84%]
test/test_wifi.py::test_wifi_wipy PASSED                                                                                                  [ 88%]
test/test_wifi.py::test_wifi_cpython PASSED                                                                                               [ 92%]
test/test_wifi_http.py::test_uplink_wifi_http PASSED                                                                                      [ 96%]
test/test_wifi_mqtt.py::test_telemetry_wifi_mqtt PASSED                                                                                   [100%]

======================================================== 24 passed, 2 xfailed in 11.01s ========================================================

What is xfailed?

[amotl]

Looks better.

Good to hear. Thanks for your feedback.

What is xfailed?

XFail tests are tests which are expected to fail [1]. A more verbose explanation is available from [2].

The xfail tests encodes our assumptions. It is subject to source control, it must still compile, and we can execute it. This prevents our assumptions from falling out of date with the current state of the software.

But during normal work with the test suite, these xfail tests do not cause the test suite to fail. This is similar to a skipped test, except that when an usually-skipped test is executed and fails, that would fail the complete test suite. Failure of an xfail test does not fail the test suite.

[1] http://doc.pytest.org/en/latest/skipping.html#xfail-mark-test-functions-as-expected-to-fail [2] https://softwareengineering.stackexchange.com/questions/382501/whats-the-purpose-of-xfail-tests

[amotl]

The issue can be closed, right? Feel free to reopen when appropriate.

[poesel]

Ran it again on macOS. Deleted dist-packages, git pull, make setup, make setup-tests

Nelson:terkin-datalogger markus$ make test
ImportError while loading conftest '/Users/markus/Documents/Github/terkintest/terkin-datalogger/test/conftest.py'.
test/conftest.py:7: in <module>
    monkeypatch()
test/util/micropython.py:16: in monkeypatch
    monkeypatch_hal()
test/util/micropython.py:323: in monkeypatch_hal
    import onewire_native
E   ModuleNotFoundError: No module named 'onewire_native'
make: *** [test] Error 4

[amotl]

Bummer! Does the onewire_native.py module exist within your dist-packages folder? Maybe try make setup again and check if it reports any errors.

[poesel]

Now its different :) This is the second run. The first run had more errors but those vanished.

============================= test session starts ==============================
platform darwin -- Python 3.7.4, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /Users/markus/Documents/Github/terkintest/terkin-datalogger/.venv3/bin/python3
cachedir: .pytest_cache
rootdir: /Users/markus/Documents/Github/terkintest/terkin-datalogger, inifile: pytest.ini
plugins: pyfakefs-4.0.2, mock-2.0.0, httpserver-0.3.4, pythonpath-0.7.3
collecting ... collected 29 items

test/test_basic.py::test_basic_esp32 PASSED                              [  3%]
test/test_basic.py::test_basic_wipy PASSED                               [  6%]
test/test_basic.py::test_basic_cpython PASSED                            [ 10%]
test/test_gprs_http.py::test_uplink_gprs_http PASSED                     [ 13%]
test/test_http_mocking.py::test_mocket_cpython_requests PASSED           [ 17%]
test/test_http_mocking.py::test_mocket_socket XFAIL                      [ 20%]
test/test_http_mocking.py::test_httpretty_cpython_requests PASSED        [ 24%]
test/test_http_mocking.py::test_httpretty_socket XFAIL                   [ 27%]
test/test_http_mocking.py::test_httpserver_cpython_requests PASSED       [ 31%]
test/test_http_mocking.py::test_httpserver_socket PASSED                 [ 34%]
test/test_lorawan.py::test_uplink_system_temperature PASSED              [ 37%]
test/test_lorawan.py::test_uplink_environmental_sensors PASSED           [ 41%]
test/test_lorawan.py::test_downlink_interval_set PASSED                  [ 44%]
test/test_lorawan.py::test_downlink_interval_reset PASSED                [ 48%]
test/test_lorawan.py::test_downlink_pause PASSED                         [ 51%]
test/test_lorawan.py::test_downlink_unpause PASSED                       [ 55%]
test/test_sensors.py::test_sensors PASSED                                [ 58%]
test/test_sleep.py::test_timesleep PASSED                                [ 62%]
test/test_sleep.py::test_lightsleep PASSED                               [ 65%]
test/test_sleep.py::test_deepsleep PASSED                                [ 68%]
test/test_sleep.py::test_maintenance PASSED                              [ 72%]
test/test_urequests.py::test_urequests_basic PASSED                      [ 75%]
test/test_urequests.py::test_redirect_cpython_requests PASSED            [ 79%]
test/test_urequests.py::test_redirect_urequests PASSED                   [ 82%]
test/test_wifi.py::test_wifi_esp32 PASSED                                [ 86%]
test/test_wifi.py::test_wifi_wipy PASSED                                 [ 89%]
test/test_wifi.py::test_wifi_cpython PASSED                              [ 93%]
test/test_wifi_http.py::test_uplink_wifi_http PASSED                     [ 96%]
test/test_wifi_mqtt.py::test_uplink_wifi_mqtt ERROR                      [100%]

==================================== ERRORS ====================================
___________________ ERROR at setup of test_uplink_wifi_mqtt ____________________

[amotl]

So, just test_httpserver_socket and test_uplink_wifi_mqtt are failing? Do you have any more details?

[poesel]

Sure:

test.zip

[amotl]

Thanks, @poesel!

1. test_httpserver_socket is flaky. It should hopefully work when re-running (a few times).
2. test_uplink_wifi_mqtt seems to be failing because Docker doesn't respond properly in order to spin up Mosquitto. Didn't you run Mosquitto locally on your machine?

   
   .venv3/lib/python3.7/site-packages/docker/transport/unixconn.py:43: ProtocolError

During handling of the above exception, another exception occurred:

@pytest.fixture(scope='session')
def mosquitto():

    # Gracefully skip spinning up the Docker container if Mosquitto is already running.
    if is_mosquitto_running():
        yield
        return

    # Spin up Mosquitto container.
    if os.environ.get('MOSQUITTO'):
        yield os.environ['MOSQUITTO'].split(':')
    else:

      yield mosquitto_image.run()

[poesel]

Ah, sorry. I don't have mosquitto on the mac or docker set up to run it. My fault - since it runs on your machine I guess it would run on mine, too, if I'd set it up properly.