integration tests

[SPlanzer]

• Python tests
• Python tests integrated with CI

[SPlanzer]

@strk This is a WIP.

But you feedback would be helpful.

Also if you have time it would be great if I could get your input in getting my make docker-qgis-test running it github actions.

[billgeo]

Thanks @SPlanzer . This looks like enough basic tests to me to merge QGIS 2to3.

[SPlanzer]

Thanks @SPlanzer . This looks like enough basic tests to me to merge QGIS 2to3.

Still need to act on feedback and merge in #109 plus get CI passing

[strk]

CI fails with: tee: /dev/tty: No such device or address when running:

docker-compose exec -T qgis sh -c "/usr/bin/xvfb-run -- qgis_testrunner.sh tests_directory.run_tests.run_test_modules"

The | tee /dev/tty snippet is in /usr/bin/qgis_testrunner.sh script, in QGIS docker container. I'm not sure why/dev/tty` should be unavailable, a quick test shows it is available for me. Maybe the xvfb-run call makes a difference ? Does it work on your machine ?

[SPlanzer]

CI fails with: tee: /dev/tty: No such device or address when running:

docker-compose exec -T qgis sh -c "/usr/bin/xvfb-run -- qgis_testrunner.sh tests_directory.run_tests.run_test_modules"

The | tee /dev/tty snippet is in /usr/bin/qgis_testrunner.sh script, in QGIS docker container. I'm not sure why/dev/tty` should be unavailable, a quick test shows it is available for me. Maybe the xvfb-run call makes a difference ?

Does it work on your machine ?

It does. I had hoped this would solve GitHub Action Display issues. I will look into why tty would be causing an issue this morning

[SPlanzer]

using tmate to log onto the github actions terminal make docker-qgis-tests runs as expected. See below.

runner@fv-az59:~/work/gazetteer/gazetteer$ make docker-qgis-test 
docker-compose up -d
gazetteer_db_1 is up-to-date
gazetteer_qgis_1 is up-to-date
docker-compose exec qgis sh -c "export COMPOSE_INTERACTIVE_NO_CLI=1"
docker-compose exec qgis sh -c "/usr/bin/xvfb-run -- qgis_testrunner.sh tests_directory.run_tests.run_test_modules"
Running test tests_directory.run_tests.run_test_modules ...
Warning: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
QGIS Test Runner Inside - starting the tests ...
QGIS Test Runner - Trying to import tests_directory.run_tests.run_test_modules
QGIS Test Runner - Trying to import tests_directory.run_tests
QGIS Test Runner Inside - executing function <function run_test_modules at 0x7fefb8d60ae8>
test_metadata (test_metadata.TestMetadata) ... ok
test_click_search (test_ui.TestUi) ... ok
test_click_tab_text (test_ui.TestUi) ... ok
test_editordockwidget_is_docked (test_ui.TestUi) ... ok
test_layers_load_on_start (test_ui.TestUi) ... ok
test_tools_disabled_on_start (test_ui.TestUi) ... ok
test_tools_enabled_on_start (test_ui.TestUi) ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.889s

OK
Finished running test tests_directory.run_tests.run_test_modules (codes: IS_DEAD=1 IS_FAILED=1 IS_PASSED=0).

Running tests to investigate the github actions temrinal reports (https://github.com/linz/gazetteer/runs/631670773?check_suite_focus=true)

$ echo $TERM
> dumb

$ echo "$(tty)"
> not a tty

Related reading: https://github.com/actions/runner/issues/241 - Github actions not tty https://bravelab.io/blog/migrating-from-travisci-to-github-actions-1/ - the only other mentions (and very brief) I have seen of such an issue with docker-compose GH actions and tty

[SPlanzer]

I have attempt to set the docker container to TERM=dumb to match thegithub action term

  qgis:
    image: linz-gazetteer-qgis:3.10
    build: src/NZGBplugin
    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix
      - ./src/NZGBplugin:/root/.local/share/QGIS/QGIS3/profiles/default/python/plugins/NZGBplugin
      - ./.docker/qgis/scripts:/scripts
    depends_on:
      - db
    command: /scripts/docker-entry.sh
    environment:
      PGHOST: db
      PGPASSWORD: gazadmin
      PGUSER: gazadmin
      PGDATABASE: gazetteer
      TERM: dumb

This is failing on

docker-compose exec qgis sh -c "/usr/bin/xvfb-run -- qgis_testrunner.sh tests_directory.run_tests.run_test_modules"
the input device is not a TTY

using tmate to SSH on the github action build test can be successfully executed in this environment

log in to the docker containers reports TERM as dumb

[SPlanzer]

back to TravisCI?

[strk]

Probably unrelated but I dubt that docker-compose exec qgis sh -c "export COMPOSE_INTERACTIVE_NO_CLI=1" would set environment for any next execution (rather than each execution having its own environment).

About make docker-qgis-test, it does NOT work on my machine, not sure why:

[strk@liz:/usr/src/linz/gazetteer/gazetteer(int_tests)] make docker-qgis-test
docker-compose up -d
gazetteer_db_1 is up-to-date
gazetteer_qgis_1 is up-to-date
docker-compose exec qgis sh -c "export COMPOSE_INTERACTIVE_NO_CLI=1"
docker-compose exec qgis sh -c "/usr/bin/xvfb-run -- qgis_testrunner.sh tests_directory.run_tests.run_test_modules"
Running test tests_directory.run_tests.run_test_modules ...
Warning: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
QGIS Test Runner Inside - starting the tests ...
QGIS Test Runner - Trying to import tests_directory.run_tests.run_test_modules
QGIS Test Runner - Trying to import tests_directory.run_tests
QGIS Test Runner Inside - [FAILED] Exception: No module named 'tests_directory.run_tests'
Traceback (most recent call last):
  File "<string>", line 70, in __get_test_function
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tests_directory.run_tests'

Any idea about that ?

[strk]

On Wed, Apr 29, 2020 at 08:36:27PM -0700, Simon Planzer wrote:

back to TravisCI?

Let's look forward. We know NZBuildings is using a similar setup with success:

https://github.com/linz/nz-buildings/blob/master/.github/workflows/build.yml#L100-L129

[strk]

@SPlanzer did you forget to push a run_tests.py script ?

[SPlanzer]

@SPlanzer did you forget to push a run_tests.py script ?

Yip in this branch.

Have been running a branch called test as not to make a mess of this once while I throw many setting at it.

This the "test" branch has run_tests.py and I will clean up this one once I learn what must be done to pass CI

[billgeo]

back to TravisCI?

Let's look forward.

I agree with @strk . Travis is being phased out. So going back to it would only be a temporary solution.

[SPlanzer]

back to TravisCI?

Let's look forward.

I agree with @strk . Travis is being phased out. So going back to it would only be a temporary solution.

I have it running now. That is, CI failing when I pass in tests that should fail and CI passing when I hand it tests that are expected to pass.

I just dont have CI streaming the output from the tests to the terminal

[SPlanzer]

@strk passing :partying_face:

The qgis_scriptrunner.sh that is part of the QGIS docker image is not compatible with GiHub actions, Hence the .docker/qgis/scripts/testrunner.sh