Closed mebel-christ closed 1 year ago
Hey, thanks, this is a very good idea! I think what you're proposing makes a lot of sense. Can you send a PR with a change to the docs and also selftest/conftest.py
for it? :)
I have one comment about the example you make:
import pytest
import tbot
def _interfaces():
with tbot.ctx.request(tbot.role.Board) as b:
return b.interfaces
@pytest.mark.parametrize("interface", _interfaces())
def test_interface(interface):
print(interface)
I don't think you should do a request()
here for collecting the available interfaces. This will try initializing the board, so potentially power it on only to power it off immediately after. A better way would be to do this:
def _interfaces():
board_cls = tbot.ctx.get_machine_class(tbot.role.Board)
return board_cls.interfaces
This will not attempt interacting with a real board instance and just retrive the "cold" data from the class definition.
I am working on the PR. Additionally I did some further testing:
I have one comment about the example you make:
import pytest import tbot def _interfaces(): with tbot.ctx.request(tbot.role.Board) as b: return b.interfaces @pytest.mark.parametrize("interface", _interfaces()) def test_interface(interface): print(interface)
I don't think you should do a
request()
here for collecting the available interfaces. This will try initializing the board, so potentially power it on only to power it off immediately after.
This is a great hint I missed that since I am working on my local machine with the dummy setup and not the real lab.
A better way would be to do this:
def _interfaces(): board_cls = tbot.ctx.get_machine_class(tbot.role.Board) return board_cls.interfaces
This will not attempt interacting with a real board instance and just retrive the "cold" data from the class definition
Your proposed improvement works perfectly well, but it has a slight drawback. I can no longer use/implement the interfaces in the board with @property
since properties belong to an instance and not the class itself. But TBH since this is static information anyway I do not really care. So I went with the following instead, but this is rather off-topic:
from typing import Dict, Set
from tbot import role
from tbot.machine import board, connector
class DummyBoard(connector.SubprocessConnector, board.Board):
interfaces: Dict[str, Set[str]] = {
"interface": {"interface0", "interface1"},
}
More importantly I moved the config load step for tbot to an even earlier point in time, namely the pytest_configure hook. This was done to make use of pytest-metadata (+ pytest-html) to read and print metadata from a Board, but the more important point IMO is what the pytest-doc has to say about the hook:
pytest_configure(config)
Allow plugins and conftest files to perform initial configuration. This hook is called for every plugin and initial conftest file after command line options have been parsed. After that, the hook is called for other conftest files as they are imported.
To me this sounds like the even more appropriate (and portable?) location (than pytest_sessionstart) for loading a configuration file. @Rahix What do you think?
Agreed, pytest_configure()
sounds better! :+1:
@Rahix I forked and added the documentation for this. TBH I am uncertain what to do about selftest/conftest.py now. I looked at it and as far as I understand the configuration for the testmachines is not loaded in any related way or am I misunderstanding something here?
Ah, you are right, the selftests do things a bit differently. In that case, ignore what I said, sorry!
Great, in that case I am going to open the PR
I am currently working on a test setup with tbot + pytest, but I am facing some challenges with the proposed skeleton integration. As you know the execution of pytest can be roughly boiled down to the following two stages:
By design pytest does not execute fixtures during collection stage which in turn makes it impossible to interact with tbot during that stage since the tbot integration is solely done as a fixture. I want to propose an alternative integration mechanism to make it possible to interact with tbot during collection stage
Use case
First up I want to outline my use case why I want to interact with tbot during collection stage in the first place with the following (minimal) working example. I have several configuration files that may look like this:
I want to manage variable interface types and amounts at the board level. For that regard I use a property in the Board class that returns how many instances of a certain type this Board has. For simplicity reasons in this example the interface is modeled by a simple string.
Assume there is a test that checks whether a single interface (of a certain type) works as expected. Naturally I want to run this test for every single interface instance present on a Board. With pytest this is possible by using the
pytest.mark.parametrize
decorator like so:With the currently described integration mechanism this results in
tbot.error.MachineNotFoundError: no machine found for <class 'tbot.role.Board'>
since parametrize runs during the collection stage and hence when requesting the Board context in_interfaces()
the fixture that loads the config was not executed yet.Proposal
To make this work I propose the following change to
conftest.py
:Move the "configuration registration" step to the pytest_sessionstart hook. This loads the tbot configuration before the collection stage runs and makes it possible to parametrize a test in the described fashion:
Are there some Gotchas I am missing here? Otherwise I would happily supply a PR to the doc. Feedback on that approach is greatly appreciated.